> But AI's problem is that its completely full of shit, sometimes

It's now quite unusual that it's "Completely full of shit". If it contradicts something your doctor said I don't see why you should feel ashamed to bring it up. Sure it complicates the doctor's work, having ignorant obedient patients must be more comfortable for the doctor, but the end result could be more accurate diagnosis.

The notion that only doctors can verify is false! Doctors are better at verification but normal people can also verify. This is just empirically true.

Examples of things normal people can verify

- procedural errors that Claude can capture like some blatantly high dosage (grams instead of milligrams)

- outdated treatment plan, maybe there’s a credible new treatment plan that’s been used for years but the doctors were not updated

- literally being injected homeopathic drugs (takes no smart person to flag this)

Let’s stop talking as if doctors have a divine right here. And let’s accept some agency.

But those are obvious errors. What if the AI tells you to up your intake of X and it seems plausible? So you up your intake of X by taking some supplement, but upping the intake of X makes your body deplete more of Y and now you have a new or compounding problem.

A doctor might have never recommended upping X, because they would know what it does to your body. Or they might have suggested additional supplementation to avoid this.

The fact that LLMs are trained on all public knowledge is a huge red flag, because there are more wrong infos out there than right ones. Especially about health, diet, etc.

I was about to respond but your last statement implies fundamental misunderstanding of how LLMs work. I don’t think you even know about RLHF and you think good and bad ideas are spread in proportion to how much they are seen on the internet.