And if "next gen SEO" (adversarial training data that cause something to be inappropriately or disproportionately represented in responses) lands in your chat search, it isn't a matter of just setting flags in a database to ignore or penalize a set of documents - you're gunnuh have to either retrain, or add a new set of layers (or similar) to filter out/penalize these results. If this starts happening at anything close to the rate that Google encounters spam, I can't see how they would keep up.

Language models are tested with hundreds of benchmarks including everything from bias to factuality and reasoning correctness. When they have a big deficiency it shows in the test scores.

Big deficiencies aren't monetizable by adversaries, tiny ones are (eg, impacting it's response to questions about one topic in particular).

In a very narrow niche there may not be many documents to pick from, either.

I don't think you can just automate this away in the context of generalized search. Search has to fulfill every niche; that seems like an indefensible position (strategically speaking, not in the moral sense). How can you benchmark bias in every niche? Your benchmarks show your reasoning is sound; what about the premises you're reasoning from? In the context of something narrowly scoped like a customer service bot, it makes sense to me how you could build expertise in constraining the model's output. But in terms of everything?

But I'll admit I don't have a crystal ball, happy to eat my words if they can operate with enough traffic and got long enough to attract the attention of spammers, and still keep them at bay. I think this space is stagnant and needs to be shaken up, and that chat interfaces have potential, so I'm not trying to be a hater. I just think this is gunnuh be a very difficult aspect.