> which I wouldn't be able to easily verify, I might even be convinced the LLM has actually demonstrated its case and has conclusively proven that it's right

I think this example is one of many that has demonstrated why no output from an LLM can be trusted without outside verification.