Can you please also hallucinate a plausible-sounding justification for this otherwise unsubstantiated statement?

Jokes aside, we do produce plausibile sounding stuff all the time well beyond the limit of what we actually know or can prove. I think there is a continuum between formulating statements about things we don't know for sure and we can't prove, guessing details here and there to fill gaps in our memory, misremembering things that we thought we knew, and making up entire facts that sound plausible but are completely invented. Yes, llms seem to have trouble introspecting what they actually know; but it sounds more like a missing skill rather than a fundamental difference in the way they reason.