I don't know how many times we have to keep saying this:
BECAUSE LLMS HAVE NO INTENTION OR ABILITY TO TELL TRUTH FROM FICTION SO WHEN THEY APPEAR CONFIDENT, THEY ARE BULLSHITTING EVEN WHEN THEY ARE CORRECT.
Even a bullshitter can be correct sometimes, it doesn't make it suddenly not bullshit. Even when LLMs get it right, they're still bullshitting.
This isn't complicated. They don't think. They have no concept of truth. They just fabricate sentences from previously copied sentences, there is no intention, no thought, no planning, no reflection.
The groups producing these LLMs are just sourcing the entire internet, they don't care how much of it is lies. There seems to be very little curation going on.
Anyone who expected anything other than this outcome is an idiot who isn't paying attention.