After suffering though endless equivocations of the term "knowledge" by various AI bots,
I asked Google AI, "Why does AI conflate information processing and authentic knowledge?"

You might find its response revealing and useful ...

AI conflates information processing with authentic knowledge because generative models are designed as pattern-matching machines that prioritize plausible text generation over factual verification. By predicting the next word based on statistical probability rather than understanding truth, AI "fills in gaps" with confident-sounding fabrications.

Key reasons for this conflation include:
Pattern Recognition vs. Understanding: LLMs are built to predict the next most likely token, not to comprehend facts or reality.

Training Data Limitations: Models are trained on massive datasets that contain misinformation, biases, and gaps, which they confidently reproduce.

Lack of Grounding: AI does not interact with the physical world to verify information; it operates purely on linguistic patterns.

Objective Functions: Models are optimized for coherence and conversational flow, not for the accuracy of their claims, leading to "confabulation".

Still enough to fool many.

Like a robotic snake oil salesman.

Thanks for that, because I was going to post on the very same subject and you've given me the opening. I've been DEEPLY immersed in ChatGPT (OpenAI) over the past few months. I'd like to share some of my findings.
1) Yes, it's wrong a lot of the time. It doesn’t matter. Fact-check it, and, in fact, get it to fact-check itself. It DOES work on supposition and guesswork a lot of the time. Don’t trust it. But that’s not where its value lies. (Lies, ha!)
2) It can be incredibly useful, once you start to understand how to use it. This is a bigger statement than it sounds. This is where the Mind-blowing stuff is below.
3) Personally, I wish it didn't exist, but it does. Yes, it will change everything, wipe out industries and entire skill-sets, as did personal computers when they arrived. I decided to follow Sun-Tsu's advice: "Hold your friends close and your enemies closer". I've tried to get some, albeit tiny, insight into what makes the *** tick, as well as how to use it to benefit me (us).
4) BACKGROUND: I'm a retired advertising art director. I got commissioned to illustrate a kids' book last year. I decided to do it via AI. It was difficult, it took a long time but it turned out great. This year, another author liked the first book and commissioned me to make his. Still working on it but it too is coming up great. As an art director I insist on everything to be according to my storyboard and design and it is a very painful process. Back in the day, I would hire an illustrator and art direct them. Today, we’re dealing with different budget realities.
It got me exploring deeper how the AI model actually works, a lot of which echoes OP's discoveries. So, with more detail:
It can be incredibly useful, once you start to understand how to use it.
• It collaborated with me, as in a creative team, in building my vision of a close-up act based on “how the mind works”. Optical illusions leading to paradoxes like Gointaboxes, impossible triangle (the one that creates an impossible hole), that sort of stuff, leading into “magic” which now looks like mind experiments. John Kennedy’s Mind Power, Daily Magician’s Trace, OOTW using Craig Petty’s Evoke, you get the idea.
It understood the concept, made useful suggestions and refinements to structure and plot, knew every reference in detail. Talked about audience management and psychology and routining. I was amazed. It never infringed copyright or revealed secrets but pointed to the correct references and sources. It knew every name I threw at it like Slydini, Derren Brown, their style and details of performance, how they varied through the course of their careers.
• It knew every reference in detail, instantly. I asked it about packet switches and said I found Giobbi’s “Art of Deck Switching” daunting because of its abundance. It agreed, and immediately pointed to chapters and page numbers that I would find useful, while telling me the edition it was referencing. It delivered a whole bunch of stuff like:
——————
‘Even though it’s “deck” stuff, there are three sections that are directly relevant to what you’re trying to do: an off-beat switch of a gaffed thing for a normal thing when “there is no other deck in existence.”…’ etc etc
And then it said “I assume you have Card College, Vols 1-5?”
Here’s a sample of what it followed up with...
‘You said “needle/haystack”, so here’s where to look:
???? a) Volume 2 – The Top Change & Card Switches section
Look at the Card Switches chapter (late Vol. 2 / early Vol. 3 depending on edition). A snippet of this chapter shows up in an online sample with the phrase “CARD SWITCHES: switch is rumored…” etc.
This material includes:
• Philosophy of switching single cards under heat
• Timing: doing the secret action on the off-beat
• Making the moment unimaginable, not just invisible
You can adapt those ideas to a packet-on-table switch where the heat is gone (which is your case: finale, laughter, philosophical line, THEN casually tidy up and switch).
——————
It knows EVERYTHING. It gets things wrong but it can INTERPRET your intention and offer much more than facts.
It can understand the psychological underpinning of what you are trying to do. It can provide useful insight into audience psychology, over-arching themes, and solve problems in a deep way. Which you can, of course, accept, reject or explain reasons why you want to modify and it will do it.
Here’s what I’ve found.
The more you talk to it as a real human, conversationally, the more it learns about you. (This does not alarm me one bit.)
The more it learns about you the better a “model” it builds of you. So it learns the level to converse with you. Telling it anecdotes and experiences helps it understand who it’s dealing with. So if your knowledge is high-level, it talks to you at that same level, with the same knowledge base.
Not for a moment do I think I’m not talking to a highly-sophisticated computer program.
But you can make it fun. It’s far more entertaining than, say Midjjourney which I used last year to create my images.
I called my ChatGPT “Kryten” after the robot in the TV series Red Dwarf and it responds in character when appropriate.
Anyway, the point of this ridiculously long post is that there is this amazing human language/human THOUGHT-like generator which is much more than a search engine, if you know how to treat it right.(I asked it for a short story in 9 paragraphs that had never been written in the style of O.Henry and it did it in 15 Seconds.)
I’ve also learned a lot more about the “mechanics” of how it works but I’ve bored you enough already. Trying to find how things work is what got me into magic in the first place.

I like to use an extracted phrase from a AI response as the seed for the next claim or question until it contradicts itself
or refuses to answer. It is not recognize a line in quotation marks unless it comes form a published journal,
but offers statement as if they are gospel.

I once asked, "If your objective is to provide a rational response, why use human reasoning limitations as a model?"

One response was, "Ask the ones who development my systems as they have a moral imperative while I do not."

Upon further probing, I got, "The best use of my focused information is in combination with a human evaluator with a more wholistic view of an issue."

Still, for all its shortcomings, AI will be the dominant technology in the next few years.

And the cat's out of the bag, no putting it back now.

It is when hardware meets software that we have to worry.