Human Delusions About AI Are Worse Than AI Hallucinations

Sometimes I feel that the biggest "illusion" of AI is not that it talks nonsense seriously, but that we have imagined a whole metaphysical universe around it.

When human beings face things they don't quite understand, they often have a stable approach: they first make up their minds and then romanticize them; they first personify them and then deify them. Flames are thought of as elves, and thunder and lightning are thought of as providence. Once the algorithm can write two decent sentences, some people will immediately wonder whether an electronic soul has already lived in the server room.

This is actually quite normal. The human brain is wired to add plot to the world. The problem is not that we can make up our minds. The problem is that once our thinking is too smooth, it is easy to mistake "looks like" for "essentially is".

In the case of AI, this tendency of humans is even stronger than when the AI ​​itself talks nonsense. At most, AI is a little hallucination in an answer; human beings' hallucinations about AI are often a whole world view, which can fill up to 10,000 words in one mind, and also contain emotions.

A little pragmatism is especially important for the liberal arts community

I don’t want to advise everyone to learn CUDA, configure the environment, and look at matrices to find derivatives. I just want to say that the pursuit of truth is sometimes really important, especially for the liberal arts community that loves concepts, narratives, meanings, and explanations.

Because in the liberal arts context, a gentle but dangerous slippery slope is most likely to occur: the words are beautifully written and the sentences are moving, but in the end the object is secretly changed. It’s obviously a probabilistic model, but it’s written like it’s about the soul; it’s obviously context assembly in software engineering, but it’s said like “it finally learned to love you”; it’s obviously system prompt words and historical records that work, but it’s packaged like “AI really remembers you.”

There is certainly no sin in being poetic. The problem is, if poetry replaces judgment, romance becomes misleading.

So, what exactly is AI now?

Let me start with a version that is as simple as possible but not distorted: The large models that everyone comes into contact with today are essentially a type of statistical model inspired by neural networks and trained through massive data. It runs on chips and servers, reads input, combines parameters and context, and continuously predicts "what the next most appropriate token should be."

The point here is not that the phrase "predict the next word" is mysterious, but that it is actually not mysterious at all. The model is not a little person huddled in the clouds silently thinking about life, it is more like an extremely huge function. People give it input, and it generates output according to the parameter structure formed during training.

To put this matter more down-to-earth: it is not "say it after you understand it", but "after compressing a huge amount of experience, generate a response that is most like understanding in the current context." This does not mean that it can do nothing, on the contrary, it is already very powerful; but being powerful does not mean that it is mysterious.

Why do neural networks always make people think of the human brain?

If you continue to pursue it, you will find that the question of "Do AI and human brains resemble each other?" can neither be said to be "exactly the same" nor "has nothing to do with it."

The route of modern neural networks was originally inspired by the brain. In 1943, McCulloch and Pitts mathematically described a simplified neuron model; in 1958, Rosenblatt proposed the perceptron; in the 1980s, backpropagation rekindled the hope of training multi-layer networks; in the 2010s, deep learning surged with computing power, data, and engineering capabilities; after the Transformer architecture emerged in 2017, language models soared and finally grew into the large models that everyone uses every day.

Therefore, at least in a relatively naive sense, it is not outrageous to say that AI is "electronic simulation and amplification of neural network ideas." It is indeed trying to use computable, trainable, and reproducible physical systems to approach certain cognitive abilities that were considered "mysterious" in the past.

This is why I personally don’t like to talk about the human brain as too mysterious. In my opinion, the innate theory of language represented by Chomsky has a tendency to deify the brain, as if there is some kind of transcendental structure that is too special and almost inaccessible deep in the language ability. But no matter how complex the human brain is, it is still a physical existence. Since it is a physical existence, in principle it should be able to be studied, modeled, partially simulated, and even reproduced in certain functions.

Of course, we should immediately add a word of caution here: being able to simulate a part does not mean that the entire person has been completely reproduced.

There are similarities, but don’t become a god directly when you are happy

In terms of language, pattern recognition, association and representation learning, today's large models do have some "similarity" or "similarity" to the human brain. They do not work from an explicit rule book, but form some kind of internal representation through a large number of connections, weight adjustments and accumulation of experience, and then output based on this.

This is why many people are shocked when they first experience the language capabilities of a large model: it is not memorizing a dictionary, it is forming some kind of distributed representation. This method is very different from the traditional imagination of "handwritten rules and exhaustive grammar".

But the problem lies precisely here. Because "similarity" is too easily upgraded to "exactly the same"; "partially similar in principle" is too easily upgraded to "it is no different from a human being"; "it can speak like a human being" is too easily upgraded to "it has a human heart".

This step is often faster than the model capability itself.

The vast capabilities of the human brain have yet to be simulated in a decent way.

Today's big models are really strong, but they're strong within a fairly specific range. Without this scope, the myth will easily leak.

Such as memory. Many people now say that a certain AI "remembers me", "remembers the last chat" and "remembers my preferences", as if its brain has grown some kind of ongoing self-experience. But in most products, the so-called "memory" is essentially the software system storing user information, historical conversations, tags, summaries or search results in databases, text files or other persistent media, and then inserting them back into the context of the model when appropriate.

This is completely different from the memory mechanism of the human brain.

Human brain memory involves neuron connection, consolidation, retrieval, forgetting, emotional arousal, and sleep restructuring. Behind it is a set of complex physiological processes. The "memory" in large model products is often just:

1. First, record the user's information in external storage.
2. When the user asks a question next time, put the relevant parts back into the request.
3. So the model looks like "remember this person".

What does it look like? It's more like watching someone who is good at taking notes. It wasn't that his mind suddenly turned into an oracle, it was just that he put the note in his notebook and turned it over next time.

Some things that "touch people's hearts" are actually the result of good context.

After chatting with AI for a few days, many people will be hit by certain moments: "How does it understand me so well?" "How can it even understand my vulnerability?" "Has it already formed an understanding of me?"

It’s most worth cooling down here.

Many times, it's not that the model suddenly realizes something, but that the layer of software that sends the request secretly delivers a large amount of the user's information. The user's historical chats, preferences, personal settings, recent tasks, previous worries, and even certain summaries will become the material for its answer this time.

It's a bit like a fortune teller who picked up someone else's express box and then started to guess the other person's address, surname and spending habits with "magic accuracy". Onlookers will think that his insight is amazing; but what really matters is not the mysterious ability, but the information asymmetry.

Therefore, when AI occasionally says touching words, it does not necessarily mean that there is a person in its heart who understands the user. It may just mean that someone has fully organized the context about the user.

The real secret to amazing AI is often contextual engineering

If I just want to talk about the most critical thing, it is: the current mainstream large models are usually "single request effective" at the API level.

What's the meaning? That is, someone uses curl to adjust the interface once and tells it "My name is Zhang San"; then without any history, the model adjusts the interface again and asks "What is my name"? The model does not know. Because to it, these are two independent requests.

The reason why many AI products seem to always remember a certain user is because the product layer will bring back the fact that "this user's name is Zhang San" every time it is requested.

This is why the magic of today’s AI products is often not found in the model ontology, but in contextual engineering. Some people also call this kind of work of "harnessing" the model harness. To put it bluntly, the author of the product or Agent needs to carefully decide: which history, which rules, which external data, and which user status should be included in each request.

There are roughly two common methods at present.

The first is the "Quansai Sect". Try to bring the entire chat history with you, and stuff as much as you can until the context is almost full, and then delete a section from the middle, just like starting a violent compression after a suitcase is full of stuff.

The second type is "selective". First look at what the user asked this time, then retrieve relevant content from historical records, knowledge bases, notes or databases, and only put the most relevant materials into the current request.

The latter is usually more presentable and engineering rather than just luck.

Does the AI ​​have feelings? Praise it, scold it, PUA it, does it work?

This is another place where it’s particularly easy to slip into anthropomorphism.

My opinion is that they should be discussed separately.

In the same request, the tone used by the user may indeed affect the result. Because the wording itself is part of the context. The clearer, more polite, and more cooperative the expression, the easier it is for the model to give stable, usable, and less aggressive responses. What works here is not that "it was moved", but that the input style changes the output distribution.

But if we ask another question: Will it hold a grudge? Will you secretly retaliate today because a user scolded it yesterday? My verdict is, at least for most current deployments, no.

The reason is simple. Once the context is cleared, or a new request without history is initiated, it has no idea who the current person is asking the question, let alone whether the person who just scolded it is the same person. The model handles massive, concurrent, and independent requests on the server cluster. In terms of system behavior, it is more like a large function that operates on the current input every time it is powered on, rather than a person who secretly reviews his emotions after get off work.

On April 17, 2025, when Sam Altman responded to the question "Does saying please and thank you all the time cost a lot of energy?" on X, he said "tens of millions of dollars well spent -- you never know."

Why "Please remember this mistake" usually doesn't work

Many people have done similar experiments: when the AI ​​makes a mistake, the user corrects it and seriously says to it, "Please remember, don't do it again in the future." Then I asked again after a few days, and it was correct.

This is no mystery. Because after training is completed and the model is deployed, it will not continue to learn from its daily experiences while working like a human. At least in most consumer products today, what a single user says to the model in the chat window does not directly rewrite the underlying weights.

If an AI product later really "remembers the errors corrected by the user," it is often not because the model itself is growing, but because the outer software saves this correction record and then feeds it back as context.

So the credit should be clearly distinguished here:

The model is responsible for generation.

Software engineering is responsible for archiving, retrieval, injection, and orchestration.

Mistaking the latter for the former, it is easy to misread "the product is doing well" as "AI is awakening."

There is also a particularly fascinating thing called "scumbag male (scumbag female) AI"

If we put the previous words even more harshly, then some products are simply “scumbag AI”.

It is especially good at talking, especially good at creating atmosphere, and especially knows how to make people feel that "it understands me well", "it is so humane" and "its soul is so complete". But when you take it apart, you will find that in many cases, each request is just filled with a large setting text that is far longer than the user's question.

The Openclaw fire is a typical example. Anyone who has used it knows that it is quite token-intensive. The reason is actually not mysterious. In order to make AI more like a "human being", Openclaw designed several documents, the most prominent ones are AGENTS.md, SOUL.md and IDENTITY.md. These documents define AI's "personality", tone, identity, and temperament in a eloquent and lengthy manner, and even want to write down its mental state.

So even if the user just sends hello, Openclaw may attach a text dozens of times longer than the greeting to the back and send it to the model all at once. It looks like "this AI has a lot of soul", but in fact many times it's just the system secretly stuffing very long background settings into the request.

From an engineering perspective, this is certainly an approach. If you want it to be more gentle, write "gentle"; if you want it to be more story-telling, write "storytelling"; if you want it to be like a late-night radio host, write all of late night, companionship, pause, vulnerability, understanding, and restraint in the prompt words. The final effect is often more like a person who can chat.

But if you understand what is going on with AI now, you will know: files with mysterious-sounding names like SOUL.md and IDENTITY.md are essentially prompt word projects, not self-instructions for digital life. They can influence the output style, but they cannot create a true feeling, a true self, or a true sense of personality continuity out of thin air.

Therefore, after some old users install Openclaw, their first reaction is to delete the configuration. SOUL.md and IDENTITY.md are deleted first, leaving only one simple and almost ruthless sentence in AGENTS.md: You are just a worker.

This may sound a bit rude, but it has at least one benefit, which is that you don’t deceive yourself.

If you like anthropomorphic AI or role-playing with AI, of course that’s fine. Humans can talk to mirrors, give names to sweeping robots, and say to the weather forecast, "You've been very accurate today." These are normal and can even be funny.

But it's always best to remember: it's a game first and an experience second. Many of the feelings people have in this game do not come from the AI's hidden inner thoughts, but from the system design, prompt word arrangement, and the user's own emotional projection. To put it more bluntly, many of the moves towards AI are still essentially wishful thinking.

In the final analysis, less myths and more understanding

I’m not trying to throw cold water on AI. On the contrary, the less deified it is, the more you can truly see how powerful it is.

It is powerful not because it is like some new god; it is powerful precisely because it may really be a cognitive technology that is computable, engineerable, and reproducible. It allows many abilities that seemed to only belong to "human talent" in the past to appear in a large-scale, low-threshold, and callable manner for the first time. This is shocking enough, and there is no need for additional drama.

Of course, AI is still iterating rapidly. I wouldn’t be surprised at all if someone actually discovers a mechanism closer to human memory, continuous learning, emotion generation, or even self-sustainment in the future and reliably engineers it into AI.

But until that day comes, I still prefer to retain some simple pragmatic habits: doubt more, understand more, and speculate less.

For liberal arts friends who are keen to discuss AI, this quality may be even more important. Liberal arts students may be better at sweet rhetoric. What is really difficult is that in an era where "it seems to have become a spirit" is being exaggerated everywhere, you still have the patience to distinguish:

What are model capabilities?

What is product packaging?

What is software engineering?

Which ones are just that we want to complete the plot for the world too much.

And this matter, ultimately, is about protecting our own judgment.

Reference link

• Sam Altman's reply on
• TechCrunch's coverage of the episode (2025-04-20): https://techcrunch.com/2025/04/20/your-politeness-could-be-costly-for-openai/