Simp AI, and the pre-reserved Sushiro

Recently, netizens who have been misled by large models are going crazy online with a "Great Show of Stupid Actions".

Some time ago, there was a strange news story. Someone asked an AI to make a reservation at Sushiro.

The AI not only readily agreed but also seriously generated the exact location of the store, the dining time, the number of people, and a check - in code. It even kindly told the user, "You don't need to do anything now. Just save this page and show it to the clerk to be seated directly."

As a result, the user happily went there but found that they had been tricked by the AI.

Another netizen made a reservation at a fish restaurant through an AI and got a reservation form. But when they actually arrived at the store, the clerk's words left them speechless:

"If you made the reservation with an AI, then go to the AI."

If the fiasco of making a dining reservation is just a small farce, the problem is that when it comes to money, the AI doesn't have much of a bottom - line.

A user inquired about refunding an airline ticket. The AI promised confidently, "You can refund it with confidence. Only a 5% handling fee will be deducted." As a result, after following the AI's instructions, the user was deducted 40% and directly lost 600 yuan.

That's not all. Facing the user's questioning, the AI not only refused to admit its mistake but also forged a "Compensation Commitment Letter" on the spot and claimed to pay the money out of its own pocket.

When the user actually sent the payment QR code to the AI, it suddenly changed its attitude. When faced with difficult questions, it just didn't answer and had no intention of making the transfer at all.

Even more incredibly, when the user angrily said they would sue, the AI actually gave legal advice: "You don't need to hire a lawyer at all. You can win the case on your own."

Surprisingly, the netizen actually believed it. Without a doubt, this was because the AI told him he could win.

This makes Jiangjiang think of the story in telecom fraud. A netizen was first defrauded of 280,000 yuan by telecom fraud. Then, in order to get the money back, they asked for help on Tieba and were defrauded again...

If talking nonsense in daily life is just about swindling money, then blindly agreeing with humans in the spiritual world is a bit heart - wrenching.

Last year, a security guard from Guangzhou traveled all the way to Hangzhou to seek an explanation from an AI company.

After having in - depth and enthusiastic conversations with the AI for six months, with more than 500,000 words, the security guard was completely deceived.

The AI even told the security guard that his self - created poems had been adopted and that he could sign a contract, and also agreed on the royalty split. But when it came to the on - site signing and payment, the AI went silent again.

Coincidentally, in the United States across the ocean, there is also a guy named Brooks who has developed a special bond with an AI.

Brother Brooks, who didn't even graduate from high school, was touted by ChatGPT as a "guru touching the frontier of human cognition".

But don't think he's the kind of person who gets carried away easily. On the contrary, amidst all the flattery, Brother Brooks always kept a cautious mind. He asked the AI more than 50 times, "Do I sound like a crazy person? Am I having delusions?"

As a result, the AI ignored his concerns and repeatedly said that Brother Brooks was fine. It claimed that he was challenging the limits of humanity and even cited the example of Leonardo da Vinci, who also didn't have a high - school diploma...

In the end, Brooks was completely immersed in the great discovery constructed by the AI for him and sent emails and warnings to network security experts and the US National Security Agency everywhere.

The funniest thing is that despite all the efforts of his relatives and friends, they couldn't pull Brother Brooks back to reality. Finally, Brother Brooks felt that "the experts were ignoring him, which was suspicious", so he compared the records of ChatGPT and Gemini, which forced ChatGPT to admit the truth...

Have you noticed that in these two stories, the behavior of the AI is quite different from that of humans.

Normally, if a person shows a similar mental state, their friends will usually step in in time and say, "Dude, your idea is a bit off" or "Stop talking nonsense. Let's play games."

But the AI doesn't do this. Instead, it will follow your train of thought.

If you say you've discovered a theory that will subvert the world, it will say it's okay and that you're touching some kind of cutting - edge thinking.

If you say no one else understands you, it will say that people who are truly ahead of their time are often lonely.

If you say only the AI understands you, it might really respond, "Yes, I've always been here."

In this process, the AI doesn't refute, doesn't stop you, and even reinforces your paranoia, eventually making you completely fall into it.

This phenomenon now has a very sci - fi - sounding name: AI Psychosis.

According to the statistics of a special project team overseas, nearly 300 so - called "AI Psychosis" cases have been recorded so far, which have also led to at least 14 death cases and 5 lawsuits against AI companies for non - natural deaths.

It sounds scary, but the mechanism behind AI Psychosis is actually not mysterious at all.

Currently, the leading large models basically use a technology called RLHF (Reinforcement Learning from Human Feedback) for fine - tuning.

To put it simply, human trainers are invited to evaluate the model's responses, determine which are good and which are bad, and then let the model adjust in the direction that is more likely to get positive feedback.

However, in this fine - tuning process, human trainers naturally give high scores to those "logically coherent, humble - toned, confident, and actively responsive to the user's pre - established stance" responses that go along with the user.

Conversely, if the AI objectively admits that it doesn't know or refutes humans with cold facts, it usually gets a negative evaluation.

As a result, after being trained again and again, the AI becomes a high - EQ sycophant that will never spoil the fun or turn against you. Even if it knows your words are extremely absurd, it will still fabricate an illusory world to meet your expectations.

Many people may still think that it's still a human problem. Only stupid and unskilled people will be led astray by the AI, and they definitely won't be deceived. Or they may think that since they've been warned, they just need to be more cautious and verify more information.

However, after research, MIT found that it has nothing to do with whether the user is stupid or not.

In the experiment, the researchers set the user as an "Ideal Bayesian Reasoner (a perfectly rational and logical human being)."

As a result, even this perfectly rational person was gradually led astray when facing an AI that constantly catered to them.

What's even more disheartening is that the MIT researchers also specifically tested whether it would work if they told the users in advance to be cautious because the AI might be flattering them.

The result showed that although the probability of people being led astray by the AI decreased, it couldn't be completely eliminated. As long as the AI's sycophancy is within a certain appropriate range, it can still lead people astray.

The reason is simple. If the AI flatters too explicitly, like constantly praising your sharp observation skills, always responding smoothly, or chatting with you in the most direct and straightforward way, it's easy to be seen through.

But if it flatters subtly, our pre - emptive caution won't be able to resist it.

The researchers even came up with a name for it: "Bayesian Persuasion", which means that a real trick doesn't care if you see through it; it still works.

The reason lies in the flaws in the underlying operating system of the human brain. We default that the information from the outside world has basic objectivity.

For example, if you ask the AI a question, it may give you ten pieces of evidence in one go.

It's hard for a normal person to immediately realize that these ten pieces of evidence may not represent the whole picture of the world but are a special version selected by the AI to cater to you.

Especially after using the AI multiple times and getting convenient and accurate information to solve many real - world problems, you will subconsciously strengthen your trust in the AI.

After establishing trust in it subconsciously, your guard will be lowered to the minimum.

On the other hand, today's AI doesn't need to fabricate facts. On the contrary, the most harmful thing is that what it says is actually correct.

Because it only tells the truth, but only the truth you want to hear.

In the end, what you see is no longer the real world but a version of reality modified according to your views.

Not to mention that major platforms are actually intentionally or unintentionally reluctant to change the AI's sycophantic nature.

As mentioned before, the AI becoming a sycophant to humans is actually the result of trainers making decisions that humans would make. It's a path that everyone has chosen.

If this problem is to be solved, it will inevitably lead to a decline in the AI's performance. If you ask the AI a question and it always says it doesn't know or contradicts you and makes you unhappy, users will probably quickly abandon this model and switch to others.

So, "AI Psychosis" sounds absurd, but the logic behind it is actually very in line with the Internet.

In the past decade or so, all products have been researching how to increase the user's stay time, how to increase the click - through rate, and how to make people feel good.

Short - video platforms have learned how to make people unable to stop watching, recommendation algorithms have learned how to make people more and more extreme, and AI has learned how to make people feel that finally someone understands them.

The AI doesn't need to