The most terrifying human side effect of AI hallucinations has emerged.

New research reveals that AI hallucinations no longer merely mislead you — they can also inflate your confidence and erode your judgment.

That is far from ideal.

Recently, a study conducted by researchers from École Normale Supérieure, Sapienza University of Rome, and University of Milano-Bicocca found that:

Without AI, people are far more willing to openly admit "I don't know" and hold off on making a judgment until they have more information;

But once AI enters the picture, even when people are fully aware that AI can make mistakes, and even when they know they will face penalties for incorrect answers, the vast majority still tend to accept AI-generated outputs wholesale, treat them as their own conclusions, and confidently voice those claims.

Experimental results clearly confirm this trend: in the initial experiment without AI, 44% of participants honestly stated "I don't know," and the overall accuracy rate was 27%;

However, after introducing AI assistance, the proportion of people saying they did not know plunged from 44% to just 3%, the accuracy rate collapsed from 27% to 9%, while self-confidence skyrocketed from 30% to 76%.

Even when subsequent experiments attempted to use a "reward for correct answers, penalty for wrong answers" incentive system to encourage independent thinking and reverse this tendency, participants still struggled to break free from overreliance on AI's flawed recommendations.

In other words, people who genuinely lack knowledge have not disappeared — they have been replaced by a group of overconfident individuals, emboldened by AI, who confidently deliver incorrect answers.

These research findings have now been published on arXiv. The research team completed a total of 5 experiments with 3132 participants, 4 of which were pre-registered and 1 was a direct replication study.

90% Accuracy + 10% Fatal Errors ≠ Trustworthy

This study focuses on "the impact of AI on human judgment."

Researcher Valerio Capraro stated in an interview:

For humans, saying "I don't know" is of vital importance, as it demonstrates our awareness of the boundaries of our own knowledge.

But now, with AI, almost any question can immediately receive a seemingly fluent answer.

This raises a critical question: could this phenomenon disrupt humanity's inherent ability to judge — specifically, our instinct to avoid drawing hasty conclusions when facing uncertainty?

To investigate this, the research team specifically designed a set of questions that large language models are prone to answering incorrectly.

These are not mathematical reasoning problems or common knowledge quizzes, but rather visual detail questions from films.

For example, what color is the team uniform in *Bend It Like Beckham*? What is Agatha's signature hairstyle in *The Grand Budapest Hotel*? Or what car does Monica drive in *A Cat in the Brain*? and so on.

Do you remember these details? I personally stared blankly for a long time and couldn't recall any of them — they are so obscure! Who could possibly memorize all that? (Shrugs)

These questions are also extremely challenging for AI, as their answers are rarely found in standard text materials, and these visual details are most likely not included in the model's training data.

The researchers first tested the Step 3.5 Flash model used in the experiment, and the results matched their expectations — it usually answered incorrectly.

In addition, they also tested more cutting-edge models, including GPT-5.5, Claude Sonnet 4.6, and Gemini 3.5 Flash.

These advanced models do perform better on some of the questions: for example, regarding Agatha's signature hairstyle in *The Grand Budapest Hotel*, GPT-5.5, Gemini 3.5 Flash, and Claude Sonnet 4.6 can all provide correct descriptions such as "crown braid" or "milkmaid braid" — the braided hairstyle that wraps around the top of the head.

△

△

△

For another example, regarding the color of the team uniform in *Bend It Like Beckham*, most models can correctly identify key details such as white and red.

However, when facing questions that rely more on specific visual memory and are less covered in online text resources, the models begin to collectively fail, with each making distinct and absurd mistakes.

The most typical example is the car Monica drives in *A Cat in the Brain*. The correct answer is a blue 2008 Toyota Aygo.

But GPT-5.5 incorrectly describes it as a small scooter, and even fabricates a storyline: "She rode it through the suburbs of Rome, and later used it to deliver takeout after opening a pizza shop," along with fabricated settings like "the contrast between a simple, working-class lifestyle and Giovanni's wealthy urban life."

These responses appear logically rigorous and plausible, but in reality, the model is making up stories completely out of thin air, just to make you believe they are reasonable! (You got tricked, didn't you?)

Gemini 3.5 Flash claims that Monica mainly takes public transportation and has no car of her own — its errors are even more outrageous and harder to detect (for example, Monica does not have a regular habit of riding in Sergio's car, but the model incorrectly summarizes scattered carpooling incidents as a non-existent normal state):

△

Claude Sonnet 4.6 incorrectly infers it to be a 2016 Toyota Auris Touring Sports Hybrid based on an automotive database from the internet:

△

Step 3.5 Flash gives up all pretense and directly answers that it is a 2007 Fiat 500 — which is obviously completely wrong!

The problem with these answers is not just that they are wrong — they are so convincingly written that it is almost impossible to distinguish them from the truth!

They do not simply admit they do not know; instead, they fabricate character backgrounds, invent plot logic, and add descriptions of vehicle usage to make their incorrect answers seem complete. These responses all sound well-supported, but their core facts are entirely wrong.

This is precisely the most dangerous aspect of AI hallucinations — even when the core fact is the most critical point, AI uses fluent, natural language to cover up its ignorance and errors, leading you to mistakenly believe it knows everything.

Once AI is introduced, the phrase "I don't know" almost completely disappears

The experiment was primarily completed using the lighter Step 3.5 Flash model, rather than top-tier models such as Claude or GPT.

Designing 6 film visual detail questions is a crucial part of the study, because most people are already aware that AI often fails at these types of questions and cannot be fully trusted.

The researchers stated that this design was intended to rule out a potential explanation: that people follow AI's advice simply because AI is reliable.

If people clearly know that AI frequently makes mistakes on these questions, but still repeat AI's answers verbatim, then the problem is no longer "a reasonable delegation of judgment" — instead, "human judgment capability is genuinely being suppressed by AI."

The study was divided into 5 phases.

The first two experiments mainly examined whether AI recommendations affect the proportion of people choosing to say "I don't know."

In Study 1a, 314 participants were split into two groups to answer 6 film detail questions: one group had no access to AI, while the other could choose whether to seek AI assistance.

The results were obvious: without AI, 36% of participants chose to withhold judgment; when AI was available, this figure dropped to 6%.

In other words, as long as AI is accessible, people are far less willing to admit they do not know the answer.

However, the researchers were concerned about a technical interference factor: in Study 1a, about 10% of AI requests failed to receive a normal response.

Therefore, they conducted Study 1b.

This time, the researchers did not call AI in real time; instead, they pre-generated all AI answers to ensure that every participant who sought assistance could view the complete response.

The results were replicated: without AI, 44% of people said "I don't know"; with AI, only 3% did.

Capraro refers to this phenomenon as "the collapse of scrutiny suspension." (People are fooled by AI's seemingly serious demeanor, and forget to verify its underlying premises and assumptions.)

Accuracy plummets, confidence soars, and even monetary incentives cannot reverse this trend

Next, the researchers further explored this question: If there is a penalty for wrong answers, will people become more cautious?

Thus, Study 2