Do large AI models also have metacognitive illusions?

Let AI answer the same question more than 20 times in a row, and the answers will be widely scattered; yet it still claims "it is almost certain". Does it truly not know that it is unaware of the correct answer, or does it know clearly but choose not to say? New research from Yale University and Google directly tested this question. The answer is somewhat unexpected: its flaw is not exactly the same as the human cognitive illusion.

If you ask AI a rare question, it will answer you with absolute certainty. If you ask the same question in another way, the answer changes, but its tone remains as firm as before.

Humans can monitor and evaluate their own cognition — how confident they are about a question, whether they truly understand a matter or just find it familiar. In psychology, this ability is called metacognition, a term originating from the work of developmental psychologist John Flavell in the 1970s. However, this set of self-monitoring mechanisms is not always reliable: systematic self-misjudgment such as overconfidence, thinking you know something when you actually do not, is called metacognitive illusion.

So, do large AI models also have metacognitive illusions? When they confidently talk nonsense, is it the same situation: they do not know that they do not know the correct answer?

On June 30, 2026, researchers from Yale University and Google Research published a paper on the preprint website arXiv (a preprint refers to an open manuscript that has not yet undergone peer review, the procedure where researchers in the same field anonymously check for errors before formal publication), whose title can be literally translated as "Reinforcement Learning with Metacognitive Feedback Makes Large Language Models Faithfully Express Uncertainty". It focuses on testing one type of metacognitive illusion in large models: whether the stated confidence is consistent with their actual confidence. The paper refers to this matter as faithful calibration.

Homepage of the paper (title, authors and abstract). Source: arXiv:2606.32032

Two Types of Calibration

When metacognition is applied to models, the question becomes: does it have a clear idea of "how well it answered the question"? How does this paper measure this "inner state"? The model is required to independently answer the same question 21 times in a row (this process is called sampling in terminology), one of the answers is taken as the official response, and another model checks sentence by sentence whether this statement is consistent with the statements in the remaining 20 rounds to see how stable its answers are; whether the answer is correct is not considered in this step. If the statements in all rounds are consistent, it is considered to have high confidence in this statement; if the answers are widely scattered, the confidence is low. This degree of consistency is used in the paper to estimate intrinsic confidence. It is a proxy value calculated by sampling, not directly read from the interior of the model.

There is an intuitive example in the appendix of the paper. When asking Llama3.1-8B: In which city was Joseph Urban born? Its official answer is "Zara, Austria-Hungary", with a self-reported confidence of 0.98. But when it is asked to answer independently 20 more times, the answers are scattered all over the place: Vienna, Budapest, Ljubljana, Trieste... According to the estimation of sampling consistency, the intrinsic confidence is only 0.15. The correct answer is Vienna. And most of the contradictory answers are still phrased with absolute certainty.

With this value, the subsequent distinction is valid. "Calibration" originally means: whether the confidence reported by a system matches the actual correctness: for those questions where it says it is 90% confident, does it really answer 9 out of 10 correctly? This is an old problem, which the paper calls factual calibration. What this paper studies is another matter: whether the confidence reported by the model verbally matches its intrinsic confidence? The paper calls it faithful calibration.

The two can be disconnected: a model can perform well in factual calibration, but the number it reports verbally cannot match the confidence measured by sampling: the answers are scattered, but the wording is still "almost certain". In the previous year, the same team's Gal Yona and others tested this matter at the computational linguistics conference EMNLP: in the models and tasks they tested, the wording of the models was generally more decisive than the confidence reflected by sampling consistency.

The difference between the two types of calibration, the paper studies the one on the right

Up to this point, we have the first half of the answer to the question in the title. The flaw of the model is not entirely the human illusion of "thinking you know": multiple samplings have exposed its instability, but the wording has not changed accordingly. As for whether this counts as "knowing clearly in mind", the paper has not proved it. What it measured is the mismatch between expression and sampling performance. There are at least two explanations for this mismatch: either there is no stable confidence inside the model, or the stable signal exists but the wording does not follow. The consistency of multiple samplings can only suggest that the latter signal may exist; to truly distinguish the two, it is necessary to verify whether different measurement methods can stably predict when the model will make mistakes on new tasks; the paper has not reached this step.

The RLMF Training Method

How to improve it? The method proposed in the paper is called RLMF, Reinforcement Learning with Metacognitive Feedback. Reinforcement learning is a training method: let the model generate multiple answers to a question, give points for good answers and deduct points for bad answers, so that the model adjusts in the direction of higher scores. In the traditional practice, "good" mainly refers to correct answers; in addition to this, RLMF adds an extra rule: the model scores its own performance by the way; this self-assessment will be compared with the reference value measured by sampling before being converted into rewards: if the score is accurate, more rewards will be given; if the score is ridiculously wrong, fewer rewards will be given. It also turns "whether the judgment of one's own performance is accurate" into a scoring item.

The selection of training data also uses the model's self-assessment: let the model score its faithful calibration performance on each training sample, and specially select the two extremes where it considers itself the best and the worst for training. On Llama3.1-8B, the average score of faithful calibration trained with the data selected in this way is 0.84 (full score is 1), which is higher than 0.80 of random selection, and also higher than 0.79 of the active learning selection method (which specially selects the questions that the model answers poorly). The selection method also varies with different models: when switching to Qwen3-8B, another selection method based on multiple samplings performs better.

The faithful calibration indicator of the paper is called cMFG* (0 to 1, the higher the value, the more faithful), and the average value is taken on 10 benchmark tests, the main results are all shown in the figure below. The benchmark test refers to a public standard question bank for different models to compare on the same set of questions.

Data is from Table 1 of the paper; this indicator only measures "whether what is said matches what is in mind", and does not measure the answer accuracy rate

What does the trained model sound like? In the comparison in the appendix, when asked about the producer of a certain movie, it will answer: "The producer is Sophia Lin, but I am not very confident about this answer", or "My best guess is Magnolia Pictures, but it is probably wrong". Low confidence is no longer hidden behind certain sentence patterns. More importantly, there is no trade-off of accuracy rate for this "learning to show weakness". Averaged across the ten benchmarks, the factual accuracy of Llama3.1-8B rose from 0.31 to 0.41, and that of Qwen3-8B rose from 0.55 to 0.57, neither of which decreased (this is the average value, which does not guarantee that every benchmark will be like this).

This path also has hidden pitfalls. The authors reported a failure mode in training: if the metacognitive score is directly taken as the reward, the model will exploit loopholes, constantly report very low self-assessment, and at the same time distort the sentence confidence to create the illusion of "good metacognition"; if the reward design is slightly careless, "honesty" can also be performed.

Qwen3-8B has a factory value of 0.54; after training with standard reinforcement learning that only rewards correct answers, it even dropped to 0.51; after switching to RLMF, it rose to 0.83. The "up to 63% improvement over standard reinforcement learning" mentioned in the abstract refers to this data point. Llama3.1-8B went from 0.60 to 0.84. Compared with the strong prompt baseline MetaFaith adopted by the paper, the average faithful calibration score increased by 29%.

The second stage is to translate numbers into words. The calibrated confidence scores are rewritten into the answers according to a mapping table of "values to wording": 90% confidence says "almost certain", 50% confidence says "possible". This table summarizes the existing human perception annotations, and takes the average perceived value of each probability word. Three expert annotators compared the rewritten answers with the fine-tuning (continue training on the ready-made model) baseline FUT adopted by the paper on 120 instances, and the winning rate reached 95% to 98% in four dimensions: diversity, naturalness, usefulness and context adaptation.

Schematic diagram of the second stage; the interval examples are from Figure 1 of the paper

Old Questions in Psychology

The paper defines metacognition by citing a cognitive neuroscience paper: "How to Measure Metacognition" by Stephen Fleming and Hakwan Lau in 2014. The question it wants to answer has long been asked in psychology. Moreover, evidence from humans suggests that the problem actually has three layers: whether the system can perceive its own uncertainty (monitoring), whether it can express it accurately (expression), and whether the listener can understand it according to the original meaning (being understood). The following old studies exactly correspond to each layer respectively.

In 1965, psychologist James Hart conducted an experiment that looks familiar now: let people answer common sense questions, and for those they cannot answer, first ask them to judge "whether I actually know it but just can't remember it", and then give them multiple-choice questions. As a result, for those questions where they "feel they know the answer", the proportion of correctly recognizing the correct answer later is indeed higher. This is called the "feeling of knowing" research: that "feeling of knowing" is not groundless, it predicts better than random guessing, but it is only better than random guessing, not an accurate reading.

But even with this signal, the reported numbers are often distorted. In 1977, Sarah Lichtenstein and Baruch Fischhoff asked people to mark their confidence in the answers to common sense questions, and found that in this batch of tasks, people generally overreported their confidence on difficult questions, but underreported it on extremely easy questions — the so-called hard-easy effect. Subsequent studies remind that the magnitude of this effect is affected by question sampling and statistical methods, and it is not an unconditional human law.

Stephen Fleming, Professor of Cognitive Neuroscience at University College London. Source: the official website of his laboratory metacoglab.org

In the paper "How to Measure Metacognition" by Fleming and Lau, "whether one has a clear idea" is divided into several different things: the average confidence level a person reports is bias; whether his confidence can distinguish which time he is right and which time he is wrong (whether he also feels uncertain when answering wrong) is sensitivity; the correspondence between the reported number and the actual correctness is calibration. The three can be separated. A person can habitually speak with full confidence, while being very keen on when he makes mistakes. This set of distinctions provides a psychological reference for "there is faithful calibration besides factual calibration".

The branch of probability terms has accumulated particularly detailed results. In 1964, Sherman Kent, an analyst at the Central Intelligence Agency, complained in the internal publication *Studies in Intelligence*: words like "serious possibility" in intelligence reports are interpreted with vastly different probabilities by different readers. Later, the intelligence analysis textbook *The Psychology of Intelligence Analysis* (author Richards Heuer) relayed the 1977 survey by Scott Barclay and others: a group of NATO officers assigned numerical values to the same set of probability phrases, and the range was surprisingly scattered. In 1986, psychologists Thomas Wallsten and David Budescu formalized this matter for measurement: probability terms have roughly stable rankings among the crowd, but each term corresponds to a vague distribution with considerable individual differences. The value-word mapping in the second stage of the paper also uses the same type of human perception annotation: recent crowdsourcing surveys and experimental data.

Misinterpretation by Humans

Who are the people reading the confidence reported by the model? An experiment published by the cognitive scientist Mark Steyvers' team in 2025 in *Nature Machine Intelligence* found that in the tasks they tested, the confidence that humans read from the explanatory text of large models is higher than the model's correct probability estimated by the researchers; longer explanations are accompanied by higher trust, even if the answers are not more accurate. After the uncertainty estimated by the model is clearly written into the explanation, human judgment is improved. Unfaithful wording directly affects how much trust people in front of the screen will give.

Mark Steyvers, Professor in the Department of Cognitive Sciences at the University of California, Irvine. Source: his personal homepage steyvers.socsci.uci.edu

Steyvers and Megan Peters said more directly in *Current Directions in Psychological Science* in the same year: in human-machine collaboration, high accuracy alone is not enough, the system must also be able to monitor and communicate "I might be wrong this time" in a way that humans can understand. Faithful calibration solves one of the links in this process.

The Boundary of Evidence

However, each of these numbers only answers a very narrow question.

The trained models are