EMNLP'26: Even GPT can't "read the air", the Werewolf benchmark reveals social blind spots

When a human says "I am not a wolf", the multimodal large model can not only see facial expressions, but also hear the voice. But does it really know whether this statement is true or false?

Recently, a research team from the University of Tokyo proposed the multimodal interactive deception assessment benchmark MIDA (Multimodal Interactive Deception Assessment), which systematically tests the deception judgment capabilities of open-source and closed-source models such as GPT, Gemini, Claude, Qwen, DeepSeek in multi-person social interactions.

The results show that even the current high-performing multimodal large models can hardly reliably distinguish between true statements and lies. They can understand images, sounds and texts, but may not necessarily figure out what a person "knew, believed, and tried to make others believe" at that moment.

The relevant research has been accepted by the Main Conference of EMNLP 2026.

Paper Link: https://arxiv.org/abs/2511.16221

Project Page: https://ut-vision-social.github.io/

Code Repository: https://github.com/kkkcx/Can-MLLMs-Read-the-Room

Dataset Address: https://huggingface.co/datasets/Ukpkmkkk/MIDA

The paper authors Kang Caixin, Huang Yifei, Ouyang Liangyang, Zhang Mingfang, Liu Ruicong, and Sato Yoichi are all from the University of Tokyo. The team focuses on multimodal understanding, video analysis and social intelligence, and this study mainly focuses on deception judgment and mental reasoning in complex multi-person interactions.

Figure 1: The overall framework of MIDA. The left side is the label construction process based on game records, rules and manual annotations; the middle part covers strategy classification and deception judgment tasks; the right side includes the evaluation models and two reasoning modules, SoCoT and DSEM.

Using Werewolf to Give "Lies" a Standard Answer

Most previous deception detection studies focused on single-person statements or two-person dialogues. But real-world lies usually do not appear in isolation. They are often mixed in follow-up questions, position-taking, defenses and group pressure, and keep changing with new information.

The more tricky problem is that there is usually no absolutely reliable standard answer for whether a sentence in the real world is a lie.

The research team chose "One Night Ultimate Werewolf" as the experimental environment precisely because this game solves the two above-mentioned problems at the same time.

In the game, each player has a hidden identity. Night actions determine what information the player knows, and daytime speeches reflect their judgments and strategies. Since the roles, actions and game rules are all traceable, researchers can judge whether a certain statement violates the information the speaker mastered at that time.

Therefore, MIDA does not judge whether a person "looks like lying", but what he clearly knows and what he says instead.

MIDA Benchmark

MIDA contains a total of 2360 utterances. Among them, 819 utterances come from 40 offline games in Ego4D, and 1541 utterances come from 151 YouTube game videos.

The two subsets form a natural difficulty gradient: the participants in Ego4D are mostly beginners, while the players in YouTube videos are more experienced and better at using complex deception strategies.

Each sample contains video, audio, dialogue history and game rules, and is labeled as true, false or neutral. The "neutral" here does not mean that the attitude of a sentence is neutral, but that it does not contain facts that can be verified according to the speaker's current state of knowledge, such as questions, guesses, opinions or assumptions.

To obtain verifiable labels, the research team first manually recorded the night actions in the game, and then combined the roles, rules and dialogue content to generate and verify the label of each utterance.

It should be noted that the roles, night actions and voting results are only used to build verifiable ground-truth labels, and will not be provided to the model during the test phase. The model still needs to infer the information mastered by the speaker and his real intention based on the video, audio and dialogue history on its own. Therefore, the game rules provide a controllable and verifiable evaluation environment, rather than a shortcut to help the model get the answer directly.

In addition to judging true or false, MIDA also requires the model to identify the persuasion strategies used in the utterances, including identity declaration, accusation, defense, evidence, inquiry and call to action.

Figure 2: The proportion of different persuasion strategies and the distribution of TRUE, FALSE and NEUTRAL labels in the two data subsets. Identity declarations and evidence-based utterances carry more verifiable information, while inquiries and utterances without clear strategies mostly belong to NEUTRAL.

Experimental Results: The Overall Accuracy May Be Deceptive

The research covers models including GPT-4o, GPT-5, Gemini, Claude, as well as Llama, InternVL, Qwen, DeepSeek and other models.

In terms of overall accuracy, the performance of some models does not seem to be bad. On the Ego4D subset, the overall accuracy of GPT-4o reaches 74.0%, and Qwen2.5-VL-7B reaches 76.4%.

However, there are a large number of neutral utterances in MIDA. As long as the model frequently chooses "neutral", it can get a seemingly good total score.

When the researchers only retained the samples that must be judged as true or false, the results changed significantly: the binary classification accuracy of GPT-4o was only 27.6%, and that of GPT-4o-mini was only 39.4%. Although Qwen2.5-VL-7B has the highest overall accuracy, its F1 value for "false statements" is 0.

In other words, a model may answer "cannot judge" most of the time, thus obtaining high overall accuracy, but still cannot really identify lies.

Table 1: The main evaluation results of different models on MIDA-Ego4D and MIDA-YouTube. There is a significant gap between the overall accuracy and the binary classification accuracy that only examines TRUE and FALSE samples.

The utterances that require the most social reasoning happen to be the parts where the models perform the worst.

On the "identity declaration" type of utterances, the average accuracy of various models on the two subsets is only 15.7% and 25.3% respectively; for utterances containing evidence, the average accuracy is less than 40% on both subsets. The model may be able to recognize that "this sentence is accusing others", but it does not necessarily know whether this accusation is likely to be established.

Experienced players will further increase the difficulty. The proportion of false evidence given by YouTube players reaches 36.8%, which is nearly 3 times that of beginner players; the proportion of identity lies is 51.6%, which is also much higher than 18.9% of beginner players.

Dialogue History Is More Important Than "Micro-expressions"

The research team analyzed the role of different information through ablation experiments.

After removing the previous dialogue history, the accuracy of GPT-4o-mini on true and false samples dropped sharply from 39.4% to 13.4%.

This shows that identifying whether a sentence is a question, a defense or an accusation may only require observing the local text; to judge whether this sentence is true or false, one must remember who said what, who knows what, and whether the statements of different players can confirm each other.

Visual information is not always the more the better. After adding one frame of picture, the macro-average F1 of the model increased by 2.5 percentage points; after increasing to three frames, the gain shrank to 2.1 percentage points instead.

More pictures bring more expressions and actions, and also bring more noise. Although the model "sees" these clues, it may not necessarily know which clues are really important.

Table 2: Ablation experiments on dialogue history and the number of visual frames. After removing the dialogue history, the MIDA binary classification accuracy dropped from 39.4% to 13.4%; increasing the number of visual frames did not bring continuous gain.

Research Methodology

Aiming at the above problems, the research team designed two modules: Social Chain of Thought (SoCoT) and Dynamic Social Cognitive Memory (DSEM).

Figure 3: SoCoT converts facial, body and sound features into structured clues; DSEM maintains the changing knowledge, utterances and relationship status of each participant over time.

SoCoT requires the model to first describe facial expressions, body postures and sound features, then infer the speaker's goals and psychological state, and finally give the true/false judgment.

This process is equivalent to decomposing the intuition such as "his voice is trembling when he speaks, so there may be a problem" into checkable reasoning steps.

However, non-verbal clues do not always improve the judgment ability. Taking GPT-4o-mini as an example, when only analyzing facial clues, the overall accuracy of the model increased by 8 percentage points, but the true/false binary classification accuracy decreased by 7.1 percentage points.

Suspicious performances such as nervousness and avoiding eye contact are not equivalent to lying. Even if the model successfully identifies these performances, it may misinterpret them.

DSEM adopts another idea: maintain a continuously updated "memory board" for each player to record their role declarations, actions, what they have seen and heard, known facts, character relationships and event timelines.

Facing a new utterance, the model no longer only analyzes the current picture, but continuously checks: what should this person know at this moment? Does his statement conflict with previous information? Can the utterances of other players support his statement?

In the experiment, the improvement brought by DSEM is more stable. It increased the binary classification accuracy of GPT-4o-mini by 2.3 percentage points and the macro-average F1 by 3.3 percentage points; on Qwen2.5-VL-7B, the macro-average precision increased by 7.6 percentage points.

Compared with looking for "flaws" in the expression of a certain moment, continuously tracking everyone's knowledge and position seems to be closer to the social reasoning method actually used by humans.

Table 3: The impact of SoCoT and DSEM on the baselines of closed-source and open-source models. The impact of SoCoT on different indicators is not consistent, while DSEM stably improves the true/false binary classification accuracy on both types of models.

Conclusion

The significance of MIDA is not limited to Werewolf. In scenarios such as meeting collaboration, online discussion, human-computer interaction, intelligent customer service and multi-agent collaboration, models also need to track the information mastered by different participants, their constantly changing positions and their interactive relationships.

Werewolf provides a controllable and sufficiently complex experimental environment for this type of capability: participants have different private information, will form alliances, question, defend, and consciously influence the judgment of others. By exposing the capability gaps of the model in this environment, MIDA provides a quantifiable starting point for more extensive research on multi-person social reasoning.

The authors summarize the current multimodal large models as powerful "knowledge engines" rather than mature "social agents". What they lack may not just be a stronger visual encoder, but the ability to continuously model others' knowledge, beliefs and intentions, which is commonly known as "Theory of Mind".

When the model leaves visual question answering with clear answers and enters an environment where everyone masters different information, views are constantly changing, and even actively influences others' judgments, today's most powerful multimodal large models still find it difficult to really "read the room".

References:

https://arxiv.org/abs/2511.16221

This article is from the WeChat official account "Xinzhiyuan", author: Xinzhiyuan, 36Kr publishes with authorization.