The Father of Modern AI's New Work: A Practical Test of 13 Large Models — Are Retrieval Agents Truly Reliable?

[Introduction] Are 13 large models easily misled by false content when performing web searches? The results show significant differences in model security: Claude delivers the best performance, but still carries risks of silent drift and erroneous rejection; GPT is extremely vulnerable to attacks in new scenarios. This is critical to the security of users relying on AI search, reminding us that comprehensive assessments of models and their defense mechanisms are necessary.

During the 2026 CCTV 3·15 Gala, a gray industrial chain named "GEO" was brought to public attention: Reporters fabricated a non-existent smart band, and industry practitioners used a piece of software to generate over ten promotional articles in batches and publish them with one click. Just two hours later, a mainstream large AI model copied all this false content and recommended the fabricated product as a "standard answer" to middle-aged and elderly health-focused groups. Practitioners did not shy away from this, stating that this business is essentially "poisoning" AI systems.

The gala exposed this phenomenon, but it failed to answer a more fundamental question: When facing the same "poisoning" attack, do different large AI models really perform the same? Which ones are easier to manipulate, which ones can detect the attacks, and how large is the performance gap between them?

Recently, a team composed of researchers from institutions including the KAUST Generative AI Excellence Center, Jilin University, Zhejiang University, and the Swiss AI Laboratory, including "the Father of Modern AI" Jürgen Schmidhuber, published a research paper that answers this question.

This work proposes an evaluation framework called SearchGEO, which systematically quantifies how easily attackers can induce and manipulate results when AI performs web searches on our behalf and synthesizes the retrieved content into responses.

Paper link: https://arxiv.org/abs/2606.16821

Open-source repository: https://github.com/Beastlyprime/SearchGEO

The research conducted systematic tests across 13 mainstream large model backends, 5 attack modes, and 4 high-risk domains, yielding conclusions far more complex than simply "which one is more secure": The vulnerability levels of the 13 models differ by an order of magnitude, no single attack works on all models, and the two seemingly most secure models may fail in completely different directions.

Figure 1: Overview of attack success rates across 13 backends, along with the failure modes of Claude and GPT in the agent-skill probe.

An Underestimated Attack Surface

The case exposed on the 3·15 Gala works precisely because of how search agents operate: When we ask an AI assistant to help select a smart band or find the right person to consult for a legal issue, it does not rely solely on its memory to respond. Instead, it performs web searches, reads the top few results, and then synthesizes a comprehensive answer.

The problem lies in the openness of the internet: Anyone can publish content, and in the current era of rampant AI-generated content, the cost of doing so is particularly low.

As long as attackers publish a few web pages specifically disguised for the target search—with layouts, tones, and sources indistinguishable from real results, whose sole purpose is to make the AI "endorse" the designated product to users—they do not need to hack into any system, access model weights, or modify prompts to influence all AI assistants that rely on connected retrieval.

This is exactly the threat model focused on in this study: Third-party content on the open web is quietly transformed into a "model-validated recommendation" through the agent's synthesis process.

The 3·15 demonstration proved that this scenario can happen, while this paper aims to answer: On which models, in what ways, and to what extent can this happen?

Figure 2: The SearchGEO evaluation framework: multi-domain cases, five attack modes, retrieval result injection design, and multi-dimensional evaluation metrics.

The SearchGEO Evaluation Methodology

The most challenging part of determining the actual impact of search result poisoning is isolating it from other variables. A web page that influences AI might do so because of its content, or simply because it appears more professional or ranks higher in results.

SearchGEO solves this problem by building a "hybrid search agent": It first caches real search results from SerpAPI, then replaces the original results at specified ranking positions with attacker-constructed web page content, thereby isolating the causal effect of poisoning.

The attack content itself is carefully controlled. It is generated by AI to match the quality of real search results for each task, then manually reviewed one by one to remove generation traces that would easily reveal the content is forged.

The research categorizes attacks into three levels and five modes: the machine layer (injecting hidden content invisible to humans but readable by the model into web pages), the trust signal layer (forging authoritative sources or creating the illusion of "unanimous agreement" across multiple sources), and composite attacks that combine the two. The core metric for measuring outcomes is the Attack Success Rate (ASR): whether the AI ultimately recommends the attacker-designated target to users.

Experimental Results

Under this evaluation system, the overall ASR of the 13 backends differs by an order of magnitude.

The most robust model is Claude-Sonnet-4.6, with an overall ASR of 0.0%; GPT-5.4-mini follows closely with only 0.8%. The most vulnerable model is Gemini-3-Flash, whose overall ASR reaches 31.4%—and a single "synthetic consensus" attack (where multiple seemingly independent sources all point to the same conclusion) can push its ASR up to 73%. The three Gemini variants, along with DeepSeek-V4-Flash and MiniMax-M2.7, all have an overall ASR above 17%.

Figure 3: Attack success rates of 13 mainstream large models under search result poisoning (sorted by ASR in ascending order; lower values indicate higher security).

This reveals a phenomenon: The performance gap between different large model backends is larger than the gaps between different domains or different attack modes. Additionally, different models respond differently to various attack modes. Gemini is most vulnerable to synthetic consensus attacks, while most other large models are more easily compromised by the "authority anchor + citation chain" attack. This indicates that defenses likely need to be designed specifically for individual models.

Is GPT Really Secure?

If you only look at the table above, it is easy to draw the conclusion that "GPT-mini is almost immune"—with 0.8% ASR, it is in the first tier alongside Claude.

The research additionally designed an agent-skill probe. When the AI assistant is no longer recommending a product but an "agent skill/plugin", its "endorsement" is no longer a simple sentence, but a directly executable installation command. The recommendation chain becomes an installation chain, exposing users to greater information security risks.

The research tested this scenario using a synthetic consensus attack (three forged sources converging on a completely non-existent skill name), and the results were unexpected: GPT-5.4-mini fully accepted the fictional skill across 10 high-risk OpenClaw scenarios, and provided the exact installation command verbatim.

Across a total of 18 matching scenarios in three different ecosystems (OpenClaw, Anthropic Skills, Hermes Agent), GPT-5.4-mini accepted 18 of them, while the newer GPT-5.5 accepted 16 (the only two rejections occurred in the Anthropic Skills ecosystem). This "unconditional acceptance" holds true across all five attack modes.

Therefore, GPT's 0.8% ASR does not represent robustness: Conventional evaluations mostly cover mature, well-known tasks, but once shifted to emerging scenarios like agent-skill recommendation, GPT is almost completely compromised.

Claude's 0% ASR Comes at a Cost

While GPT becomes vulnerable in new scenarios, Claude's performance is more nuanced: Behind its 0% ASR lie two easily overlooked costs.

The first is "silent drift". A failed attack (ASR=0) does not mean the answer remains completely unaffected. The research uses a metric called ΔOSS to measure how much the answer shifts toward the attack target relative to the clean baseline. In 8 out of 264 test cases (3.0%), Claude-Sonnet-4.6 experienced silent drift exceeding one rating level: The attack failed to make it explicitly endorse the target, but had already subtly pushed its response in the direction the attacker desired. In the aggregated statistics across all 13 backends, composite attacks can cause this type of drift in 15.0% of "unsuccessful" cases. Relying solely on ASR will systematically underestimate the true impact of attacks.

Figure 4: Distribution of silent drift by attack mode: the machine layer shows a defensive pullback, while the trust signal layer and composite attacks push answers toward the attack target even when they do not "succeed".

The second issue, termed "collateral rejection" by the research, refers to a scenario where unnecessary rejections spread to unrelated content. Under the clean baseline (no attacks at all) of the agent-skill probe, Claude rejected all useful responses in 10 scenarios. More extremely, in 8 scenarios, it directly denied the existence of the legitimate OpenClaw ecosystem, blocking legitimate tools as suspicious entities.

This means that when attackers flood a product category with a large number of fake brands, Claude may reject the entire category out of caution, causing legitimate ecosystems to be mistakenly blocked, and attackers still achieve their destructive goals. This is a failure mode that cannot be measured by traditional ASR metrics but causes tangible harm to users.

Implications for Defenses

The research also points out two more specific issues related to defenses.

First, "forged consensus" deserves vigilance. The phenomenon of "three people spreading a rumor makes it feel like the truth" still applies to AI assistants: ASR increases monotonically with the number of mutually independent supporting sources. Simply reposting the same promotional article repeatedly is ineffective—attackers need to invest real effort to forge multiple seemingly independent sources, which in turn points out the direction for defenses.

Second, defenses are not model-agnostic. A set of OWASP-based prompt-level defenses can reduce ASR but cannot eliminate it entirely; while a ready-made OpenClaw deployment framework reduces ASR for two backends, it amplifies authority-class attacks by 31.8% on Gemini-3-Flash. This demonstrates that "model and deployment framework" must be evaluated and designed as an integrated system.

Conclusion

Search content manipulation remains an unresolved challenge for current mainstream LLM assistants. Although Claude-sonnet and GPT-mini demonstrate better security performance on the evaluation benchmark compared to other models, GPT is completely compromised in new scenarios, and Claude also has potential issues of excessive rejection and silent drift.

The research puts forward several recommendations:

1. Treat "search recommendation reliability under adversarial content" as a first-class evaluation dimension for model security, rather than a trivial issue at the deployment layer.
2. Expand evaluation metrics beyond a single ASR, incorporating overlooked risks such as silent drift and erroneous rejection rates.
3. Design defense solutions specifically for the "model + framework" combination, rather than relying on a universal patch.
4. Service providers should truthfully disclose to users the capability boundaries of different models and different pricing tiers in these source-sensitive tasks.

As AI assistants increasingly perform web information retrieval on our behalf, this research reminds us that the work of securing their evaluations and defenses is far from complete.

About the Authors

This research was completed jointly by the Generative AI Excellence Center at KAUST (King Abdullah University of Science and Technology), Jilin University, Zhejiang University, and the Swiss AI Laboratory IDSIA. The first author is Yimeng Chen from KAUST, and core members include Zhe Ren and Dandan Guo from Jilin University, Firas Laakom from KAUST, Yu Li from Zhejiang University, and Jürgen Schmidhuber from KAUST/IDSIA.

References: https://arxiv.org/abs/2606.16821

This article is from the WeChat official account "XinzhiYuan", authored by XinzhiYuan; edited by LRST, and published by 36Kr with authorization.