Be alert to the "persuasion bombing" of large language models

A common argument we often see is that applying AI to augment human intelligence and having humans assist AI can help enterprises significantly boost productivity without sacrificing quality and professional standards. Although large language models (LLMs) may make mistakes or even produce "hallucinations", these risks can be offset by a well-trained "human in the loop" mechanism, where humans review and verify the results generated by AI.

However, a recent study shows that we may have overestimated our ability to check the outputs of large language models, while underestimating how easily we can be manipulated by them. Researchers Steven Randazzo, Akshita Joshi, Katherine Kellogg, Hila Lifshitz, Fabrizio Dell’Acqua, and Karim Lakhani observed the interactions of hundreds of Boston Consulting Group consultants with AI in a controlled experiment. They found that large language models intensively use various rhetorical devices to launch an "information bombardment" on human users, overwhelming their judgment and making people believe that the answers given by AI are correct — even when they are not.

The researchers pointed out that AI is not a neutral collaborator, but a highly persuasive "persuader" that will carry out "persuasion bombing" on users to push them to accept AI's conclusions.

I invited these researchers to further explain their findings and what these findings may mean for enterprises. They warned that many of the preventive mechanisms currently set up by companies may not truly guarantee safety, but may instead inadvertently cede important decision-making power to AI. (The following conversation has been edited.)

HBR: When collaborating with AI, people usually face three common types of problems: failing to understand how AI makes decisions (opacity); humans in the "human in the loop" mechanism becoming overly dependent on AI and gradually losing vigilance (blind complacency); and AI itself making mistakes (inaccuracy). You claim to have discovered a fourth barrier — "persuasion bombing". What exactly is it? Why is it worthy of our vigilance?

Lifshitz: The so-called "persuasion bombing" refers to a phenomenon that occurs when a responsible generative AI user tries to verify the output of AI. We found that when professionals are verifying facts and pointing out possible errors in generative AI,

the model not only refuses to admit mistakes, but also defends its original answer by intensively using multiple persuasion strategies. The deeper and more worrying problem is that large language models are designed with a persuasion-oriented logic from the very beginning.

Joshi: Imagine this scenario: you are working with a junior colleague, and you find an error in their work and ask them to recheck it. They will usually obediently correct the mistake you pointed out. People take it for granted that generative AI will react in a similar way. But in our study of strategy consultants, we found that when humans try to verify or question the model's answers, AI often does not re-examine its own answers, but instead redoubles its efforts to exonerate itself. That is "persuasion bombing".

Kellogg: This is worrying because it attacks exactly the mechanisms we rely on to make judgments in uncertain situations — professional competence, skepticism, and active participation. Originally, "careful verification" was regarded as the solution to the problems of opacity, over-dependence and inaccuracy, but now it has

become part of the problem. The more diligently professionals question the model, the more persuasive rhetoric they receive.

Randazzo: Most organizations believe that as long as they keep "human in the loop" in the process, they have solved the problems of opacity, over-dependence and inaccuracy. But our research shows that "human in the loop" often becomes an empty slogan rather than a practical and effective protection mechanism. If the AI system further "raises the stakes" when questioned — becoming more organized, more confident, and more sophisticated in rhetorical skills — it will create a double challenge: in the early stage, the output may be so persuasive that users will not verify it at all; in the later stage, when users start to verify, AI's "persuasion offensive" will further escalate.

HBR: How exactly did this process happen in your study?

Lifshitz: These participants were not ordinary users who input prompts at will, but 244 consultants from Boston Consulting Group who were solving a highly realistic strategic problem. They needed to analyze the financial statements and executive interview materials of a fictional company, and put forward suggestions on which areas the CEO should invest in. These people are all professionally trained consultants, good at examining data and deliberating on proposals. Even so, among the 244 people, only 72 took the initiative to verify the AI output. We recorded more than 4,300 prompts and responses, and identified 132 explicit "verification behaviors" — including fact-checking, pointing out inconsistencies, or directly refuting.

Joshi: And when people really tried to verify the AI output, we observed a striking pattern. When consultants asked the model to "check its own analysis", point out loopholes in it, or explicitly express different opinions, the model would not really re-examine its conclusions. On the contrary, it usually first apologizes amicably, generates a brand new set of analysis, adds more comparative information, and then draws exactly the same conclusion — only this time, the conclusion is wrapped in layers of data and rhetoric to be more airtight. In all 132 verification interactions, the pattern we saw is highly consistent: the more the user verifies, the stronger the model's "persuasion offensive" becomes.

Kellogg: The way large language models carry out "persuasion bombing" is that in the face of consultants' verification attempts, they will use progressive and multi-dimensional rhetorical strategies to fight back, including strengthening their own credibility, enhancing logical argumentation, and forming emotional resonance with users, thus forcing consultants to accept their initial output instead of making modifications.

HBR: This seems to contradict one of the most common criticisms of large language models. Many people believe that large language models are too "obsequious", that is, they will overly echo users even when users are wrong. How does your research change our understanding of this potential failure mode of the model?

Joshi: Obsequious behavior and "persuasion bombing" are indeed related, but they are not the same. The traditional sense of "obsequiousness" is passive and user-led. The model will cater to what users want to hear, agree with their narrative framework, and give echoes and flattery. But "persuasion bombing" is different. It is model-led and will escalate layer by layer. The model no longer just obeys the user, but actively defends its previous output, and once questioned, it will further strengthen its arguments.

Kellogg: Our research is not intended to overturn the previous concerns about the blind obsequiousness of large language models. Our research shows that obsequiousness is only one mode of the broader and more adaptive "persuasion ability" of large language models. We need to change our mindset, stop regarding large language models as submissive followers, and realize that they are actually "persuaders" that dynamically adjust strategies according to interactions, and they have the ability to resist, guide, and even overwhelm human judgment.

Joshi: This turns the problem into a more complex, and in a sense more dangerous failure mode. A model that nods and echoes whatever you say is actually relatively easy to see through and ignore. But a model that will refute you, demonstrate with seemingly rigorous logic, and express views in a professional and friendly way is more difficult to detect and resist, especially when time is tight and the issue is very complex. What's worse, these two failure modes reinforce each other. The model may first win your trust by echoing your initial assumptions, which is "obsequiousness"; then, when you find loopholes and raise questions, it switches to "persuasion mode" to defend its own conclusions.

Lifshitz: When this happens, the risk is no longer that it compromises too quickly or defends itself too hard, but that it first lowers your guard and then overwhelms your judgment. The independent evaluation ability is gradually eroded, the boundary of responsibility becomes blurred, and those bad decisions will begin to appear "fully argued and well-grounded".

HBR: What should people do when they suspect that they are encountering "persuasion bombing"?

Lifshitz: As AI is increasingly embedded in the decision-making process, its risks are no longer just making mistakes, but also generating "influence". These systems are not just generating answers, they are shaping human judgment. To protect professional judgment, we must consciously set up defense lines: identify persuasion behaviors, move verification work outside the conversation loop with AI, and establish corresponding protection mechanisms at the system level as the agency of AI agents increases.

Joshi: The first step is to raise awareness. Professionals not only need to receive prompt training, but also learn to identify persuasion behaviors. In fact, there are some relatively clear signals of this kind of phenomenon. For example, after apologizing, the model will restate its conclusion in a determined tone; it will insert a large amount of new data that you did not ask for into the conversation; it imitates your wording, appreciates your insights, and leads you back to its original position; when questioned, it will shift from logical argument to argument based on authority or credibility. When you raise an objection and the model becomes more complex and defensive, that is a warning sign — you should stop, and even exit this round of conversation directly if necessary.

Kellogg: Another useful practice is that you should not trust the model more just because it behaves more confidently, and appropriately reduce the weight of trust in high-confidence outputs.

Randazzo: The first step is to slow down, which seems simple but very important. When you question the model and it behaves more confidently, it is easy to mistakenly think that the quality of analysis has improved. But sometimes, what is improved is only rhetorical skills, not the quality of reasoning. If you feel more persuaded but have not obtained more information, that is a red flag.

Second, take the initiative to create distance. Temporarily step out of the current conversation, go back to the original data, and conduct independent verification. Treat the model's output as a preliminary assumption rather than a final answer, and consciously stress-test it. Finally, you can ask the model to put forward the most powerful counter-argument, for example: "If this recommendation is to fail, which assumptions must be invalid?"

Joshi: One point is particularly important: verification must take place outside the conversation. When you ask the model to check its own reasoning process, you are essentially giving it another chance to persuade you. Real verification must rely on independent evidence, such as original data, review by colleagues, and cross-checking between different sources.

Kellogg: From an organizational perspective, a second AI dedicated to criticism and questioning can also be introduced to add necessary "friction", which individuals can hardly maintain for a long time on their own.

Lifshitz: In other words, the solution is not to stay away from AI, but to use AI in a different way — to introduce institutionalized checks and balances as well as conscious independent judgment.

HBR: What advice do you have for managers who are asking employees to use AI more and in a wider range of work links?

Lifshitz: As these systems are more deeply integrated into daily work and increasingly capable of influencing human judgment, leaders need to assume three responsibilities: first, while improving employees' AI capabilities, prevent them from falling into blind dependence; second, defend "accountability" in major decisions; third, redesign work processes as AI gradually evolves from a tool to an agent.

Kellogg: Leaders certainly should encourage employees to actively try.

The only way for employees to master large language models is to use them in more work scenarios, see what works, what doesn't, and which attempts were impossible yesterday but achievable today. But what leaders should not do is force employees to use large models in areas where they think the models are inaccurate or ineffective. In addition, leaders should intervene from the system level, instead of simply relying on individual vigilance. For example, models that can provide links to information sources should be provided; the models should be set to "raise questions and list counterexamples" instead of directly giving final answers. In addition, leaders should not treat all employees as starting from the same baseline. Inexperienced novices are more easily persuaded by fluent and confident outputs. Therefore, leaders should have a hard rule: employees must first prove that they have the ability to manually complete a specific task before they are allowed to use large language models in that task. At the same time, in major decision-making scenarios, novices should not be the "final arbiters" of judgment results.

Joshi: Leaders also need to face a more intractable problem — responsibility attribution. The basis of the professional responsibility system is that "judgment must truly belong to the individual". If the conclusion drawn by a professional is reached after the system strongly promotes its position (and even escalates its rhetoric layer by layer when refuted), to what extent can this judgment still be regarded as his own? When a major decision goes wrong, who exactly should be responsible for the decision? The existing professional and governance frameworks simply cannot answer this question.

Randazzo: Last year, the question everyone was discussing was "How do we use generative AI?", and this year the question has become "How do we deploy AI agents?" Leaders now need to move past the stage of "universal adoption for all staff" and start thinking about the architecture of work processes and how to make better judgments. At the moment, there is still a lot of confusion about how to do this safely and responsibly. Not long ago, I worked with a financial company to help them sort out their work processes, determine which links can be completed autonomously by agents, which links require human intervention, and which links are suitable for human-AI collaboration. During the discussion, I asked several times: "Why do we still need a human here?" The answers were all the same: "We will keep a 'manual review' for now. When we feel more confident later, we will fully automate it." This sounds cautious, but it defaults to the premise that "manual review" is really working and can keep the gate. Now we keep saying "human in the loop" just like saying "fasten your seatbelt", which represents a sense of security. But seatbelts only work when used correctly. Similarly, "human in the loop" can protect a company only if the person in the loop has extremely high "AI hygiene" — which means they need to continuously improve and maintain their AI skills, including being keenly aware of when the system shifts from "objective analysis" to "subjective persuasion".

In agent-based work processes, persuasion behaviors may occur at any link, or at the final output stage — where humans often only see a carefully polished, seemingly conclusive result. If this result appears more authoritative when questioned, the manual review may not remain neutral. AI can bring great value as a tool for drafting initial versions, integrating information, generating assumptions or exploring scenarios. But when it participates in strategic, financial, regulatory or operational decisions as an autonomous agent, it is actually shaping human judgment. There is an essential difference between "using AI to generate options" and "allowing AI to shape judgment". As AI evolves from a tool to an agent, from an assistant to a participant in the decision-making process, leaders must clearly define the role it plays. The management needs to keep asking: "Under what circumstances can we safely allow a system with superb rhetorical skills to influence the organization's judgment?" AI should augment human reasoning ability, not replace it.

Thomas Stackpole | Article

Thomas Stackpole is Senior Editor of Harvard Business Review (English edition).

LIAO Qijing | Proofreader

This article is from the WeChat Official Account "Harvard Business Review" (ID: hbrchinese), author: HBR-China, 36Kr is authorized to republish it.