How to make large language models think more accurately

In July 2026, the director of the IT department at a tertiary hospital came to me with a thoroughly confused look on his face.

They had built an AI-assisted diagnosis system using GPT-4, which delivered impressively high accuracy during testing. But just two weeks after launch, the ICU director slammed his fist on the table: the AI had recommended potassium supplementation for a patient with elevated blood potassium levels.

The reason was simple — the model spotted the symptoms "patient fatigue" and "ECG abnormality" and statistically associated them with hypokalemia, completely ignoring the blood potassium reading of 6.2 on the same lab test sheet.

AI is not incapable of thinking; it just thinks in a flawed way.

Stanford University's 2026 AI Index Report shows that even among state-of-the-art models, hallucination rates in complex professional domains remain as high as 15%–30%. That means for every ten responses, there could be one to three instances of "speaking nonsense with a straight face."

How do we make large language models think more accurately? This is the most central question in the AI field in 2026.

1. First, Understand: Why Large Models Are Bad at "Thinking"

To understand the thinking flaws of large models, we must first grasp their essence.

The current mainstream large language models are essentially probability generators. They predict the next most likely word based on statistical patterns in their training data.

Using the framework of Nobel Prize-winning economist Daniel Kahneman, large models are essentially a supercharged version of "System 1" — relying on intuition and association, reacting quickly, but prone to errors. They excel at producing "statements that sound reasonable" rather than "statements guaranteed to be correct."

Genuine logical reasoning requires "System 2" — slow, energy-consuming, but capable of breaking down problems, verifying hypotheses, and self-correcting.

Here's the problem: for a large model to evolve from System 1 to System 2, it must cross four critical hurdles.

The Four Hurdles:

① Inability to Decompose: When facing complex problems, the model directly "dives for" an answer instead of breaking it down into smaller steps.

② Inability to Verify: If an error occurs mid-reasoning, the model cannot self-inspect, and the error propagates all the way to the final output.

③ Inability to Backtrack: Once linear reasoning takes a wrong turn, there is no mechanism for the model to step back and retry.

④ Inability to Stop: The model overthinks simple problems and underthinks complex ones.

These four hurdles correspond exactly to four sets of solutions.

2. Four Keys: Moving the Model from "Guessing" to "Thinking"

Key 1: Chain-of-Thought (CoT) — Let the Model "Think Deeper"

In 2022, Jason Wei's team at Google Brain discovered a striking phenomenon: simply adding the phrase "Let's think step by step" to the prompt could boost the model's accuracy on mathematical reasoning tasks from 17.7% to 78.7%.

This is Chain-of-Thought (CoT) — having the model write out its reasoning steps before giving a final answer.

Why does it work? Four reasons:

First, Decomposition. Break a large problem into small steps, making each step far easier to get right. Just like solving a complex math problem, calculating step-by-step is far more reliable than doing mental math.

Second, External Memory. Intermediate steps act as a "scratchpad" for the model, helping it track its reasoning process so it doesn't forget the beginning halfway through.

Third, Forced Explicitness. It forces the model to explicitly write out its reasoning process, reducing the tendency to directly "guess" answers.

Fourth, Opportunity for Self-Inspection. Mid-reasoning, the model can look back at previous steps to spot contradictions and correct them in time.

But CoT is not a silver bullet. Its limitation is that if a mistake is made in the reasoning chain, the error propagates all the way to the end. Moreover, applying CoT to simple factual questions (e.g., "What is the capital of France?") only adds latency and cost, which is pure waste.

In a nutshell: CoT makes the model shift from "intuitive quick answers" to "slow step-by-step thinking," at the cost of more tokens and higher expenses.

Key 2: Self-Consistency Sampling — Let the Model "Try Multiple Times"

CoT reasoning chains can be error-prone. So what's the solution?

In 2022, Wang et al. proposed a simple yet effective idea: instead of trusting a single reasoning chain, let the model independently derive answers K times from different perspectives and then vote — the answer that appears most frequently is most likely correct.

This is called Self-Consistency Sampling. Its core insight is that complex reasoning problems often have multiple valid reasoning paths. If multiple paths all point to the same answer, that answer has a much higher confidence level.

Real-world test data is compelling: on the MATH benchmark, single-pass reasoning hits 54% accuracy, Self-Consistency with K=5 reaches 65%, and K=20 gets to 72%. When used together with CoT, accuracy can be further improved by 12–18%.

But the cost is straightforward — expenses grow linearly. 20 sampling runs mean 20 times the computational overhead. Therefore, Self-Consistency Sampling is best suited for scenarios with definite answers, limited answer spaces, and extremely high accuracy requirements (mathematical calculations, code generation, fact-checking), and is not ideal for open-ended creative tasks.

There is also a pitfall: if the temperature parameter is set to 0, all sampling runs follow the exact same path, making voting completely meaningless. A high temperature of 0.7–0.9 must be used to ensure diversity.

Key 3: Tree-of-Thoughts (ToT) — Let the Model "Turn Back When It Hits a Dead End"

CoT is a single line that cannot backtrack once it goes wrong. Tree-of-Thoughts (ToT) unfolds reasoning into a tree — each node is a "thought state," and each edge is a "reasoning step."

On the 24-point game, using GPT-4 directly yields only 4% accuracy, but applying ToT can skyrocket performance to 74%. Why? Because the 24-point game requires testing multiple combinations, backtracking when a path fails, and switching to an alternative — a natural advantage of tree search.

ToT enables three capabilities that CoT cannot deliver: Branching (exploring multiple directions from the current state), Backtracking (abandoning hopeless branches and returning to the previous node), and Evaluation (judging how close the current state is to the goal).

However, ToT is a "heavyweight weapon" — its computational cost is 10 to 100 times that of CoT, and the number of LLM calls equals the number of branches multiplied by the search depth. For problems that CoT can solve, using ToT is simply burning money.

Key 4: Self-Refine / Reflexion — Let the Model "Check Its Own Homework"

The final key is self-correction. The core workflow is: Generate → Evaluate → Improve → Repeat.

The model first produces an initial draft, then obtains feedback, revises itself based on that feedback, and repeats until it meets standards or reaches a predefined limit. On the HumanEval code generation task, Reflexion can boost accuracy by 24%.

But there is a counterintuitive finding here: the quality of the feedback source determines the upper limit of performance. Sorted by reliability —

Feedback Source Reliability Ranking:

★★★★★ Code Execution (unit tests, compilation errors)

★★★★ Rule Checking (format validation, constraint satisfaction)

★★★ Tool Invocation (calculators, search engines)

★★★ Human Feedback

★★ Cross-Validation by Another LLM

★ Self-Evaluation by the Same LLM (the worst, lacking external reference points)

In other words, letting a model check itself delivers the poorest results. Truly effective self-correction must rely on external feedback — unit tests, compilers, search engines, rule engines, or even a separate model.

3. How to Choose Between the Four Keys

These four keys are not mutually exclusive; they are progressive. The correct usage sequence is —

① Start with a direct question (baseline)

→ Not accurate enough? Add CoT (Chain-of-Thought)

→ Still unstable? Add Self-Consistency with multiple sampling and voting

→ Dealing with combination/search problems? Use ToT (Tree-of-Thoughts) for exploration

→ Have access to external feedback sources? Use Self-Refine for iterative improvements

The key principle: Start simple, then move to complexity. Most production systems do not need ToT at all. Try CoT first, and only upgrade if it fails. Jumping straight to the heaviest solution often wastes money without delivering the expected results.

4. Paradigm Shift: From "Prompt Engineering" to "Reasoning Models"

The four keys discussed above all improve accuracy using prompt techniques and reasoning strategies, without modifying the model's underlying weights.

But starting from the end of 2024, a completely new path emerged — Reasoning Models.

OpenAI's o1/o3 series and DeepSeek R1 no longer rely on users to add the phrase "Let's think step by step." Instead, they embed reasoning capabilities directly into the model weights. When given a problem, the model automatically unfolds its reasoning, self-validates, and backtracks to correct errors when detected.

This is underpinned by three critical technological breakthroughs:

Breakthrough 1: Process Reward Model (PRM)

Traditional RLHF uses an Outcome Reward Model (ORM) — giving high scores for correct final answers and zero scores for wrong ones. This works well for creative writing but is a disaster for mathematical and logical tasks. A correct answer might have reached the result through a convoluted, flawed path; a wrong answer may only have failed at a single step.

The Process Reward Model (PRM) assigns a score to every intermediate step in the reasoning process. This forces the model to learn to "think honestly," drastically reducing logical hallucinations in intermediate steps.

Breakthrough 2: Reasoning-Side Scaling Law

We used to believe that "the bigger the model, the smarter it is." The o1/o3 series proved a new law: By increasing computational effort during inference, a small-parameter model can outperform a large-parameter model.

A vivid metaphor: Give o3 10 milliseconds to think, and it performs like a high school student; give it 10 seconds, and it acts like a PhD; give it 10 minutes for deep search, and it can surpass the level of human experts.

This is "trading computation for intelligence" — intelligence is no longer static parameter weights, but a dynamic search process.

Breakthrough 3: Self-Play and Emergent Reflection

DeepSeek R1 is trained purely using reinforcement learning, without relying on human-annotated reasoning steps. The model autonomously "invented" reasoning behaviors on math and code tasks — including reflection, backtracking, and step-by-step verification. This Emergent Behavior was not part of the training objective, but a capability that naturally arose from large-scale reinforcement learning.

Even more striking is the cost data: DeepSeek R1 achieves near state-of-the-art reasoning performance at less than 5% of the cost of OpenAI o3. Its 671B parameter MoE architecture activates only 37B parameters per token, with API pricing at just $0.55 per million tokens. The open-sourcing of reasoning capabilities allows far more enterprises to access "thinking AI" affordably.

5. Enterprise Practice: Three Lines of Defense to Stop AI from "Talking Nonsense"

Whether it's prompt techniques or reasoning models, they only solve the problem of "how to think." But in enterprise scenarios, "whether the thinking is correct" also depends on "whether the knowledge being used is accurate."

Stanford's 2026 report shows that hallucination rates in professional domains remain 15%–30%. Enterprises need to establish Three Lines of Defense:

First Line: Input Constraints — Draw Clear Knowledge Boundaries for the Model

Explicitly tell the model in the prompt: What its knowledge scope is, that it must say "uncertain" when going beyond the scope, and that it must cite sources when answering. Require the model to show its reasoning process for complex problems to facilitate manual review.

This is like telling students before an exam: "This is an open-book test, but you can only use the specified textbook. For questions you are unsure about, you must write 'I don't know' and are not allowed to make up answers."

Second Line: RAG Architecture — Equip the Model with an "External Brain"

RAG (Retrieval-Augmented Generation) is currently the most mainstream hallucination mitigation solution for enterprise AI. Its core idea is: before the model answers, it first retrieves relevant documents from a trusted knowledge base and feeds them to the model as context. This anchors the AI's responses to real, verified information.

Practical data is compelling: with RAG, the hallucination rate of professional domain Q&A can drop from over 20% to below 5%.

RAG technology also evolved in 2026. From the simple "vector retrieval + generation" approach, new paradigms have emerged, including Adaptive Retrieval (dynamically adjusting retrieval strategies based on problem complexity), GraphRAG (building knowledge graphs to support multi-hop reasoning), and Real-Time Streaming RAG (synchronizing knowledge base updates in seconds).

The latest practices also introduce a Factuality Gate — adding a verification step between generation and output, using a lightweight model to check that every factual claim is supported by retrieved content. Data from an insurance company shows that this can reduce the hallucination rate from 9% to 2.2%, with a per-validation cost of less than $0.0003.

Third Line: Human-Machine Collaboration — Human Oversight Is Mandatory for High-Risk Scenarios

No matter how perfect technical safeguards are, high-risk scenarios cannot be fully delegated to AI. Enterprises should implement tiered review mechanisms:

Low Risk (internal knowledge Q&A): AI generates outputs independently, with periodic spot checks

Medium Risk (customer communication, content creation): AI generation + rapid human review

High Risk (legal advice, medical recommendations, financial decisions): AI assistance + final expert review

6. Implementation Guide: Three Steps

After covering all these methodologies, how exactly should enterprises get started? I recommend a three-step approach:

Step 1: Classify Your Problems

Not every problem requires deep reasoning. Build a lightweight query classifier that divides problems into simple, medium, and complex tiers. Route simple problems to traditional LLMs for fast answers, apply CoT + Self-Consistency Sampling to medium-difficulty problems, and reserve reasoning models or ToT only for complex problems.

This is called a Hybrid Routing Architecture. Real-world tests show it can cut reasoning costs by 60–75% while maintaining quality.

Step 2: Build an Evaluation System

No evaluation means no optimization. Enterprises must create their own evaluation datasets — covering question-answer pairs across typical business scenarios, annotated with standard correct answers. Every time you adjust prompts, switch models, or upgrade technologies, run the full evaluation set to quantify performance changes.

Track four core metrics: Accuracy, Latency (P50/P95/P99), Cost (token consumption