How is a RAG project "created" in real training?

RAG technology is far from simple data injection. Instead, it reshapes the core framework of AI understanding and decision-making. This article deeply dissects the real dilemmas in RAG projects - from corpus screening, contradiction handling to result delivery, and reveals why 90% of the work still relies on human judgment.

In previous articles, I spent a lot of space explaining why RAG is important. But when you actually get to the project site, you'll quickly realize one thing: RAG is not just a "module addition" technical problem. It's a whole set of data and judgment systems.

Many people who are new to it may think that a RAG project is simply:

Feed more materials to the model and let it speak according to them.

But the real situation is that what truly determines the effectiveness of RAG has never been "whether there are materials", but "how the materials are used".

Let's start with a very real work scenario

In the scenario of a conversational AI assistant, what a RAG project usually faces is not "standard Q&A", but a structure like this:

• A piece of historical conversation that may be single-round or multi-round
• The latest question raised by the user
• 1 - 3 reference materials retrieved by the system

What the model needs to do is not simply repeat the materials, but:

Understand the conversation context → Judge which materials are useful → Integrate information → Give a "helpful" answer to the user

From a training perspective, this essentially involves doing one thing: Material reading comprehension + Question understanding + Information integration + Expression control

The "trilogy" in a RAG project: Question, material, answer

If you break down a RAG project, it actually consists of three parts, but none of these parts is "naturally reliable".

1️⃣ The question itself may have problems

You'll frequently encounter situations like this in the project:

• The question has unclear semantics
• There are contradictions in the context
• There are serious logical leaps
• It even contains obviously unreasonable or harmful intentions

This means that not every question is worth answering seriously.

2️⃣ The reference materials may not be "reliable for reference"

Many people will subconsciously think that "reference materials" are authoritative when they first see them. But in real projects, common problems with the materials include:

• They are not relevant to the question
• The information is incomplete
• There are conflicts among multiple materials
• There are even common-sense errors

So in a RAG project, the "materials" are not the answers, but just candidate evidence.

3️⃣ The answer is the final deliverable

What is finally delivered is not "whether it matches the materials", but an answer that the user can directly use. This means that the answer needs to meet the following requirements simultaneously:

• Understand what the user really wants to ask
• Not violate the facts in the materials
• Have sufficient and complete information
• Be naturally expressed and not sound like "reading from the materials"

Why can't a RAG project be "automated"?

Many people will ask a question:

Since the current models are already so powerful, why is a large amount of manual intervention still needed?

The answer is actually quite realistic: In a RAG project, 90% of the difficulties lie in "judgment", not "generation".

For example:

• If the materials are incomplete, should they be supplemented?
• If there are errors in the materials, should they be corrected?
• If multiple materials conflict, which one should be trusted?
• If there are problems with the historical conversation, should it be skipped directly?

Essentially, these problems cannot be solved by the model itself. Instead, humans are setting the judgment boundaries for the model.

What abilities are truly trained in a RAG project?

On the surface, a RAG project trains the model to "answer questions with materials". But at a deeper level, it trains three abilities:

1. Information selection ability: What should be used, what shouldn't be used, and what can only be used as background.
2. Context alignment ability: The answer does not exist independently but is embedded in a conversation.
3. Result-oriented ability: It's not about "what the materials say", but about "whether the user can use it after reading".

That's why a RAG project is often a key step for many large models to become "usable".

An easily overlooked fact

In many teams, RAG projects are regarded as "interim solutions". But in real business, they are often long - term infrastructure.

The reason is simple:

• The business is changing
• Knowledge is changing
• But the model can't be retrained every day

And RAG is precisely the bridge connecting the "stable model" and the "changing world".

This article is from the WeChat official account "Everyone Is a Product Manager" (ID: woshipm). The author is Qinglanse de Hai. It is published by 36Kr with authorization.