Between Algorithms and Reality: Xiaohongshu's AI Governance Dilemma

Today's AI-generated content is often substandard in both dimensions of "accuracy" and "human authenticity".

01 A "Perfect" AI-generated Homestay Photo

In the late autumn of last year, Lin Wei, a young woman from Hangzhou, planned to spend the weekend in Dali. She opened Xiaohongshu and entered "recommendations for homestays in Dali", only to be greeted by full-screen views of the Cangshan Mountain and Erhai Lake, Bai ethnic courtyards, and sunrises in front of French windows — every photo was as beautiful as a movie still. She finally selected a "niche designer homestay" with a 4.9 score. The note wrote: "Open the window and you will see Erhai Lake. You will be woken up by seagulls in the morning, and the flower pastries made by the landlord are so sweet that they warm your heart."

She took a three-hour flight, then transferred to a one-hour drive on winding mountain roads. Only after arriving did she find that the homestay was hidden deep in a dusty alley, the so-called "French window" faced a mottled cement wall, and the blue Erhai Lake in the note was nothing more than a blue backdrop cloth the landlord set up on the roof. Lin Wei later learned that the pictures attached to the note that made her heart beat were all AI-generated.

This is not an isolated case. On September 3, 2026, Xiaohongshu released an announcement through its official account "Xiaoshuguanjia": from June to August this year, the platform removed and disposed of nearly 120,000 fake notes produced and published by AI. Among them, three types of typical violating contents were listed: AI-fabricated medical aesthetic experiences, AI-fabricated homestay text and photo experiences, and AI-fictionalized family backgrounds and educational experiences for selling teaching supplementary materials.

120,000 notes. Assuming the average number of views is 100, that means 12 million users who may have been misled behind it, which is also a systemic crisis that a content community is facing.

02 Why Are The "Fake Grasses" Grown by AI Overflowing?

What is the core value of Xiaohongshu? It is authenticity. A user shares the facial cleanser he has used, a traveler records the real hiking route, a new mother writes down her parenting experience — these UGC contents based on real experiences form the strongest moat of Xiaohongshu. Users come here not to watch advertisements, but to see the lives of "real people".

But AI is changing all of this.

In the past, counterfeiting had costs. You had to take photos on site, conceive copywriting, retouch images, and maintain the persona of the account. A fake store exploration note required at least one real person, one mobile phone, and several hours of investment. Now, with tools such as Midjourney, Stable Diffusion, and GPT-4o, one person can generate hundreds of "personal experience notes" a day. Enter "Dali Erhai homestay morning sunrise French window", and AI can output a set of photos that are so realistic that they can pass for genuine. Then let the large model write a "genuine" experience copy, with the tone of "girls, go for it", and a fake note is born.

What is even more terrifying is that these AI-generated contents are forming an industrial chain. The three types of typical violations mentioned in the announcement exactly correspond to the three tracks that are easiest to realize monetization:

Medical aesthetics. An AI-generated "before and after comparison photo" with the copy of "after doing this project, my boyfriend said I looked like a different person" can induce users to send private messages for consultation, and finally divert them to unqualified underground clinics. From June to August, Xiaohongshu removed more than 4,000 various violating notes per day in the medical aesthetics field, and banned more than 4,200 violating accounts.

Homestay and cultural tourism. AI-spliced photos forge real scenes, fictionalize labels such as "niche secret place" and "only locals know", and trick users into merchants that are not as good as their reputation. These notes are often attached with group-buying links, forming a complete closed loop from content to transaction.

Education. AI fictionalizes the personas of "985 top student mother" and "Haidian parent", fabricates children's learning experiences and grades, and the ultimate goal is to sell teaching supplementary materials, courses, and "internal materials". What you see is not the anxiety of a real parent, but a sales funnel carefully arranged by AI.

When "planting grass" (recommending good products) becomes "planting fakes", the content ecosystem of the community begins to rot. The more hidden risk is that users often do not know they have been cheated — AI-generated content is getting more and more realistic, and ordinary netizens do not have the ability to identify it at all.

03 Use the Spear to Attack the Shield: Govern AI with AI

Facing this crisis, Xiaohongshu's choice is: to govern AI with AI.

Almost every governance measure mentioned in the announcement is inseparable from technical means: strengthen the identification ability of unmarked AI content, strengthen the detection of the repeated use of the same image or video across regions and accounts, and clean up historical AI fake content from multiple links such as notes, comments and accounts.

This sounds like a perfect closed loop — since AI can generate fake content, AI can also identify fake content. But is the thing really that simple?

Technically, AI detection of AI-generated content currently mainly relies on several paths:

First, watermark and metadata detection. Some AI generation tools will embed invisible watermarks in images or texts, and platforms can identify AI content by detecting these watermarks. But the problem is that the content generated by open source tools often has no watermark, and the watermark technology itself is constantly being cracked.

Second, generation feature recognition. AI-generated images have subtle differences from human-shot photos in pixel distribution, noise patterns, and texture details. AI-generated texts also have their specific statistical features — such as the overuse of certain words, the regularity of sentence patterns, etc. Platforms can train special detection models to identify these features, but as the saying goes, "the law is strong, but the outlaw is stronger". With the iteration of the generation model, these features are getting less and less obvious, and today's detection model may become invalid tomorrow.

Third, cross-platform duplicate detection. The "repeated use of the same image or video across regions and accounts" mentioned by Xiaohongshu is a very smart idea. If the same photo of "Erhai Lake sunrise" appears in the notes of three different accounts in Beijing, Shanghai and Guangzhou at the same time, and all claim that "I took it last week", then at least two of them are fake. This kind of detection based on graph analysis does not depend on the authenticity of the content itself, but on the rationality of logic.

Fourth, behavior pattern analysis. Accounts that generate content in batches by AI often have common behavior characteristics: concentrated registration time, abnormal publishing frequency, mechanized interaction mode, and fan growth curve that does not conform to the natural communication law. By analyzing these behavior characteristics, the platform can identify suspicious accounts before the content level.

These methods have their own advantages and disadvantages, but the common problem is that they are all "after the fact" defenses. Every time AI generation technology makes progress, detection technology has to catch up. This is an asymmetric war — the attacker only needs to find one vulnerability, while the defender needs to plug all vulnerabilities.

Moreover, AI detection has a fundamental paradox: if the detection model is too radical, it will mistakenly kill a large number of normal contents. Is a photography enthusiast who uses AI to assist in image retouching considered to be generating AI content? Should a creator who uses ChatGPT to polish copywriting be marked? If the platform confuses "AI assistance" and "AI forgery", it may stifle the enthusiasm of creators, and even trigger a rebound of "AI stigmatization".

04 Are We Governing "AI" or the "AI Vibe"?

This leads to a deeper proposition: when we govern AI content, what on earth are we governing?

Is it the "authenticity and trust" mentioned in Xiaohongshu's announcement? Or something more essential — the existence of human beings?

Let's do a thought experiment. Suppose one day, AI-generated content completely surpasses human beings in factual accuracy. The medical popular science written by AI is more rigorous than that of real doctors; the homestays recommended by AI are more reliable than those recommended by real bloggers; the educational experience sorted out by AI is more practical than that of real parents. At that time, is it still necessary for us to reject AI content?

The answer may be no. But the problem is precisely that today's AI-generated content is often substandard in both dimensions of "accuracy" and "human authenticity".

First look at "accuracy". AI has no physical body, it cannot really stay in a homestay, receive a medical aesthetic treatment, or raise a child. All its "experiences" are probabilistic inferences based on training data. When AI writes a "Dali homestay experience", it is not recalling, but "hallucinating" — using the most statistically possible word combinations to construct a scenario that has never happened. That's why AI content is particularly good at "looking real", but often shows flaws in details: it may write the wrong location of Erhai Lake, make seagulls appear in the wrong season, or fabricate a homestay name that does not exist at all.

Then look at "human authenticity". Even if the AI content is completely correct in fact, it still lacks an irreplaceable thing: human perspective, human emotion, and human presence. When we read a travel note, we don't just want to know "whether this homestay is good or not", we also want to feel the mood of the author when she opened the window that morning, want to know the accidents and surprises she encountered during the trip, and want to find resonance from her story. This kind of "human authenticity" cannot be simulated by AI.

Therefore, the governance standard should be dual: not only to pursue factual authenticity and accuracy, but also to guard the human attribute of content.

The expression of Xiaohongshu in the announcement reflects this dual standard to a certain extent. On the one hand, the platform "encourages creators to use AI to assist creation" — which recognizes the value of AI as a tool; on the other hand, the platform requires "do not fictionalize places you have never visited, products you have not actually used, or events you have not experienced personally" — which defends the core trait of human content: "personal experience".

But implementing this standard is far more difficult than formulating it.

05 A Blurring Future

Let's go back to Lin Wei's story. If Xiaohongshu's AI detection system is powerful enough, before she sees that fake homestay note, the system has already intercepted it. She may miss a homestay that "looks very beautiful", but she will not have to experience the bumpy three-hour mountain road drive, let alone feel the disappointment of only seeing a cement wall after arriving.

This is the ideal picture of technical governance. But the reality is that the platform has to process hundreds of millions of contents every day, and the AI detection system cannot achieve 100% accuracy. Between "letting a fake content go" and "mistakenly killing a real content", the platform must make a trade-off. The current industry practice is that it is better to kill by mistake than to let it go — that is why Xiaohongshu can clean up 120,000 AI fake notes in three months, but at the same time a large number of creators complain that their normal content has been misjudged.

From a longer-term perspective, this will be a war without end.

When the ability of AI to generate content becomes stronger and stronger, and the technology of AI detecting AI becomes more and more mature, the confrontation between the two will continue to escalate. Eventually, platforms may have to rely on more complex systems — such as using multiple AI models to verify each other, or introducing a hybrid model of "human auditor + AI". But this also means that the operating cost of content platforms will rise sharply, and small and medium-sized platforms may not be able to afford such technical investment at all.

One possible future is: content platforms become stratified. Head platforms have the ability to build advanced AI governance systems and maintain a relatively healthy content ecosystem; while small and medium-sized platforms will become the hardest hit areas for AI fake content, and users will gradually concentrate on head platforms. Will this lead to further intensification of the "Matthew Effect" in the content field?

Another possible future is: users' acceptance of AI content gradually increases. Just as we are already used to advertisements and soft placements today, tomorrow we may get used to "this is written by AI, but as long as the information is useful, it is fine". At that time, "whether it is contributed by humans" may no longer be a core issue, and "whether it is useful to me" will be the core. But at the same time, this also means that "human authenticity" as a content value will be completely replaced by instrumental rationality.

This article is from the WeChat official account "Competition and Cooperation Artificial Intelligence", author: Jinghe, 36Kr is published with authorization.