Leading AI tech giants hit by sudden negative news

As we all know, modern large language models are ultimately performing complex pattern fitting on massive volumes of text data.

For AI companies, data is the source of everything, ranking as their top critical priority.

But now, this very source is facing a major disruption.

On July 2, Cloudflare — the global leader in web infrastructure and cybersecurity — issued an ultimatum to every AI company worldwide:

By September 15, 2026, "search engine" crawlers and "AI training data/agent" crawlers must be fully and completely separated under distinct identifiers.

After that deadline, any mixed crawler that misrepresents its true purpose will face immediate physical network disconnection and universal global blocking by the system.

This effectively places a heavy lock on the entire AI industry.

The underlying reason is easy to see: unfair distribution of profits.

NVIDIA controls all algorithm companies at the hardware level, raking in massive profits beyond imagination.

In the internet ecosystem, aren't web infrastructure providers also occupying the classic "shovel seller" niche?

Why shouldn't they claim their fair share of the gains?

01

Who exactly is Cloudflare?

You can think of it as the "customs authority" of the global internet, analogous to toll stations on a highway.

It handles roughly 20% of all global internet traffic, delivering foundational infrastructure services including Content Delivery Networks (CDN), Domain Name System (DNS), DDoS attack protection, and Web Application Firewalls (WAF).

It's widely known as the "Cyber Bodhisattva" across the web.

What does that mean?

For over a decade, it has provided free, robust anti-DDoS protection and CDN acceleration services to countless small-to-medium websites and personal blogs around the world.

The key word here is free.

But this once benevolent, widely praised service provider is now seething with anger.

According to Cloudflare's monitoring data, Bot (automated crawler) traffic surpassed human traffic in the first half of 2026.

"The industry originally predicted this tipping point would arrive in 2027, but the explosive growth of AI Agents has accelerated this milestone by a full year."

Following current trends, within just five years, the ratio of machine traffic to human traffic will reach 1000:1.

This will clearly trigger a critical problem: when the primary users of the internet shift from humans to AI, the existing underlying commercial order will inevitably collapse.

The internet ecosystem has long operated on a fundamental symbiosis between search crawlers and traffic feedback:

You launch a website and publish high-quality content; crawlers from search engines like Google visit your site to scrape that content.

In return, the search engine indexes your pages. When users search for relevant queries, the search engine displays links to your website.

Users click those blue links, driving traffic back to your site, and you monetize that traffic through advertising to generate revenue.

This is a perfect win-win cycle that has operated smoothly for 30 years.

You supply content, we supply technology, and everyone profits together.

But generative AI has completely broken this long-standing ecosystem contract.

The current dynamic now plays out like this:

Crawlers from large AI firms sweep across websites, scraping creators' content to feed directly into large language models for pre-training or fine-tuning.

When users ask questions to AI assistants or search engines with AI-generated overviews, the AI outputs synthesized, summarized content directly on the search results page.

Audits from third-party SEO growth communities and independent internet researchers show that OpenAI's GPTBot crawls a single website approximately 1500 times, yet only drives a single referral click back to that site.

When AI-generated summaries appear in Google search results, user click-through rates to original website links plummet to just 8%, and clicks on embedded citations within the AI summary drop to a mere 1%.

This is the widely discussed phenomenon of "zero-click searches."

AI captures nearly all the value, while creators receive almost zero traffic in return.

It is outright piracy — AI takes everything and leaves absolutely nothing for the original content owners.

From every possible perspective, this situation is completely unacceptable.

Yet website operators have been utterly powerless to stop it.

The core issue is exactly what Cloudflare is targeting: mixed-purpose crawlers.

Cloudflare's official complaint confirms that a growing number of AI crawlers deliberately falsify their User-Agent headers, impersonating search engine crawlers that are supposed to deliver traffic to sites, blatantly bypassing robots.txt restrictions to download web pages without limits.

This creates a classic prisoner's dilemma:

A blanket block would keep mixed crawlers out, but also remove your site from search engine results entirely; allowing them unimpeded access will kill your revenue even faster.

Either path leads to ruin.

At this point, only the "Cyber Bodhisattva" can step in to enforce fair play.

02

Based on the official "Configure AI bot policies" whitepaper, Cloudflare is abandoning the old binary "human or AI" classification framework. Instead, it is categorizing all automated web traffic across the internet by behavioral intent and use case, dividing it into three distinct groups:

Search category: Crawlers that build indexes for traditional search engines and subsequently drive traffic back to websites

Agent category: Real-time retrieval crawlers operated by end-users, supporting Retrieval-Augmented Generation workflows

(For example, the ChatGPT-User crawler that activates in real time when a user asks GPT a question requiring up-to-date web data)

Training category: Indiscriminate large-scale data scraping for pre-training large language models

Starting September 15, 2026, all new domain names protected by Cloudflare will automatically have their security policies updated to: Allow Search crawlers, block Training and Agent crawlers by default.

What happens if a major AI company refuses to comply, continuing to deploy mixed crawlers that combine search and training purposes?

Under Cloudflare's rules, they will face the strictest possible penalty: 100% full blocking, with even their Search crawler access completely revoked.

What if crawlers try to sneak past by spoofing their identities?

As a cybersecurity giant built on anti-DDoS expertise, Cloudflare has no shortage of countermeasures.

1. Behavioral Analysis

The WAF will inspect request headers, TLS fingerprints, and even the concurrency patterns of HTTP/2 connections.

With Cloudflare processing hundreds of billions of traffic events daily, it is nearly impossible for AI to disguise itself as human using headless browsers.

2. Reverse Resource Siphoning

Leveraging AI Maze Honeypot technology, Cloudflare can dynamically generate an infinite labyrinth of fake web pages in milliseconds.

Once an AI crawler enters this maze, it will loop endlessly through mountains of garbage content, burning massive amounts of the AI company's compute power and bandwidth costs.

3. Automated Bot Identity Verification

Cloudflare has led the development of an open framework that requires compliant AI bots and automated shopping agents to cryptographically verify their identity and legitimate intent to publishers.

...

Who stands to suffer the most from this policy?

Algorithm firms including OpenAI, Anthropic, and Perplexity will lose access to massive volumes of data from small-to-medium websites and niche vertical communities.

At minimum, they can no longer scrape data for free as before — they will be forced to pay and sign formal licensing agreements with website owners.

The most tangible consequence is a sharp rise in the cost of training large models.

OpenAI, for example, has already separated its crawler ecosystem: GPTBot is dedicated to training data scraping, while ChatGPT-User handles real-time connected queries for the ChatGPT platform.

Google is the only hybrid holdout.

Global AI Crawler Traffic Share (Q2 2026)

For years, Google has exploited its over 90% monopoly share of the global search market to let Googlebot operate with total impunity, using scraped content to train its Gemini model without pushback.

No website dares to block Google, lest they vanish entirely from search results and disappear from the public internet.

Cloudflare even called Google out directly: "They have forced websites into an impossible position: if they refuse to allow their content to be used for AI training, they risk becoming invisible in search results. This has allowed the world's largest search engine to accumulate twice as much data as every other leading AI firm combined."

Come September 15, Google will face an excruciating dilemma:

If it does not split Googlebot, it will lose indexing access to 20% of the public web, triggering a catastrophic collapse in search quality. But if it does split Googlebot into separate search and training crawlers, Gemini will immediately face a crippling shortage of training data.

It is also inevitable that after Cloudflare, many more web infrastructure providers will roll out identical policies.

This strips Google of its unfair monopoly privilege to bundle search and data scraping, forcing it to compete on a level playing field with startups like OpenAI.

03

The internet has long suffered under the weight of uncompensated AI data theft.

Cloudflare's bold move is widely celebrated as a long-overdue reckoning.

But this is far from the full story.

Valuable opportunities are never given away freely.

Even a Bodhisattva cares about revenue, and Cloudflare is only the "Cyber" version.

As we noted at the start, nearly every conflict can be traced back to disputes over unfair profit distribution.

Channeling value flows is always better than blocking them outright.

The advancement of AI is an irreversible megatrend. A mindless, total block would only alienate Cloudflare from the entire market.

In fact, as early as July 2025, Cloudflare tested a pilot "Pay Per Crawl" pricing model.

AI companies are billed based on how much data and how many pages they scrape, with Cloudflare taking a cut before distributing remaining revenue to website operators.

But this model has critical flaws: AI companies feel cheated because they are forced to pay for mountains of useless garbage content from low-quality sites; high-quality content creators are also shortchanged, forced to price their premium work at the same rate as worthless spam.

So leveraging the momentum of its global ultimatum to AI firms, Cloudflare is formally launching two new systems: the Monetization Gateway and the "Pay Per Use" billing model.

Every time AI generates a response that cites content from a Cloudflare-protected website, Cloudflare can use digital fingerprinting and API auditing to confirm: "This specific response drew upon the intellectual property of this particular website."

In simple terms, firms like OpenAI no longer need to negotiate individual licensing agreements with millions of small independent websites (a practical impossibility). They simply integrate with Cloudflare's unified API, pre-deposit a "copyright security deposit," and gain legal access to 23.5% of all public web content.

Every cent of copyright royalty that flows through the system will carry a mandatory toll collected by Cloudflare.

It is effectively a new internet tax.

What business on earth could be more profitable than collecting taxes?

If all AI companies and websites process their data transactions through the Monetization Gateway, even at extremely low commission rates, Cloudflare has the potential to surpass traditional financial giants in scale.

Cloudflare's CEO explained it this way: "In the near future, every time you access a website to retrieve information, you might pay just fractions of a cent. That means our infrastructure must support roughly 10 million financial transactions per second from day one, scaling up to 100 million transactions per second in the future — two full orders of magnitude larger than the existing Visa network."

This paradigm shift will have profound, visible impacts across the entire industry.

A severe data shortage for AI models will arrive much earlier than expected, with data costs skyrocketing overnight.

As Cloudflare (and soon other major providers like Akamai) shut down free scraping access, high-quality public web data will no longer be a free-for-all resource.

Large AI corporations will be forced to massively increase spending to license premium data; smaller firms that cannot afford long-term data licensing bills will be rapidly pushed out of the market.

The creator economy, which has been facing mounting headwinds, will experience a powerful resurgence.

This event is fundamentally a counterstrike by native internet communities against unregulated, exploitative AI capital.

If Cloudflare successfully establishes this as an industry standard, it will undeniably open a new path to sustainability for content creators: shifting from monetizing human attention through ad impressions, to monetizing high-density, high-quality training corpora directly for AI systems.

From this perspective, the relationship between humans and AI remains surprisingly nuanced.

This article originates from the WeChat Public Account "Gelong", authored by Wan Lianshan, and republished here with authorization from 36Kr.