In the Agent era, why are some people starting to rebuild Google?

When Google has already indexed almost the entire open internet, is there any need to rebuild a search engine from scratch?

Andrey Styskin, former head of Yandex's search, AI and cloud businesses, gives the answer: Yes. Only this time, the most important users of search engines may no longer be humans.

Recently, a startup named Keenable officially came out of stealth mode, and announced the completion of a $26 million seed round financing led by Accel. Styskin is the company's co-founder and CEO. After nearly 20 years at Yandex, he joined Amazon AGI to take charge of web infrastructure; the other co-founder, Matthias Petri, also came from Amazon AGI and previously participated in the construction of the web grounding infrastructure behind Alexa.

This is a team of only about 15 engineers, but the work they are doing is quite "heavy": re-crawling the internet, building its own Web Index, and then turning it into a search infrastructure dedicated to serving AI Agents.

According to the disclosure on Keenable's official website, its index has covered more than 100 billion documents, with a query latency of less than 250ms (p95) in the eastern United States; for large-scale customers with over 100 RPS, the price can be as low as $1 per thousand requests. More notably, it claims that its API has been accessed by several AI Labs and inference service providers in production environments, for both model training and runtime retrieval, though it has not disclosed the specific names of its customers.

Styskin has set a rather ambitious goal for it: to become the "next Google" in the AI Agent era.

Agents search the internet in a completely different way

Keenable is different from the common AI search products on the market today. It does not want to make another AI search box for ordinary users, nor does it simply add a large language model layer on top of Google or Bing Search API. Keenable re-makes Crawl, Index, Retrieval and Ranking from the bottom layer, and then delivers search capabilities to Agents through REST API, MCP Server and CLI. The company even states directly that current Agents use a set of "human-optimized" web access tools, and it hopes to make web knowledge access for Agents cheap enough to be almost equivalent to calling the model's own knowledge, so that Agents will no longer "be reluctant to search" due to cost and latency constraints.

This is also why a startup that has just come out of stealth mode is willing to take on the extremely costly task of re-indexing the entire internet.

In an interview with TechCrunch, Styskin explained that the service cost of scanning the entire internet is extremely high, and if the index structure is not optimized for specific tasks, the cost will quickly get out of control. One of the real technical problems is whether the search space can be narrowed down fast enough for a single Query. He even used the phrase "painfully expensive" to describe the cost of building a huge Web Index.

But behind this expensive investment, there is actually a more fundamental change than "search quality": humans and Agents use the internet in completely different ways.

Over the past two decades, the typical workflow of a search engine is:

Human puts forward a Query → Search engine returns a result page → User browses links → Clicks → Reads the webpage.

Therefore, traditional search has long been optimized around human behaviors: which result deserves to be ranked higher, what users are more likely to click, and whether users will continue searching after finding the answer.

The workflow of Agents may become:

Receive Task → Search → Fetch → Read → Discover information gap → Modify Query → Search again → Cross-verify → Final Action.

A Research Agent can perform dozens of consecutive searches to answer a complex question; a Coding Agent needs to constantly look for the latest documents, Issues, GitHub repositories and Changelogs; a Sales Agent may query the company's official website, financing news, recruitment information and executive dynamics at the same time to judge whether a customer is worth following up. For these Agents, "search" is no longer an action triggered by a user actively clicking a button, but a step in the task execution loop.

This also means that Agents do not need to face "ten blue links" like humans do. What they really need is context that is complete enough, fresh enough, from credible sources, and as streamlined as possible to facilitate the model's subsequent reasoning.

Styskin therefore believes that Agents will form a feedback flywheel different from the one Google learned from human click behaviors. Traditional search focuses on clicks, ranking and user satisfaction, while Agent Search will pay more and more attention to Recall, Freshness, Latency, QPS, machine readability, Token Efficiency, Provenance, and the total cost of completing a task in the end.

In other words, search engines used to be optimized to "help people find webpages"; what Agent Search aims to solve is more like "finding the information sufficient to complete the task at the lowest possible cost".

Keenable is not the first company to spot this business opportunity

Exa, Keenable's most direct competitor, already has an independent Web Index of the same magnitude. According to official data from Exa, its vector database currently covers 100 billion documents, tracks about 1.4 trillion URLs, crawls billions of documents every day, and has more than 500,000 developers on its platform. Cursor, AWS, Groq and others are all on its customer list. Walden Yan, co-founder of Cognition behind Devin, also publicly stated that traditional search solutions cannot meet the demand, and Exa has been used in multiple links of Devin.

In May this year, Exa completed another $250 million Series C financing with a valuation of $2.2 billion. It even predicts that the number of web searches initiated by AI Agents will exceed that by humans this year, and believes that the search volume generated by LLMs in the next few years may reach a thousand times of today's Google search volume. The latter figure is obviously the company's forward-looking judgment, but capital has already started betting on this possibility.

Exa's product has long been more than just a Search API. It has covered Search, Contents, Deep Search, Agent and Monitors: it can complete low-latency searches in about 200ms, and can also launch multi-step Agents for in-depth research. Its Contents service also extracts webpage content related to the task, providing so-called token-efficient highlights instead of feeding the entire webpage to the model indiscriminately.

This also leaves a very realistic question for Keenable, which has just come out of stealth: Since Exa already has a 100B Index, clear customers, and a more complete product matrix, why is it worth indexing the internet all over again?

At present, the two companies have different publicly visible priorities. Exa has extended from underlying Search to research, Agent and structured data services; Keenable focuses more on being the underlying Search Infrastructure for AI Labs and inference platforms, putting low latency, high QPS and scaled call costs in a very core position. The price of Keenable for ordinary developers is $4 per thousand requests, and the dedicated capacity for over 100 RPS can be reduced to $1; the current public base price of Exa's Search API is $7 per thousand requests. Since the number of results, content processing and service levels of the two are not exactly the same, these figures cannot be simply converted to say which one is "several times cheaper", but they can already show that: when search enters the high-frequency call loop of Agents, the single-call cost itself has become a competitive indicator.

Similar competition has also emerged in China.

Bocha AI provides Web Search and AI Search for AI applications, and search results can be directly delivered to Agents through Function Call, MCP and other methods; Xinliu Search positions itself more directly as "linking the real world for intelligent agents", providing Web Search, Web Fetch, Image Search, and supporting MCP, Skills, OpenClaw, Claude Code, LangChain and AutoGPT. The latter even puts "extremely token-saving" on the product homepage: it first filters advertisements and redundant webpage content, and then delivers the re-ranked refined snippets to the model.

Major tech giants are also revamping their original search capabilities. Tencent Cloud's connected search API, based on public internet resources and Tencent's content ecosystem, is redesigned for large models from data inclusion, recall to fine ranking, providing minute-level updates and a fastest 300ms return, and it further launched the connected search MCP for Agents on August 24. Alibaba Cloud's OpenSearch has also provided an independent connected search API, which allows LLMs to rewrite Queries, filter search results, and choose to return abstracts or the full text of up to 3000 characters.

The starting points of these types of products are different: Exa and Keenable hope to master Crawl, Index and Retrieval; Bocha and Xinliu are competing more directly for Agent Search APIs and processed Context; Tencent and Alibaba have their original search accumulation, content ecosystem and cloud platform entry.

But they are all vying for the same position: who can become the default interface for Agents to access the internet.

Agent search may bring the next huge bill

The reason why Agent Search may become an independent layer of infrastructure has a very practical reason: Agents can search extremely frequently.

A traditional user's one search behavior may only generate a few Queries, while an Agent may search dozens or even hundreds of times to complete a complex task. Assuming that a task calls Web Search 100 times, according to Keenable's price of $4 per thousand requests for ordinary developers, the search request cost alone is $0.4; according to Exa's current base price of $7 per thousand requests, the cost is $0.7. After entering the production environment, Fetch, webpage processing, model Context and Reasoning Tokens will also be added to the cost.

Therefore, the truly cost-worthy unit in the Agent era may change from cost per search to search cost per completed task.

The billing structure of Alibaba Cloud's OpenSearch can well illustrate this point. One connected search may trigger the large model to perform Query Rewrite in addition to Search Invocation; result filtering will also consume model Tokens. Its API return fields even separately record the input and output Tokens of the rewrite model and the filter model. A seemingly simple "connected search" has actually become a series of calculations of Query understanding → rewriting → retrieval → filtering → Context sorting.

This is also why more and more Agent Search products are beginning to emphasize Token Efficiency.

What Agents really need is usually not the full text of a 5000-word webpage, but several paragraphs of content in it that can help the current task continue reasoning. If the search layer can first delete advertisements, navigation, recommendation modules and irrelevant paragraphs, and only send the most relevant passages into the context, it will not only save the search cost, but also further reduce the subsequent model's input Token and reasoning cost.

Thus, a new cost chain begins to form:

Search Quality → Context Quality → Token Consumption → Agent Cost.

Search Infra and model Infra begin to appear on the same bill.

But cost is only the first layer of changes brought by Agent Search. The deeper layer is that what Agents ultimately need may no longer be "search" in the traditional sense.

For example, let the Agent complete such a task:

"Find European AI companies that have completed Series A financing in the past six months, whose founders are from Google, and are currently recruiting inference engineers."

It is very likely that there is no single webpage on the internet that directly gives the complete answer. Financing information may be in news, founder resumes are on the company's official website or LinkedIn, and recruitment status is hidden in the Careers page. Agents must complete Search → Fetch → Join → Filter → Verify on their own.

This is also the most noteworthy part of WebQueryLanguage that Keenable is still developing. Styskin revealed in the interview that he hopes AI systems can combine multiple web sources to answer questions that no single page can provide a complete answer to. This product has not been officially released yet, and there is not enough technical details to prove that it already has capabilities similar to database Query Planner, but the direction is very clear: search systems are trying to turn the open web from a "collection of webpages" into a machine-queryable dataset.

Search may thus continue to evolve from find documents to query the Web.

If this route is established, the end point of Agent Search may not be a "smarter Google API", but more like some kind of SQL for the open Web: Agents no longer care which webpage ranks first, but put forward the problem they really want to solve, and the underlying system is responsible for finding entities from different sources, combining facts and retaining the chain of evidence.

This may also be the biggest change that Agents bring to search: webpages still exist, but "webpage" may no longer be the most important organizational unit for machines to acquire knowledge.

Rebuilding the internet for Agents is not that easy

However, turning the web into an external database that Agents can query at any time is not as simple as it sounds.

The first problem is who provides data for this "database". In the Google search era, there is an exchange relationship that has lasted for more than 20 years: websites allow Googlebot to crawl content, and Google sends users and click traffic back to websites. Agents may read ten websites and then directly synthesize the answer, or even continue to perform the next operation on behalf of the user, and users do not need to open the original webpage from beginning to end. Content producers provide knowledge, but may not get access volume, subscription conversion and advertising revenue.

This contradiction has already appeared at the product layer. Cloudflare Pay Per Crawl allows website owners to choose to open to AI Crawlers for free, block them completely, or require crawlers to pay. The problem behind it is very straightforward: AI companies need to continuously read the web, while Publishers need to re-establish value exchange after content is consumed by machines.

This brings a paradox to independent Web Indexes such as Keenable and Exa: The more successful Agent Search is, the more likely websites are to lose traffic; the more actively websites restrict AI Crawlers, the more expensive it will be to re-build and maintain the Web Index.

The second problem comes from security. Traditional SEO mainly affects what users "see", while the results of Agent Search may further affect what machines "believe and do". If malicious webpages enter the Agent's context through search, and then use indirect Prompt Injection to affect an Agent that has permissions for GitHub, email, enterprise systems and even payment, the consequence of a retrieval error will no longer be just recommending a spam website.

Therefore, the metrics that Agent Search ultimately needs to handle may continue to increase: in addition to relevance, freshness and latency, they also include provenance, trust and security. Search engines used to be responsible for providing users with information entrances, and the future Agent Search will, to some extent, act as the "perception layer" of Agents for the real world.

In China, this matter has another special difficulty: The Chinese internet itself is not a completely open web.

At present, there are already Agent Search APIs such as Bocha and Xinliu in China, and Tencent and Alibaba are also opening up connected search capabilities. But from public information, it is hard to find a domestic startup that, like Keenable and Exa, clearly takes "independently crawling the entire web + building a 100B-level autonomous Index" as its core technical and commercial barrier, and continuously discloses infrastructure indicators such as Index, QPS and Latency.

The problems behind this are not only about cost. A large amount of valuable real-time Chinese information exists in WeChat, Xiaohongshu, Douyin, Zhihu, e-commerce platforms and various apps, which do not naturally belong to an Open Web that can be crawled uniformly. Even if a company's public webpage index is large enough, it does not mean that it has truly seen the Chinese internet.

The product design of Tencent Cloud's connected search has already reflected this difference: in addition to the entire network's public resources, it also integrates Tencent's own content ecosystem such as Tencent News, Sogou Encyclopedia, and Penguin Account.

Therefore, a 100B Index does not equal seeing the Chinese internet.

Agent Search in the US is more competing for who can re-index the Open Web more completely and at lower cost, while the Chinese market has an extra layer of problem: who can connect more closed and semi-closed content ecosystems. Platform authorization, copyright, anti-crawling mechanisms and the commercial relationships between content platforms may all become barriers beyond search algorithms.

A Chinese Agent with an index of hundreds of billions of webpages may still not know that a certain topic has just gone viral on WeChat Official Accounts, or fail to notice that a new product is exploding rapidly on Xiaohongshu. For Chinese Agents, the size of the Index does not represent the real search capability, and how much of the real Chinese internet it can access is more important.

Summary

What Keenable is betting on is actually a very clear business: When Agents access the web more and more frequently, someone will be willing to pay for search infrastructure that is faster, cheaper and more suitable for machine consumption. To this end, it is willing to bear the high cost of re-crawling, indexing