Free data on the Internet has been completely depleted, and ordinary users have reported that it has long been useless. A former NVIDIA engineer claims that the original sin of Transformer is wasting computing power, and chatbots have squandered all GPU resources.

Right now, the entire Silicon Valley is going crazy over one thing: snapping up NVIDIA Blackwell chips, hoarding power supply quotas, and scaling data centers to the gigawatt level, all just to make chatbots spit out text to users at millisecond-level blistering speed.

But for Neil Movva, who once wrote low-level operators (Kernels) in person at NVIDIA and now founded Sail Research, the entire industry has most likely fallen into a "latency trap" from the very beginning.

Neil experienced the era when NVIDIA guarded every inch of wafer area to fight for an extra 5% for matrix multiplication, and he knows deeply that GPU is essentially a machine built for high-throughput computing, but now it is forcibly turned into a real-time customer service serving human keyboard interactions. In his view, the endgame of AI is by no means low-latency real-time conversation, but background Agents that can run autonomously for hours or even days.

"The best latency is actually zero latency — when you wake up in the morning, the machine has already finished all your work for you overnight." As long as humans are no longer stuck in the middle of the interaction loop, there is no upper limit to the consumption of intelligence.

To drive the cost of this kind of background inference down by 1000 times, Neil is executing an extremely bold "computing scavenger strategy": "There are no bad chips in the world, only bad pricing." In the latest episode of the *Invest Like the Best* interview, host Patrick O'Shaughnessy and Neil calculated from the physical limits of underlying wafers and the inherent flaws of the Transformer architecture all the way to the electricity bill ledger of the power grid, fully showing the future of computing that truly forms a commercial closed loop in the eyes of a low-level engineer.

Quick Overview of Key Points

• The best latency is zero latency: I don't want you to wait at all. When you wake up in the morning, all the work is already done overnight. As long as you still need to participate in the "prompt and wait" loop, humans themselves are the biggest bottleneck for Agents.
• There are no bad chips in the world, only bad pricing: As long as the price is right, I am willing to purchase any chip at any time and anywhere. The market's prejudice against non-NVIDIA hardware is the best arbitrage opportunity for me.
• The "original sin" of Transformer: It forcibly combines an attention layer that is extremely limited by memory bandwidth and a matrix multiplication layer (MLP) that is extremely limited by computing power, which is extremely difficult for a single chip to balance, leading to huge waste of underlying resources.
• The Internet is a one-time subsidy for data: High-quality text across the entire network has been completely sifted through, and the preference feedback from the general public has long been invalid. The evolution of the next generation of intelligence relies entirely on reinforcement learning games in verifiable task sandboxes.
• Buy "cheap" data centers with 95% availability: Traditional data centers pay sky-high prices for 99.99% SLA and generators. But if the service targets long-cycle Agents, with weather model scheduling, intermittent cheap green power can instead become an asymmetric advantage.

The full text of the interview is as follows:

01 / Turn intelligence into an extremely cheap commodity

Host: I think at the beginning of the conversation, it's best to get straight to the point: what exactly are you building, and what can it do now? So, why don't you briefly define for us what the system you are building is, and why it needs to exist?

Neil Movva: Sail Research is a "Token Factory". We provide an API that allows anyone to send us requests and call open-source large language models to complete any task they want. We will provide them with token inference services at an unrivaled price in the market. At the same time, we also support customers to build various Agents on this base. We host so-called "sandboxes", which are long-life Agent virtual machines running in the cloud, specially tailored for Agents that run for hours, days or even weeks.

Host: That means you are in the same industry ecosystem as other companies that provide different types of inference services. You focus on a very specific type of inference, aiming to become the absolute cheapest supplier and promote the popularization of a specific form of intelligence?

Neil Movva: Exactly right. The core theme of our company is "abundance". We hope to deliver the new commodity of "intelligence" to as many people as possible at a cost that almost all industries can afford. We believe that as long as you reduce the cost of something by 10 times, it will spawn a whole new product category, and we are determined to do the same thing for tokens. We think the fact that machines can think is extremely far-reaching, and our current task is to get as many machines around the world running as possible to think.

Host: If we set today's theme as "token cost", is this the right way to understand the problem? Or would you have other ways of expressing it at the beginning?

Neil Movva: Of course. At the moment, token cost is my north star metric: I want to have the lowest single-token cost in the whole industry, and be far ahead. I don't think token is the ultimate measure of workload or intelligence, but it is our common unit today, so this measurement method is very straightforward. I think after tokens, the industry will turn more to "outcomes" — although this is still a relatively vague direction. For example, today when you consume tokens through an Agent, you can't accurately control how many tokens the Agent will consume for inference.

It may think for a while, or call a certain number of tools. The increasingly obvious trend is that we will let the Agent complete a certain work unit, and let it take as many shots on goal as possible, and how many tokens it finally consumes to achieve the goal becomes a dependent variable that depends on the task itself. It's like the Agent starts to manage its own token budget, instead of the company setting a fixed monthly token consumption quota for engineers.

Host: Why do you have the opportunity to solve this problem? At present, the whole world seems to be operating around "more, better, faster, cheaper tokens", and the whole world is trying to solve this problem extremely aggressively. What unique breakthrough point have you seen that the market is inefficient at when trying to solve this problem?

Neil Movva: I think there are two main tailwinds driving the development of our company.

The first one must be the rise of open source, which I have to talk about first. We see that more and more customers and even the broader market are beginning to care about "owning the ownership of intelligence". They want control and sovereignty over the technologies they rely on. This creates a more booming market for customized models, even pure open-source foundation models that no one can take away. You will always own the model weights, and always have the right to deploy them in any way you like. In the past few years, the market for providing large-scale hosting services for these open-source models has been quite active.

But the challenge is that whether it's BaseTen, Fireworks or Together, all these companies are focusing on "low-latency inference". They are pulled in this direction by an extremely important big customer — that is Cursor. About a year ago, this was indeed the right choice; but since six months ago, the situation has changed: people begin to realize that your demand for Agents may be far more than just "low latency".

You want stronger persistence and the ability to handle longer-cycle tasks. Now for me, the future of agentic inference is obviously for long-horizon tasks. The machine will run continuously for hours or even days at a time. At this point, it doesn't matter at all whether it outputs 100 tokens or 10 tokens per second. Relaxing the latency requirement will bring huge efficiency and cost advantages.

02 / The best latency is zero latency

Host: Why are you so sure about this? It seems to me that people always want everything to be as fast as possible. When you are sitting there waiting, you deserve the fastest possible reply.

Neil Movva: My trick is that I don't want you to wait at all. I want it to be initiated actively, running silently in the background. In other words: The best latency is zero latency — when you wake up in the morning, the work has been done overnight, and you don't even need to ask questions. This is the ultimate dream, and we haven't fully reached that point yet.

But more importantly, as long as you still need to participate in the human-in-the-loop of "prompt the Agent and wait for the response", you yourself actually become the bottleneck that limits the Agent to handle more or less work. We want the Agent to operate on the human time scale. You won't micromanage your colleagues every 5 minutes; you will assign them a high-level task, and then check in once a day, or more commonly sync up once a week. In my opinion, this is the future of human-Agent collaboration — close to the time rhythm of human beings themselves.

Host: Tell us more about the early signs that this trend is happening to support the founding of your company?

Neil Movva: The first and most critical point is the concept of test-time compute scaling — that is, as long as you give the Agent more time, it can give better answers.

This theory was put forward about two years ago, but it was not until the launch of Opus 4.5 at the end of last year that it really became a direction we could bet on. Opus 4.5 was the first Agent that was slightly competent for long-cycle tasks. It was actually quite mediocre when it first came out, but looking at the newer models and the work we have done in the open source field, you will find that current Agents can already run continuously for an hour. I won't say it can run for several days now, but running for a full hour is absolutely completely feasible today.

As long as you see that the average number of interaction rounds or task execution duration is constantly lengthening, you don't need many data points to draw that exponential curve, and realize that letting Agents run for longer is extremely valuable.

Host: What market share do you think long-cycle running Agents will occupy in about three years?

Neil Movva: The reason why I am extremely optimistic about this market is that its upper limit is infinite. Long-cycle operation has no people in the loop, so you can consume an unlimited number of tokens in the background at will. In contrast, human attention has a limit.

If you ask me to consume 10 times more tokens on Codex or Claude Code now, I actually can't do it. As long as I sit in front of my laptop, most of my whole day is already highly immersed in the closed loop of writing code. What really has no upper limit is the number of tokens consumed in the background or actively by the system. So in the long run, I think the load ratio of background tasks to real-time tasks will be about 50:50 by the end of this year, but this ratio will become 90:10 in the future, with background load occupying an absolute dominant position.

Host: What are the specific scenarios? What's your favorite example — which tasks work far better as background tasks than tasks with people in the loop?

Neil Movva: The vast majority of deep research is like this. For example, when you want to conduct a comprehensive analysis of not 100, not 1000, but 10,000 or even more data sources to give a definite answer. Or when you want to build an authoritative information index — for example, our customer Parallel Web Systems, they are trying to index the entire Internet and monitor changes across the network in real time.

This extremely exaggerated exabyte-scale task requires a completely different form or scale of intelligence to complete. Deep research is the number one application scenario for us, and then we see the cybersecurity field also catching up rapidly. Think about it, you can indeed generate a huge amount of code, but the methods to break through this code are exponentially more than the methods to generate it.

There are some very good customers who are going all out to build Agents that can break into any software and actively patch it. For example, when Fable or Mythos were first released, the cybersecurity community set off a boom, people let Fable run through every line of code we wrote, looking for vulnerabilities in 20 different ways — including memory errors, business logic errors, network vulnerabilities and so on. All of these require you to write specialized Agents to handle.

You won't just let Fable read the source code once and that's it, but let it actually set up a test environment and conduct pentest on these applications. Later, the industry even began to joke that security has become "proof of work".

To get software security, you essentially depend on how many dollars you spend on Anthropic's API trying to hack into your own system.

This has become the best indicator to measure the security of software, because it is the sharpest spear in the world. And we are increasingly finding that the intelligence frontier here is jagged. It's not that the bugs found by Fable are the full set of all software bugs. Sometimes a very small model can find vulnerabilities that large models miss; Haiku can find some bugs that Fable can't, and vice versa. This leads to a very diverse sampling strategy, where everyone builds various cybersecurity Agents to break into software autonomously so that patches can be applied in time.

03 / For every verifiable problem, the price of tokens is already marked

Host: If you make some more forward-looking and imaginative deductions, what can this kind of Agent with extremely low cost and extremely long running time unlock? We just talked about some very real cases, such as deep research, cybersecurity and so on. But if you think a little more grandly — what new product categories will this kind of reasoning unlock? If you achieve the greatest success, imagine what kind of world that will bring?

Neil Movva: Of course. For individual users, what excites me the most is the concept of "proactive intelligent Agent". You can imagine a Siri that always runs in the background, it understands all the emails and text messages you receive throughout the day in real time, has an extremely encyclopedic global view of your life, and always knows how to help you.

Current AI is still in the stage of single-point solutions, so you end up having to do a lot of prompt interaction. Current Siri is not proactive at all. And this is the problem we can completely solve through abundant and cheap reasoning. If you trust the machine enough and are convinced that it is reliable and trustworthy (such as protecting privacy), you can even imagine that the machine can deeply understand your interaction pattern with it, and every time you open your phone, it has actively presented the next step for you. Can we build an accurate model for the user's "next behavior"?

My judgment is: absolutely yes. And the key to achieving this goal lies in extremely cheap intelligence. You must be willing to consume tokens recklessly without any promise of immediate return. That's the breakthrough point. From a long-term perspective, we now have a form of intelligence that can tackle any "verifiable problem". The so-called verifiable problems cover the vast majority of software engineering, a large number of formal mathematical proofs, and may also include scientific discoveries. These are all verifiable in a relative sense.

At present, all these things essentially have a hidden dollar price tag. It's like "how many tokens can you mobilize to get this done?" And we are actually starting to make the dollar cost of these long-cycle tasks visibly reasonable. It's no longer millions of dollars, but thousands of dollars; in the near future, the cost of finding a definite answer to any scientific problem or research topic may be only a few hundred or even tens of dollars.

So if we imagine that future, humans will eventually be limited only by the problems we can put forward? Basically it's the problems we can put forward. The current models are already ready to go — even for a broad high-level problem, the model can actively track every possible subsequent branch and advance almost entirely on its own. The only problem now is: what's your token budget? And we will completely solve the token budget problem.

Host: What about those unverifiable tasks?

Neil Movva: I basically classify all human "human taste" into this category. We have not solved the problem of human taste yet, and I don't even know if it is fundamentally solvable. I will be excited if there is a pleasant surprise; but at present we focus on highly quantitative problems, and we leave the literary grace of writing and the beauty of art to humans.

04 / Lesson learned from NVIDIA: Give up the ultra-low latency of private cars and drive the "big bus" of throughput

Host: Okay, now let's talk about this extremely sophisticated full technology stack you want to eventually build — build this huge Token Factory and become an extremely low-cost intelligence supplier. I know you think about this problem from the levels of software, hardware and electricity. Tell us your grand plan, why