From TPU to self-evolving Agent, how does Jeff Dean judge the next step of AI?

At YC Startup School 2026, Jeff Dean's voice was a little hoarse.

At the very beginning of the interview, he explained that he had lost his voice and sounded different from usual. But this did not distract the audience. Diana Hu, the YC partner sitting in front of him, rattled off a list of names that are enough to go down in computer history: MapReduce, BigTable, TensorFlow, TPU, Gemini.

Any of these projects is enough to become a representative work in an engineer's career. Yet they all appear on the resumes of Jeff Dean and a group of Google engineers around him.

Diana did not turn the interview into a review of achievements. She cared more about another question: When generative AI has swept the software industry, what exactly are people like Jeff Dean, who are best at reconstructing systems from the bottom up, focusing on now?

The answer is not a larger model.

In this nearly hour-long conversation, Jeff Dean repeatedly talked about inference hardware, energy, data movement, context engineering, long-running Agents, automated experimental systems, and how startups can avoid being directly crushed by general-purpose models. What he talked about seems scattered, but there is a very clear main line behind it: The next stage of AI is not just to train models to be smarter, but to put models into a system that can work for a long time, continuously try and error, automatically verify, and continuously accumulate capabilities.

This also means that AI competition is shifting from "who has a larger model" to "who can better organize intelligence".

I. AI is already like a junior engineer, but this is not the most important change

In May 2025, Jeff Dean made a widely discussed judgment: The capability of AI has approached that of a junior engineer.

A year later, Diana asked him how well this prediction had come true.

Jeff Dean's answer was very direct. He thought this judgment was "quite accurate". The progress of models in Agentization, long-process coding and complex tasks is even faster than he expected at that time.

"The ability of models to accomplish increasingly complex tasks is growing faster than I expected," he said.

More notably, this capability is no longer limited to writing code. More and more Agent systems are beginning to enter scientific, engineering and other professional fields. They do not just answer questions, but split tasks, use tools, run experiments, read results, and then continue to act based on feedback.

Comparing AI to a junior engineer makes it easy for people to focus on manpower substitution. But Jeff Dean cares more about another layer of change: When a "junior engineer" can be replicated into dozens or hundreds of copies and work in parallel for days or even weeks, what changes will take place in the way production is organized?

In a traditional team, junior engineers need to get familiar with the business, understand tools, and constantly get feedback. The same is true for Agents. Except that its training materials are no longer just documents, but prompts, tool descriptions, skill files, test systems, evaluators, and the entire context environment.

This has created a new division of labor in AI engineering.

In the past, engineers were mainly responsible for writing code. In the future, more engineers will be responsible for defining problems, building environments, writing specifications, designing feedback loops, and then scheduling a group of Agents to complete tasks.

Jeff Dean's prediction for 2027 is exactly this. He believes that machine learning systems will increasingly participate in improving machine learning systems themselves. They will split goals into sub-problems, automatically run a large number of experiments, compare results, and then combine effective solutions to form a stronger new system.

"As long as there are measurable goals in a field, there is an opportunity to make great progress."

This sentence is the first key to the whole interview.

The first areas that AI automation will break into are not necessarily the fields with the most knowledge, but the fields with the clearest feedback. Whether the code can pass the test, whether the chip layout can reduce the area, whether the model structure can improve the accuracy, and whether the material properties meet the requirements, all these problems have relatively clear evaluation criteria. As long as the evaluator is reliable enough, the machine can test repeatedly at a very high frequency.

Therefore, the truly important unit in the AI era may no longer be a single answer, but a complete closed loop: proposing solutions, executing solutions, measuring results, and correcting directions.

II. What changed Google Search was an arithmetic problem

Many of Jeff Dean's representative works come from a very simple starting point: first clarify the order of magnitude.

In 2001, Google Search still relied heavily on hard disks. Hard disks have large capacity but slow access speed. Jeff Dean and Sanjay Ghemawat made an estimate and found that Google's entire search index at that time could already fit into the memory of all servers.

Today, this sounds like just a storage medium upgrade. But at that time, it meant a completely different system design.

If the index mainly resides on the hard disk, queries need to wait for mechanical seek. As long as the index is put into memory, access latency can drop sharply. The two quickly wrote a new version and put it into the production environment within a few days. Google Search became significantly faster as a result.

This story is most easily packaged as a genius's flash of inspiration. But Jeff Dean's account is more like an engineer stating common sense: When system conditions change and solutions that were not feasible before suddenly become feasible, you should recalculate them.

Many industry innovations happen at such moments.

An old problem has existed for a long time, and people have been accustomed to patching around it. Later, when hardware prices, memory capacity, network bandwidth or model capabilities cross a certain critical point, the original constraints disappear. But most people still use the old architecture, because the old architecture has become common sense.

What Jeff Dean is good at is turning common sense back into assumptions.

He will ask: Why must it be this way? Is today's order of magnitude the same as yesterday's? If we replace the most expensive step, will the whole system take on a completely different form?

This is also his advice to entrepreneurs. Don't just look at where the existing solution is not good enough, but re-examine the problem from the first principle. Can you improve the performance by an order of magnitude? Can you reduce the cost by two orders of magnitude? Can you stop following the industry's default implementation path?

"Sometimes, you just need to squint at a problem, not be anchored by today's solutions, but think about how to solve it from the first principle."

This sentence does not sound mysterious. The real difficulty is that after most people enter an industry, they will quickly learn all the default answers of this industry. Experience helps people improve efficiency, but it also makes people lose the ability to ask questions again.

III. Why did three minutes of speech give birth to a TPU

In 2013, Google's deep learning speech recognition began to significantly outperform old systems. The error rate dropped by half, which was equivalent to the progress of speech recognition in the past two decades concentrated in a few months.

The product team was of course excited. But Jeff Dean first did the math.

If speech recognition really gets better, users will be more willing to use it. Assuming that each Google user only uses speech recognition for three minutes a day, how many servers does Google need to support it?

The result was not optimistic. According to the efficiency of CPUs at that time, Google might need to double the scale of its servers.

This is the starting point of TPU.

It was not because the research team suddenly wanted to make chips, nor to prove that Google had the ability to make hardware, but because a successful model was about to create an unbearable service cost.

This history reveals a frequently ignored rule in AI products: the improvement of model effect does not always reduce costs. On the contrary, the better the effect, the greater the usage, and the heavier the system pressure.

When speech recognition is not good, users rarely call it. System cost is not a problem. When the error rate drops significantly and demand is suddenly released, the computing power constraints originally hidden in the background will surface.

The path chosen by TPU is to make dedicated hardware for the most core computing mode of machine learning. It does not need to run a browser, nor does it need to handle all general-purpose programs. It is mainly good at low-precision, dense linear algebra. This type of computation is exactly at the center of modern machine learning.

The first-generation TPU finally brought gains on the order of magnitude. According to Jeff Dean, it is 30 to 80 times more energy-efficient than the CPUs and GPUs at that time, and its latency is 20 to 30 times lower.

There is also a design scale that is easy to ignore here.

TPU is very dedicated, but not so dedicated that it can only run one fixed model. The team knew that machine learning algorithms would continue to evolve rapidly, so they designed the chip as a more general-purpose linear algebra system. It sacrificed the ability to run Chrome or Word, but retained the space to support the evolution of future algorithms.

This is a balance that is hard to grasp. If it is not dedicated enough, the benefits are not obvious. If it is too dedicated, the hardware will become obsolete as soon as the algorithm changes.

Jeff Dean's judgment on today's inference hardware clearly echoes the TPU back then. He believes that the next round of important opportunities still lies in specialization, but the focus will further shift to low-latency and low-energy-consumption inference.

"Imagine what you can do if latency can be improved by 50 times."

When the model takes more than ten seconds to reply, people will treat it as an occasional consultation tool. When the latency is close to real-time, it can truly enter interactive interfaces, robots, real-time videos, operating systems and continuous decision-making processes.

Waiting is not a small experience problem. Waiting will change the product form.

IV. The cost center of AI is not computation, but data movement

If we want to update a version of "Latency Numbers Every Engineer Should Know" for AI engineers in 2026, Jeff Dean believes that the focus should shift from hard disk seek, cache miss and transcontinental network latency to the data flow inside the chip.

Engineers need to know: what is the bandwidth from main memory to on-chip memory, what is the bandwidth from on-chip memory to multiplication units, how much energy a multiplication takes, how chips are interconnected, and how network efficiency will drop when 500 chips are scaled up to 10,000 chips.

These numbers seem far away from products, but in reality they determine what products can be established.

Jeff Dean gave a very striking proportion. Completing a mathematical multiplication only takes about one picojoule of energy. Moving data from high-bandwidth memory to computing units may cost about 1000 times more energy.

In other words, today's expensive operations in AI systems are often not "computing", but "moving the things to be computed".

This also explains why batch processing is so important.

After a set of model weights is moved from memory to computing units, if only one token is processed, the cost of data movement is entirely borne by that single token. If a larger batch is processed at the same time, the same set of weights can serve more computations, and the energy and bandwidth costs are diluted.

But batch processing and low latency are inherently conflicting. In order to gather a batch of requests, the system often needs to wait. Throughput is improved, but the response of a single user may become slower.

Therefore, many problems that seem to belong to the model layer are actually hardware and system problems. Why training uses large batches, why inference needs KV Cache, why models pursue low precision, and why systems need quantization, all of which are inseparable from data movement and energy constraints.

Jeff Dean is paying more attention to inference recently, precisely because inference is extremely sensitive to latency. A slightly slower training task often only means that the experiment ends later. Every extra second of waiting for an inference task will directly affect user experience and Agent work efficiency.

If an Agent needs to call the model 1000 times in a row, a 50% reduction in single latency will make a huge difference in the completion time of the entire task. Not to mention that in the future, Agents will run for days or weeks.

Therefore, the "energy problem" of AI is not a distant environmental issue. It directly determines whether models can serve more people at low cost, whether Agents can run continuously, and whether the gross profit margin of startups is healthy.

V. The model is just a part, and the context is the Agent's workplace

In the past few years, the AI industry has been accustomed to measuring progress by the number of parameters, training data and benchmark scores. In 2026, Jeff Dean emphasizes everything around the model more.

A truly useful AI system, in addition to the model, also requires retrieval, tools, memory, historical information, execution environment and feedback mechanisms. The model needs to know what tools are available, when to call them, how to split complex problems into a series of actions, and also be able to compare multiple solutions to judge which one is more likely to succeed.

This is why "context engineering" has come to the center of the stage.

Jeff Dean said that the information the model has seen during the training phase is finally "stirred" into hundreds of billions or even trillions of parameters. They are like a pot of thick soup, the knowledge exists, but it is not necessarily clear. The information that is actually put into the current context is more direct to the model and easier to be used accurately.

This leaves an important opportunity for small teams.

Training a foundation model requires massive amounts of capital, data and computing power. But context engineering can start with an API. Entrepreneurs can organize domain knowledge, tool processes, customer data and evaluation standards around specific businesses, so that general models can perform more reliably in a narrow scenario.

Jeff Dean gave an example of his own.

He and Sanjay Ghemawat often optimize the underlying libraries inside Google. These data structures may run in millions of processes, and small performance differences will be amplified by the scale. The traditional approach is that engineers first write micro-benchmarks, measure the current performance, then modify the code, re-run the benchmarks, observe cache usage and performance changes, and then continue to iterate.

The two wrote this set of working methods into an Agent skill. The model learned how to run benchmarks, modify code, compare results, and then continue to optimize based on measurements.

"We just gave the model the methods that humans would use, in a form that the model can use."

This sentence can almost be regarded as the simple definition of context engineering.

It is not a mysterious prompt skill, nor is it just stacking more background materials. It is answering three questions: what steps experts will take to do things, what reliable tools the system has, and how the results should be verified.

When these contents are structured, what the model gains is not more knowledge, but a set of repeatable methods.

This is also why "skill" will become a key asset in the Agent ecosystem. An excellent skill file may encapsulate years of tacit experience of the team. It tells the model what to do first when encountering a certain type of problem, which errors are most common, which tools are trustworthy, and what results count as completion.

The differentiation of future companies will most likely not only exist in model weights, but also in these experiences encoded into workflows.

VI. Why do Agents start to get out of control at the 30th step

Almost all teams that have actually worked on Agents have seen the same scenario.

The first few steps go smoothly. The model can read requirements, call tools, and write code. By the 30th or 50th step, it starts to forget the goal, misunderstand the state, repeat actions, or go further and further in the wrong direction.

Jeff Dean attributes one of the reasons