Jeff Dean's last conversation before leaving his position: I underestimated AI, and I have clearly recognized the only viable path for entrepreneurs.

A year ago, Jeff Dean, Chief Scientist at Google, predicted at the AI Ascent 2025 summit that by 2026, AI systems capable of working around the clock and rivaling the capabilities of junior software engineers would likely emerge.

A year later, exactly 6 days ago, he admitted in a YC interview that he had underestimated the pace of AI progress. The models' ability to handle complex tasks has grown far faster than he anticipated at the time. From Jeff Dean's perspective, how fast will AI continue to advance in the next stage? And how can startups survive in an era where general-purpose models keep expanding their capability boundaries?

In the early hours of today, this interview took on a whole new level of significance. Jeff Dean announced that tomorrow will be his last day at Google. After 27 years of service at Google, the legendary engineer known in Silicon Valley as "the programmer among programmers", who was deeply involved in the construction of Google's system architecture and AI technology, will co-found Discovery Loop, a public welfare company focused on cutting-edge research in machine learning, science and engineering, with long-term partners Sanjay Ghemawat, Oriol Vinyals and Quoc Le. Google will continue to cooperate with them as the founding investor and cloud computing partner.

According to WIRED, this startup idea only surfaced a few weeks ago. To retain this core team, Alphabet CEO Sundar Pichai even tried to persuade them to "keep their employee badges" in multiple meetings. But in the end, the group decided to leave the large corporate system in exchange for the fun, speed and freedom that only a startup can offer.

Screenshot from: https://x.com/JeffDean/status/2085035498222002595/photo/1

In his farewell letter, Jeff Dean wrote that he witnessed Google grow from a company with only 25 people to a tech giant with more than 190,000 employees. Today, 13 of Google's products have more than 1 billion users. From search, email, translation and video to large-scale computing, autonomous driving and AI systems, the technologies he helped build run through almost the entire development history of Google.

An important reason that prompted them to leave is precisely the inertia that large companies cannot get rid of. As Oriol Vinyals said, within a large organization, to push through any radical change, you must first overcome layers of resistance; they want to build something different. Interestingly, so far, this new company has not even had time to recruit employees or rent an office. As for who will be the CEO, after a short pause in the team, everyone turned their eyes to Jeff Dean — "I guess it should be me," he said.

Therefore, this interview released on the eve of Jeff Dean's departure is like his concentrated judgment on the next stage of AI at the turning point of his career. In the program, he discussed with Diana Hu, YC Partner, the paradigm shift of AI from "model-centric" to "context engineering", the huge opportunities emerging in inference hardware, and how entrepreneurs can find a truly worthy niche to survive in an era where general-purpose models are getting more and more powerful and more and more applications may be directly swallowed by models.

This article is organized based on the interview video and edited by InfoQ.

TL;DR Version

Q: Last year you said AI systems with capabilities close to junior engineers would appear by 2026. A year has passed, has this prediction been proven wrong?

A: It's pretty accurate, but I underestimated one thing: the models' ability to handle increasingly complex tasks is growing much faster than I expected. Moreover, this capability is spilling over into fields beyond coding, and Agent-based systems are beginning to truly show their strengths.

Q: What's your bold prediction for 2027?

A: Deep learning systems will achieve fully automatic problem decomposition and automated experimental loops: break problems into sub-problems, automatically run experiments, integrate results, and get an improved system. And this will not only apply to machine learning, but also to any scientific and engineering field with measurable goals.

Q: In 2001, Google put the search index into memory. What is that "put into memory" moment in 2026?

A: High-performance, low-energy dedicated inference hardware. Inference is the key to making Agent systems accessible to more people, and dedicated hardware can bring 50x latency improvement and 30-80x energy efficiency improvement. What was saved back then was the physical latency of disk addressing, and what is saved today is the redundant steps of general-purpose computing. But the stakes are higher this time, because Agents need thousands of consecutive inferences, and each latency accumulates.

Q: What is the biggest misconception people have about AI right now?

A: Everyone thinks that Agents only run temporary tasks for an hour or two, but with a sufficiently powerful underlying model, you can let Agents run continuously for days or even weeks.

Q: What exactly did that napkin math for TPU's birth calculate?

A: In 2013, speech recognition performance improved dramatically. We did the math: if each user speaks to their phone for three minutes a day, CPUs simply can't handle the load, so we built the TPU. A few years later, when Transformer came out, it just became the foundation.

Q: Among the list of latency numbers in the AI era, which number is the most memorable?

A: A single calculation consumes about one picojoule of energy, but moving data a short distance into the processor consumes 1000 times that amount of energy. This gap means you have to do batching, combine many samples for processing, and amortize the cost to "1000 times divided by the batch size".

Q: How should founders choose the field to work on?

A: Use the current most powerful general-purpose model to test the field you want to work on. If the model success rate is 0% or 1%, that's a good signal; if it's 20%, that's a dangerous signal, indicating that the capability has already taken root, and the next generation of models will cover it soon.

Q: When Agents write all the code, what skills will become scarce?

A: Taste, knowing what problems you want your Agents to solve. First, rely on the experience of solving a large number of problems. Second, write down ten things you think may be important in the next 12 months, and come back a year later to review which ones have really become important. Third, do crazy thought experiments, such as "If transistors make mistakes 20 times a day instead of once every million years, how should the system be designed?"

The High-Performance, Low-Energy Inference Hardware Revolution

Diana: Last May at the AI Ascent conference, you said that around 2026, AI systems capable of working around the clock and with capabilities close to junior engineers might appear. How close are we to that prediction now?

Jeff: I think models have made extremely great progress in handling long-running Agent-based coding tasks. It seems obvious now that they are already quite capable. Depending on your precise definition of "junior engineer", this prediction is quite accurate at the moment.

Diana: What did you underestimate back then?

Jeff: I think the ability to handle increasingly complex tasks is growing much faster than I expected. Moreover, in fields beyond coding, these Agent-based systems are also beginning to truly show their strengths, and I think this will be an important trend in the future.

Diana: What do you think the 2027 version of the prediction will be?

Jeff: I think you will see more automation of machine learning systems themselves. Specifically, let machine learning systems improve their capabilities by running a large number of experiments, break problems into sub-problems, run these sub-problems in a tight automatic experimental loop, then integrate the results, and get an improved system from this fully automatic problem decomposition and automated experiment. And this applies not only to machine learning, but also to other fields of science and engineering. Basically anything with measurable goals can make great progress today.

Diana: In 2001, Google Search was still running on hard disks. You and Sanjay did the math and realized that at some point, the entire search index could finally fit into the memory of all running computers. You made that radical decision, and basically pushed a brand-new search version based on memory instead of hard disk to the production environment with Sanjay in just a few days. It is this incident that made Google what it is today.

History always repeats itself. So at the current moment in 2026, what is that "fits into memory" moment? What is that turning point that everyone in the audience should think about and design for?

Jeff: The situation is somewhat different, but what I want to say is that you will see more and more high-performance, low-energy inference hardware systems emerging. Because everyone now realizes that inference is the key to making these Agent systems accessible to more people. Latency really matters, and hardware specialization is the core way to make you better than general-purpose computing devices like GPUs or CPUs in both energy efficiency and latency.

Diana: So you mean we might not have to wait anymore?

Jeff: Exactly, imagine what you could do if latency could be improved by 50 times?

Diana: There are about 6000 people in the audience. Which assumption about AI in their minds is actually already wrong?

Jeff: I think it may be that people have not fully realized the potential of how long Agent systems can run. Everyone always thinks that Agents only run temporary tasks for one or two hours. But in fact, in certain problem domains, with a sufficiently powerful underlying model, you can let these Agents run continuously for days or even weeks, to truly solve those complex, long-term problems. This is not just a difference of "waiting a little longer", but a qualitative change.

Diana: What tasks that ran for several weeks have you actually run? What do you let Agents solve?

Jeff: For example, you can ask an Agent to implement a brand new version of software, rewrite it in a different programming language, which may have better security or performance features. It can really take this task seriously and complete it quite reliably.

The Origin of TPU

Diana: Everyone has long known that you are particularly good at "napkin math". There is a story about you that when Google's speech recognition was first launched in 2013, you made an estimate: if every Google user speaks to the speech recognition system on their mobile phone for three minutes a day, the entire server scale would have to double, and the cost would be staggering. So you built a custom chip, and that's the origin story of the TPU.

Jeff: At that time, the deep learning-based speech models we trained started to perform very well, but the computational overhead was much more expensive than the old speech systems, though it cut the error rate in half. This is equivalent to condensing 20 years of progress in the field of speech recognition into just a few months, we just adjusted the model, scaled it up a bit, and got better data. So we started to worry: if speech recognition performance is greatly improved, people will use it much more frequently.

So that rough calculation was actually about: what would happen if people started using speech recognition more often to dictate emails and talk to their phones? After the calculation, we found that CPUs simply couldn't handle the load at that time, so we built the TPU, a chip specially designed for low-precision dense linear algebra, which is the core of almost all machine learning algorithms today.

Although it can't do anything except linear algebra, it can't run Chrome or Word, it doesn't matter for machine learning inference. A few years after this chip was launched, its energy efficiency was 30 to 80 times higher than that of CPUs and GPUs of the same era, and its latency was 20 to 30 times lower.

Diana: You certainly couldn't have predicted back then that TPU would become so foundational when the Transformer architecture came out later, after all, Transformer appeared several years after TPU was invented.

Jeff: That's exactly why we made it a general linear algebra system. We knew that machine learning algorithms were still evolving, so we couldn't over-specialize, but we had to specialize enough to get huge performance gains. So we made large multiplication units, high-speed memory, high-speed interconnection, and later TPUs can efficiently coordinate hundreds of chips to work on the same problem. We iterated generation after generation like this until today.

Diana: If anyone wants to be a future founder, what kind of "napkin math" should they do to create something as influential as TPU?

Jeff: I think, think about what problems and bottlenecks you see in the field you are thinking about, and then ask yourself: is there a very different way of thinking that can bring an order of magnitude or even two orders of magnitude improvement to the solution? Sometimes you just squint at a problem, don't get anchored by "how this problem is solved today", but think about how you would solve it from first principles, and you can come up with some really good ideas that no one has thought of.

The "Latency Numbers" List for the AI Era

Diana: Many years ago you wrote a very famous list called "Numbers Every Programmer Should Know About", which lists the time spent on various system operations: how long a cache miss takes, how long a disk seek takes, how long a network packet takes to travel from California to the Netherlands... This list has since become the bible for countless distributed system engineers. Now, it's time to update the AI version.

Jeff: If you look at what's most important in today's AI systems, first you have to know the bandwidth between main memory on the accelerator and on-chip memory, and then to the multiplication units. You have to know how much energy a multiplication operation consumes, you also have to know the interconnection bandwidth between chips, and how many chips you can connect under this bandwidth. On a larger scale, when you need to communicate with 10,000 chips instead of 500, to what extent will the network bandwidth degrade? These numbers are the underlying constraints that truly determine how you think about problems, and they directly determine what kind of ideas you will use to solve what kind of problems.

Diana: You said that now the ruler for measuring everything has become energy. A calculation or mathematical operation consumes roughly one picojoule, but the energy consumed by moving data and doing data I/O is 1000 times that.

Jeff: Exactly. It's just the short path of moving data from the HBM on the accelerator into the processor to make it actually compute, the energy consumption differs by three orders of magnitude.

Diana: This gap is quietly determining which products can be built and which algorithm paths can work. Since data movement is so expensive, if a founder comes to you and says he has a "model problem", but essentially it is an energy problem or data I/O problem. How do you tell the difference between the two?

Jeff: That 1000x difference you mentioned actually shapes a lot of decisions in machine learning. Without this 1000x gap, you don't need to do batching at all. But the reality is that you have to combine many samples and many tokens for processing to amortize the cost of data movement, instead of paying 1000 times the price, you pay the energy cost of "1