Fei-Fei Li's Dialogue with Jeff Dean: AI Can Accelerate Science, But Not Infinitely

In the closing keynote speech at Stanford HAI, Jeff Dean introduced his new company Discovery Loop, which he co-founded after leaving Google. Its mission is to build AI systems that can automatically solve important problems in machine learning, science and engineering: proposing experiments, running them, evaluating results, deciding the next step, and letting each round of experiments feed into the next. Later, Li Fei-Fei and Jeff Dean held a dialogue around the differences between "self-improvement" and "recursive self-improvement", the impact of AI solving difficult mathematical problems, the origin of Transformer, the accuracy and inaccuracy of technology predictions, as well as the anxiety of this generation of young people when facing AI.

The following is the full transcript of their dialogue:

Li Fei-Fei: Let's talk about Discovery Loop first. Why set up such a company right now?

Jeff Dean: Our mission is to build AI systems that automatically solve critical problems in machine learning, science and engineering. We believe this will ultimately bring far more discoveries to the world. Many disciplines share a common cycle: proposing experiments, implementing and running them, observing results and taking measurements, then deciding the next step, which usually means running another experiment. We want to automate this kind of experimental cycle. If we can shorten the iteration time, automatically propose and run a large number of experiments, learn from each evaluation to make subsequent experiments better, and improve both the quality and quantity of experiments at the same time, breakthrough results are likely to be achieved.

Li Fei-Fei: Why make it an independent public benefit corporation?

Jeff Dean: We believe that if technology is developed the right way, it can have an amazing positive impact on the world, but everyone must work to ensure that this positive impact comes to fruition. That's why we registered the company as a public benefit corporation, with a public mission to develop technologies that solve important social problems, make the benefits of these technologies widely accessible, and improve people's lives.

Li Fei-Fei: What about the team?

Jeff Dean: I have three amazing co-founders: Quoc Le, Sanjay Ghemawat and Oriol Vinyals. We have worked together for about 14 to 30 years, and we love solving hard problems together. We have been involved in Google Search, ad systems, Gmail, Google News, Google Translate, large-scale machine learning infrastructure, TPUs, model distillation, MoE, neural architecture search, as well as projects including AlphaChip, AlphaFold and neural weather forecasting. We have also managed large-scale teams: I led Google Brain to 600 people, and later the Google Research and AI team had around 4400 people. But then I found myself spending a lot of time in Q2 operating expenditure meetings, so I decided to try a different approach. We are also very proud that many members of our team later co-founded companies like Anthropic, OpenAI and Physical Intelligence.

Li Fei-Fei: Why now? What technical conditions are ripe?

Jeff Dean: Over the past 10 to 15 years, the performance of the hardware layer has improved significantly, with progress in TPUs, GPUs and more specialized hardware; software abstractions are better, enabling more efficient mapping to hardware; model architectures have continued to improve from word2vec, seq2seq, Transformer to MoE; training algorithms are better, and distillation can produce high-quality small models; inference algorithms have also made great progress, allowing people to use more computation during inference, chain-of-thought reasoning, multiple attempts and checks to get better results. But good foundation models are not enough. You need agents paired with models that can effectively use tools to complete complex tasks, and especially excel at creativity: not just repeating what has been done before, but integrating many insights to come up with something truly new, and verify whether it works. Agents and models should also learn from previous experiments and experiences, including the ability to learn from the most relevant problems after solving 5000 problems.

Li Fei-Fei: Where are the bottlenecks of the traditional experimental cycle?

Jeff Dean: There is a lot of manual work at every stage of the traditional cycle. When solving larger problems, you have to break them down into sub-problems, and the decomposition itself is very difficult; many people run experiments and try different methods for each sub-problem, leading to high coordination overhead, resource conflicts and systemic effects. Worse still, once a large project is launched, the early decomposition structure is very difficult to modify. What we are thinking about is: a small number of domain experts, paired with a system that can automatically decompose problems into interesting sub-problems and solve those sub-problems automatically; humans can guide the decomposition, point out important sub-problems, and allow a small team to solve high-impact problems. The prerequisite is that the infrastructure is in place: thousands of automatic experiments, the system can understand the value of running this experiment versus that experiment, use reinforcement learning or evolutionary algorithms to make automatic decisions, allow early decisions to be revisited more flexibly, and every early experiment can provide information for future experiments.

Li Fei-Fei: What have you done since the company was founded?

Jeff Dean: We conducted 400 interviews in the first three weeks, many of which were at Cafe Venetia on University Avenue, to recruit outstanding talents; we looked for office space, and now we are located in the old Zibibbo restaurant; we built our own agent harness for all development and automation. It has collaborative features: you can talk to the model and ask it to solve problems, you can also invite teammates to collaborate with the model and agent together, and others can leave comments or propose design plans to the agent. We built an agent-first infrastructure, with the goal of scaling up to tens of thousands or hundreds of thousands of agents doing work, instead of just hundreds of people doing work. We have about 20 self-improvement cycles running, evaluating how to make experimental iterations more effective and faster, and how to learn from previous results.

Li Fei-Fei: Which fields will you focus on first?

Jeff Dean: We will first focus on science and engineering problems where the entire cycle can be completed in digital form, because they are the easiest to accelerate. Physical processes will slow down the cycle, and some fields require hybrid methods. Our current priorities are machine learning research, machine learning engineering itself, and hardware design. We have expertise in these three fields, and they can also unlock other opportunities. For example, in healthcare, if we have a system that can automatically solve machine learning engineering tasks, we can point it to problems in healthcare, materials science and other fields, and ask it to develop a model for a specific task. This is self-improvement, but not recursive self-improvement. Healthcare is a long-term important direction, but at this stage we will first work on the most mature fields with fully digital cycles.

Li Fei-Fei: Let's clarify "self-improvement" and "recursive self-improvement". Many people use them interchangeably.

Jeff Dean: A self-improvement system is: you give the system a goal, usually with measurable metrics, such as model accuracy; the system tries different model architectures to improve the accuracy. Many machine learning algorithms can do this, but this is not recursive in itself. For example, if you give the system a large amount of MRI data and corresponding results, and ask it to develop a model for interpreting MRI images, the system will automatically try different methods and see if the accuracy improves. But this does not improve the underlying system that "proposes what to try next". Recursive self-improvement means: can you use this self-improvement method to improve the underlying system itself that generates improvement suggestions.

Li Fei-Fei: But RSI is being heavily hyped, as if it can accelerate infinitely. Where are the physical limits?

Jeff Dean: The way humans improve model capabilities is to have capable researchers and ML engineers run a large number of small-scale experiments together, judge which ideas work and which don't, and then integrate the effective ones into the next generation of models. That's how Gemini 2 came after Gemini 1. Automation can speed up this process: if a human runs an experiment that ends at 2 a.m., they may not check the results until 8 a.m.; an automatic system can check the results and decide the next step much faster, and may also make better decisions at the level of the entire large project. But creating a better foundation model requires actual training and experiments, running computations, checking results, saving data, and letting another model look at the results before deciding the next step. This will not be infinitely fast, there are actual physical processes involved. Automation will increase the speed of improvement, but it will not make the cycle accelerate infinitely. Treating it as a pragmatic engineering system makes it less scary and easier to understand.

Li Fei-Fei: Are humans themselves self-improvement systems, or RSI systems? For example, learning to make fried eggs.

Jeff Dean: Humans are both. You learn a lot of things throughout your life, which is self-improvement; but you also learn "how to learn". Learning how to learn allows you to learn more things faster, based on your understanding of what kind of knowledge and information presentation is easiest for you to master. This is closer to recursive self-improvement. Recursive self-learning and recursive self-improvement may also be related to evolutionary algorithms.

Li Fei-Fei: AI has made great progress in mathematics in recent weeks, from work related to Navier-Stokes to many newly published papers. Mathematics is the pinnacle of human wisdom accumulated over hundreds and thousands of years. Mathematicians not only focus on technical solutions, but also on aesthetic beauty. What do you think of this impact?

Jeff Dean: Models and agent systems that can use models in a highly parallel way can indeed solve extremely difficult unsolved mathematical problems, which reveals what is possible right now. Many people feel uncomfortable, because these proofs are often 500 pages of Lean code that can be checked by machines, but no one really understands the elegant and simple path leading to the result. It is good to know that a problem has been proven or disproven, but mathematicians often develop elegant techniques in the process of solving problems, and these techniques will generate new mathematics that can be used for other problems. Elegant solutions often bring reusable and clever techniques, but in very long machine-generated proofs, people may not be able to see such clear and elegant techniques. However, knowing that something is solvable will make people work harder in many fields, know that they can achieve it, and then may achieve it in a different, more elegant way. So on the one hand, it is really disturbing that so many problems are solved at the same time; on the other hand, this has become possible, and people need to think about how to use this capability to do more mathematics.

Li Fei-Fei: Let's do a pop quiz: What is the significance of June 12, 2017?

Jeff Dean: Is that the day the Transformer paper was uploaded to arXiv?

Li Fei-Fei: That's right, A+. Did you read it on June 12?

Jeff Dean: No, I probably read it on June 10.

Li Fei-Fei: Looking back at the past ten years from that starting point, what is the most important technical moment for you personally? Which predictions came true, and which ones were wrong?

Jeff Dean: Several things stand out. Transformer is obviously a much more efficient model architecture: it preserves all the states it has experienced and applies attention to them, instead of constantly updating a small vector and forgetting previous states. This allows the model to learn far more with much less computation. Distillation is also critical. The work I did with Jeff Hinton and Oriol was rejected by NeurIPS in 2016, which said it was unlikely to have much impact. We uploaded it to arXiv, and later it got a lot of citations. It allows you to use a large model as a teacher to teach a lightweight student model, and produce inference-efficient models. MoE has also proven to work very well. One prediction I got wrong was that I thought MoE would move towards wilder, more heterogeneous architectures more similar to organic neural networks, with neural activity columns and occasional large cross-connections; but in reality MoE is still very regular, all experts are the same size and converge immediately. The models generated by neural architecture search look more organic than human-designed ones, for example, they connect to the 17th layer above and make strange connections, but we haven't reached the wild state I expected. Another thing I didn't fully foresee was the programming capability: it has become a key unlocker for models to solve a variety of problems. We valued it in Gemini, but we didn't expect it to be so profound.

Li Fei-Fei: At HAI, we have to talk about the "H". The social impact of AI, employment, psychological anxiety, discussions about human extinction, alignment. What are you most worried about?

Jeff Dean: This is an extremely powerful and extremely general-purpose technology that affects almost every aspect of society: mathematics, programming, image generation, autonomous driving. Previous technological revolutions had a relatively narrow scope of impact, but this one affects many fields at the same time. Excitement, worry and fear all exist. I co-wrote "Shaping AI's Impact on Billions of People", examining seven fields including employment and the economy, AI for science, misinformation and disinformation, and media, and communicated with many domain experts. What I am most worried about is: how to capture the upside benefits of AI as much as possible in various fields, while minimizing the downside harms. This is not a purely technical problem, but also a problem for regulators, policymakers and citizens. We all need to find ways to harness new technologies, ensure that they serve everyone, and do not harm us in foreseeable ways. For example, when people's work is affected, there should be good ways to adapt to the new needs of the world, get paid work and live a prosperous life. This is a multifaceted problem that requires many people to work together to solve.

Li Fei-Fei: One last question. This generation of college students is very anxious, even Stanford students worry about their jobs and economic future. As an AI pioneer, what do you want to say to the students in the audience who are both anxious and hopeful?

Jeff Dean: I fully understand the anxiety. The world is changing very fast, and it is very difficult to plan for a rapidly changing future. My general advice is: try to understand which new technologies are emerging or will emerge soon, and make sure you learn to use this technology for the things you are most passionate about. That could be being a doctor, it could be writing, it could be writing software more efficiently. AI can assist and empower these things, but the human factor still exists: humans need to decide what we collectively want to do. You should figure out what you want to do, do the best you can, which means using all the tools available now. Human passion remains eternal in the AI era.

Li Fei-Fei: Thank you, Jeff.

Jeff Dean: Thank you.

Conclusion

The core of Jeff Dean's dialogue is not "whether AI will replace scientists", but AI is turning scientific discovery itself into an engineerable cycle. The ambition of Discovery Loop is to let a small number of domain experts use the automated experimental cycle to solve large-scale problems that used to require a huge amount of manpower to advance. But Jeff Dean also gave a clear and sober boundary at the same time: automation will accelerate progress, but it will not accelerate infinitely; recursive self-improvement is worth researching, but it should not be hyped as magic that has no physical cost. For research institutions and entrepreneurs, the real opportunity is not to chase the "RSI" slogan, but to digitize, automate and make the experimental cycle in a certain field iterable. For young students, his advice is simple but effective: understand new technologies, do what you love, and make full use of all available tools.

This article is from the WeChat official account "Postwave Evolution Planet", author: Mark, published with authorization from 36Kr.