Why are graphics cards getting more and more expensive? NVIDIA Vice President: Stop arguing. Moore's Law has long been completely dead.

Around 2005, AI was still an unpopular niche term in academia. To avoid cold shoulders from peers, researchers had to package their work as the more neutral-sounding "machine learning". In 2008, newly graduated Bryan Catanzaro brought a paper on running deep learning on GPUs to top academic conference ICML. Scholars in the audience coldly questioned him: "We only discuss advanced mathematics here. What are you doing bringing a gaming graphics card to this conference?"

That was a cold era for "intelligence", and very few people believed those bulky, heat-emitting graphics cards could have any connection to the human brain.

To find like-minded peers, Bryan chose to leave his job in 2014 and joined Baidu's newly established AI lab in Silicon Valley. There, he met Andrew Ng, and Dario Amodei, a young researcher who was still working on bioinformatics at the time and later founded Anthropic. What shocked Bryan most from this experience was his extremely smart and hardworking Chinese colleagues: They not only open-sourced their developed technologies completely with an extremely open mind, but also pushed the implementation of algorithms with almost instinctive intuition.

After two and a half years working at Baidu, this experience completely shaped Bryan's respect for underlying technology and open-source collaboration. So when many years later, he heard mainstream Western public opinion mock that Chinese AI only knows how to "wrap existing models and copy from closed-source distilled models", he directly refuted: "This is completely biased. China has long been a global leader in the openness of open source collaboration."

Today, Bryan is back at NVIDIA, serving as Vice President of Applied Deep Learning Research. In this latest podcast recorded with Matt Turck, this veteran chip researcher who emerged from the early barren era talked about the staggering computing cost of large models, technical survival after Moore's Law fully died, and why he firmly believes that "an engine without wheels can't go anywhere":

Mocking Chinese AI for only being able to "wrap and copy" is completely an arrogant prejudice. In terms of understanding and practicing "open source collaboration", China has actually walked at the forefront of the world.

Don't expect graphics card prices to drop easily anymore. Moore's Law is completely dead from an economic perspective. Now, as transistors get smaller, their manufacturing cost is skyrocketing exponentially. The era of easily earning dividends just by shrinking chip design is over, and today's acceleration must be squeezed out through "extreme co-design" from algorithms, software to chip manufacturing.

Hyping the "technological singularity" every day is a wrong and one-sided view. Intelligence is highly multi-dimensional and context-dependent. The type of intelligence required for a top corporate CEO is completely different from that required for an Olympiad math champion.

Pretraining a model with 4-bit formatting is extremely easy to fall into the abyss of model divergence. In actual training, a little carelessness will make the model fail to converge and diverge completely, turning a millions-of-dollars worth of computing power bill into waste.

Open source is safer than closed source, and sunlight and diversity are the best disinfectants. In terms of security, trying to monopolize the field by a few giants and forcefully build a wall at the top to define "which ideas are safe and which are not" is an extremely dangerous approach in itself. Supporting diversity of ideas and allowing the whole society to participate in the evaluation and self-correction of technology is a governance path that human society has proven to be safer in practice for hundreds of years.

Host: Brian, I'm really looking forward to this conversation. This year seems to be a year when open source shines. NVIDIA just released Nemotron 3 Ultra, which is an important milestone and currently the best open-source weight model in the US, just released a few days ago.

And just more recently, GLM 5.2 was also released, which is also an important milestone. It seems the pace of the open source AI field is accelerating. This seems like a good entry point. How do you evaluate the stage we are in now, and how big is the current gap between closed-source and open-source models?

Brian Catanzaro: It is really exciting to see so much energy invested in open AI technology, because we know that open technology allows people to truly innovate. The Internet is a perfect example.

We actually experienced a closed Internet before. Do you remember services like America Online and Prodigy back in the day? They were all great. But the open Internet is just as amazing, because so many different companies can use this open technology to find ways to completely transform their businesses.

The way the Internet is used in retail is completely different from how it is used in healthcare or manufacturing. But all these industries have been completely changed because of the Internet.

I believe AI is also a transformative technology, and this technology needs to be applied in extremely diverse ways. That is why I believe open AI technology is fundamental. It is really exciting to see continuous investment and development in open AI technology from different institutions around the world. I hope this trend continues.

Host: How much do you think open source lags behind closed source now? This has been a big trend in the past few years—the gap has been shrinking continuously. Do you think open source is about to catch up, or are closed-source models constantly raising the bar?

Brian Catanzaro: I think this question is maybe a bit "loaded", because it's fun to frame it as a race, but I actually think the entire AI community is moving forward very fast.

If you look at the progress of AI—whether it's closed source or open source—just in the past three months, the progress is amazing. So when you are in a field that is developing so rapidly, I think that is more important than the gap between any specific models, because the most important question is: how is AI evolving as a field?

Host: What do you think is the driving force that keeps open source AI progressing? Is it the community? Is it backed by big companies like NVIDIA? Or is it global competition with China? What is pushing open source AI forward?

Brian Catanzaro: I think there are many factors pushing open AI technology forward. First is the demand itself. A large number of institutions want to customize AI, and want to deeply integrate it into their own workflows, and this kind of integration really requires open AI technology. So demand definitely exists.

I also believe this is inherently the best way to develop technology. We have seen for decades that technology developed in an open environment develops faster, because we can learn from each other.

In this era—the development and implementation of AI is the most exciting technological event our generation has seen in our lifetimes—what more worthy thing can computer scientists devote themselves to than making AI more powerful? If working together as a community to achieve this is the best way, then that itself will push the entire community towards open technology development.

Host: I want to ask a question that may be a bit critical: at least part of the community questions that the open source ecosystem (not NVIDIA, but as a whole) can continue to progress partly because it can distill closed-source models. Now, institutions like Anthropic and Fable Five are starting to discourage distillation. Do you think the progress of open source AI may slow down or be affected because of this?

Brian Catanzaro: In my opinion, there is no doubt that when the entire technology community invests heavily in the most transformative technology of this era, progress will definitely be rapid. Moreover, this technology will never be controlled by a few people, because this industry does not work that way.

We do our best work and achieve the greatest impact when we can think independently and apply it in our own ways.

So I really like the closed-source APIs provided by Anthropic or other institutions. I think they are excellent, and I deeply admire the work done by these labs. But they are not the only labs in the world. There are many labs around the world, and many people with great ideas. It is not the case that only a few labs have a monopoly on all good ideas, that is not how it works. Humans have never worked that way. There are a lot of smart people on this planet.

Moreover, the community obviously cares deeply about this technology, after all it is obviously so profoundly transformative and has such a deep impact on all aspects, so naturally many people want to participate. So I think over time, we will see community-oriented AI development and deployment continue to grow and be widely adopted, because this is indeed how humans as a species have always built things.

Chinese Peers Have Long Led the Way in Open Source Collaboration

Host: Do you think this principle also holds true globally? Especially when it comes to China, many people hold the view that yes, there are many people with great ideas around the world, but much of the progress of Chinese models is directly inspired by closed-source models, or even obtained by distilling them. Is this just clickbait from the media, or as a top AI researcher, are you also impressed by the original ideas coming from China?

Brian Catanzaro: Maybe it's a bit unusual, but I did work for a Chinese company for about two and a half years. I worked at Baidu, in the Silicon Valley AI lab, with Andrew Ng and Dario Amodei. We all worked for a Chinese company, and I witnessed firsthand how smart, hardworking, creative and innovative my colleagues from other Baidu teams were, and that experience has stayed with me.

I think it is completely wrong to claim that all achievements from other countries are just "copycatting".

Do we all learn from each other? Of course we do. But what I want to say is that the Chinese AI community has always kept open the results they have built, which is actually a very good thing for the whole world. I think this has allowed a large number of companies to build things that would not have been possible without this community, and I also think this has promoted technological progress in the entire AI ecosystem.

So I am very grateful for the contributions made by our Chinese peers over the years. I also want to take this opportunity to encourage other AI labs outside of China to embrace this open spirit.

I was very excited when OpenAI released the GPT-OSS model, and then of course Google did excellent work on Gemma, it is really exciting to see that. And we at NVIDIA are also advancing Nemotron.

So I think the rest of the world has a chance to catch up with China, if we can understand the benefits of building AI technology together as a community — frankly, China has been leading the way in this regard for a long time.

Host: Okay. So today, what are the reasons for customers to use open source models? Where do your core advantages lie?

Brian Catanzaro: Every company is built on some kind of "secret". This secret involves not only intellectual property, but also the company's entire platform — how it interacts with problems and customers, and how it thinks about the solutions customers need.

AI is extremely data-dependent. The more valuable the input data, the more valuable the final solution.

Today, when every company thinks about how to deploy AI, they must think about what this means for the company's core confidential information. In many cases, due to trade secrets, business models, or even legal and regulatory requirements, some data must be handled extremely carefully in accordance with the law. In this case, being able to figure it out and implement it yourself is much better than leaving it to others.

When thinking about how AI integrates, how it interacts with customers, and what guardrails need to be set — every company has a specific understanding of its own customers and what customers really need. The great thing about open AI technology is that it allows this customization.

Companies can figure it out on their own and build something that really makes sense for them. I mentioned the Internet at the beginning of this conversation, and how different the deployment of the Internet is in different industries. Today, as AI changes the way people work and live across the entire economy, this demand for customization is equally strong. This is greatly stimulating the demand for open AI technology.

Host: Okay. Before diving into Nemotron, I want to spend a few minutes talking about your experience and background. How did you get to where you are today, including your experience at Baidu?

Brian Catanzaro: I joined NVIDIA in 2008. I was a graduate student back then, trying to design parallel computing solutions for AI. I already believed back then that NVIDIA could change the way computers work.

Host: AI — that must have been a pretty lonely path to explore back in 2008, right?

Brian Catanzaro: Oh, it was really chaotic back then. Everyone thought I was crazy.

I remember going to ICML in 2008, I presented my first paper about training models on GPUs. Someone asked me why I was even there. They said: "This is not a proper paper for ICML, we only do advanced mathematics here."

I thought to myself, "But I really think computing power is important for AI. If we can train larger capacity models, maybe we can solve more problems." They nodded, and it felt like they were thinking, "Uh, I'm not sure why you're here."

Host: GPUs were originally for gaming, right?

Brian Catanzaro: Yes, that is one of their uses, and we still encounter this view today.

Actually, a GPU is whatever NVIDIA says it is. Because we built it. So a GPU is something we built to accelerate the world's most important computing tasks — in 1995, that most important task was graphics rendering, and for a long time now, it has been AI.

Anyway, that's how I joined NVIDIA. I was in the research department back then, working on something pretty "alternative" — trying to build compilers and libraries for AI on GPUs. That later led to the birth of Copperhead, a language embedded in Python that compiles to run on GPUs, and I think it foreshadowed the direction of TensorFlow and PyTorch in many ways.

Later, that led to the birth of cuDNN, NVIDIA's first product for GPU deep learning. I loved that work very much. But I always wanted more direct contact with the practical applications of AI, and at NVIDIA I mainly worked on AI libraries and compilers.

So when Andrew Ng invited me to join Baidu to build the Silicon Valley AI lab with him, I thought it was a great opportunity, because even back then, Baidu was already quite leading in applying AI to core business. So for me, that was a perfect opportunity. Baidu Silicon Valley AI Lab was an amazing place, full of a group of talented, extremely hardworking people.

Host: What was it like working with the young Dario back then? Were there any signs back then that he would grow into the figure he is today?

Brian Catanzaro: Dario was outstanding from the very beginning. I remember I was one of the interview panel members when he applied. He was working on bioinformatics back then, and hadn't gotten into deep learning or what we now call AI yet. But it was obvious that he learned extremely fast and thought extremely deeply.

What I admire most about Dario is how firm his belief is. I have worked in this field for a long time, and I have always believed that AI will change the world, but I don't think I have ever believed in it as completely and thoroughly as Dario does.

Maybe that's because the academic training I received during my PhD was full of caution. Do you remember back in 2005, AI was considered old and "outdated". Everyone said it would never work, people had been trying it since 1945. So there were too many grand promises that ultimately failed to deliver back then.

So when I first entered