Challenging GPUs, bypassing HBM, digging deep into Cerebras behind the largest chip in history
Ten years ago, this was a company that no one had high hopes for. From the very first day of its establishment, it only wanted to build a type of chip that had never been made before. It tried to challenge Nvidia's dominant position, but repeatedly hit a wall in the early days of its founding and was on the verge of bankruptcy many times.
Ten years to sharpen a sword. In May this year, Cerebras finally went public, with its market value once approaching 100 billion US dollars. CEO Andrew Feldman said: We are willing to cooperate with every hyperscaler to improve AI inference speed, except for Nvidia.
We had the honor to interview Cerebras' early investors, the researchers who first proved that Cerebras' products could be deployed, and discussed with senior executives in charge of products inside Cerebras about the company's transformation in the past two years and its moves after IPO. After piecing together these clues, we were surprised to find that to understand the glory of Cerebras' listing, we need to go back 10 years.
Go back to a whiteboard, a startup that was about to go bankrupt, a physics gamble that almost no one believed in, and an AI lab that owned the world's largest language model at that time, and how it became the most critical page in this story at an impossible point in time.
The following is the story of Cerebras.
01
Baidu AI Lab
Discovering the Scaling Law
In 2014, Andrew Ng left the Stanford AI Lab and joined Baidu.
Inspired by Google Brain, Baidu invested 300 million US dollars to establish an AI lab located in Silicon Valley. Their main goal was to make deep learning generate commercial value. For Internet companies at that time, most applications of deep learning were at the physical level: autonomous driving, robotics, and speech recognition.
Greg Diamos
Current Architect at TensorWave ScalarLM, Co-founder of MLCommons
The reason I was so excited back then was that when I was at Nvidia, I was thinking that we should turn the entire CUDA roadmap to deep learning, because AI was finally starting to work for real. At first, this was mainly proven on visual tasks, such as image recognition. But later, after Andrew (Ng) joined Baidu, we started to rethink this issue. His idea was: we should do something that has real product impact, something that people will actually use. So we started with speech. Because the user scale of speech is much larger than that of image search.
Greg Diamos told us in an exclusive interview that back then, Andrew Ng brought a group of top AI researchers to Baidu, and Greg was one of them. Before that, Greg built the CUDA software architecture at Nvidia: and at that time, CUDA was still a neglected internal project.
Greg Diamos
Current Architect at TensorWave ScalarLM, Co-founder of MLCommons
At that time, CUDA was almost a joke, no one used it.
Shortly after the AI lab was established, the Baidu team published a well-known paper titled *Deep Speech*: an end-to-end speech recognition model that allows people to click the microphone icon and speak to the search engine. They also trained several of the world's first deep neural networks, and their next plan was to train language models.
Greg Diamos
Current Architect at TensorWave ScalarLM, Co-founder of MLCommons
We had a crazy curly-haired researcher named Dario Amodei back then. We worked on language models together, and also studied how to scale up language models. During that time, we had the world's largest CUDA cluster for deep learning, which later became quite famous.
The language model researched by the Baidu team was the largest in the world back then: it had 300 million parameters. Training this model took three months. In this process, they discovered a law: The larger the model, the more data, and the stronger the computing power, the smarter the model will be. And this improvement is predictable and linear.
This law is called the Scaling Law. A few years later, this verified result was written into a paper by the Baidu team, titled *Deep Learning Scaling is Predictable, Empirically*. Dario Amodei later brought this discovery to OpenAI, and it became the core logic of GPT.
Greg Diamos
Current Architect at TensorWave ScalarLM, Co-founder of MLCommons
The first time I saw this law, I felt that this was the path to intelligence. I couldn't believe it at the time that there was a way to reach intelligence. Imagine, for people of my generation who studied computer science, we were always taught that progress in computer science is very difficult, and every bit of improvement is hard-won. Some problems are almost untouchable, such as speech, language, and vision, which are very, very difficult. But this thing just works. And it's simple. All you need to do is train a larger language model.
But the problem also came: if the Scaling Law is correct, the computing power required to train models that are truly useful in the future will be at a level that existing GPU clusters cannot provide at all. Baidu's researchers realized that they needed new hardware.
02
Five Engineers
Reinventing the Computer
In the same year that the Baidu team discovered the Scaling Law, five Silicon Valley engineers decided to leave AMD. Before that, they worked together at a microserver hardware company called SeaMicro, which was acquired by AMD for 334 million US dollars in 2012. The names of these five people are: Andrew Feldman, Gary Lauterbach, Michael James, Sean Lie, and J-P Fricker. Among them, Andrew and Gary are co-founders of SeaMicro.
When they got together for the first time, the five of them wrote down their wish on a whiteboard: they hoped that one day their names would appear in the Computer History Museum in Silicon Valley. From day one, Cerebras' goal was to build hardware born for AI.
We know that early AI training relied on a combination of CPUs and GPUs. The CPU is the commander in chief, responsible for scheduling tasks. The GPU is like a huge factory with thousands of small workstations that can perform large-scale parallel computing. But AI training is far more than a single calculation: parameters, gradients, and activation values all have to be moved back and forth between video memory, cache, computing units, and GPUs. Once the model is too large to fit in a single card, it has to be cut into several parts and distributed to multiple cards to make them cooperate with each other.
So what really slows down AI is often the process of data moving back and forth between different chips, different memories, and different machines.
Many chip companies solve this problem by combining GPUs into a cluster and spending a lot of engineering effort to solve the problem of how they communicate with each other. Cerebras' approach is to put as many computing units, memory, and interconnections as possible on a whole wafer, to make a Wafer-Scale Engine (WSE for short).
Ordinary chip manufacturing is to print hundreds of small chips on a wafer at the same time, then cut them apart and use them one by one. But Cerebras makes the entire wafer into a huge AI processor. The area of this wafer is 58 times that of the largest GPU. A lot of data movement that originally needed to be done across GPUs, servers, and networks can now be completed inside the wafer.
Angela Yeung
Senior Vice President of Products at Cerebras
In the traditional GPU architecture, HBM (High Bandwidth Memory) is separate from the computing chip. As long as data needs to be transmitted between memory and computing units, it will take a long time. This process is limited by the so-called memory bandwidth. Because our memory and computing units are on the same piece of silicon, we have extremely high memory bandwidth, reaching thousands of times that of GPUs.
In March 2016, Cerebras was officially established, with Series A led by Benchmark, raising 27 million US dollars.
Speaking of which, we have to admire Andrew's vision. At the end of 2015, when Cerebras decided to bet on the Wafer-Scale Engine, the Transformer architecture had not even been born, and today's big model-centered AI industry was far from taking shape. Back then, deep learning was of course already exploding, and Nvidia had already begun to push GPUs into data centers and neural network training. But the mainstream route was still to combine GPUs into larger and larger clusters, and then use software and high-speed interconnection to solve the problem of training speed.
What makes Cerebras different is that from the very beginning, it asked a more fundamental question: if AI will become a brand new computing load, can we redesign a machine for it?
An investor recalled that when he met Andrew at the end of 2015, Andrew had a slide listing seven core problems faced by deep learning, and these problems essentially all pointed to the same thing: training time is too long. This is exactly the bottleneck faced by the Baidu team mentioned at the end of the previous chapter.
03
A Crazy Gamble That No One Believed In
Investor Nan Zhou joined Baidu AI Lab in 2016 and became Greg's colleague. Her first task was to find a new computing solution built exclusively for deep learning.
Nan Zhou
Early Investor of Cerebras, Former Partner at Qualcomm Ventures
As an investor, the researchers gave me the task: to find a very powerful AI computing solution other than Nvidia's. Because they predicted that in the next five to ten years, more realistically maybe around ten years, there will be very large models with parameter scales far exceeding 500 million, like a Wikipedia-level model. So at that time, they all said: We need a different type of GPU, or a different computing architecture, to ensure that the model can continue to expand and train faster. Because the GPUs at that time were not architectures optimized for deep learning.
In fact, there were many attempts to challenge general-purpose GPUs back then. Google had already used TPUs internally, which were ASICs (Application-Specific Integrated Circuits) designed specifically for matrix calculations in machine learning. Intel acquired the deep learning hardware startup Nervana, UK-based Graphcore was making IPUs, and hardware companies such as Groq, Habana, and SambaNova were also established around this stage, trying to redesign AI chips in different directions.
Cerebras' solution is the most radical among them.
Nan Zhou
Early Investor of Cerebras, Former Partner at Qualcomm Ventures
At that time, Cerebras' answer to this problem was very bold. My understanding back then was: if AI will continue to grow, if models will continue to become larger, then the entire computing system must be reinvented. And Cerebras' approach is wafer-scale computing, which is the Wafer-Scale Engine. So at that time, the real problem and risk was: if computing power will become the bottleneck of AI progress, can this structure really work? Can this architecture really accelerate AI computing? This idea sounded very crazy. But the entire due diligence I did ten years ago eventually became a