Meta Layoffs, OpenAI Restructuring: A Comprehensive Review of How the AI Epic Initiated by Google Has Been Rewritten by "Heroes" in a 10,000-Word Analysis

Recently, major AI companies have been making frequent significant moves. Meta's FAIR department has undergone layoffs, OpenAI has carried out a capital restructuring, and AWS has conducted large - scale layoffs... This series of upheavals indicates that the AI industry is shifting from an "unlimited gold rush" to a brutal "trench warfare." Capital has begun to re - evaluate value, and tech giants in the fierce arms race have also had to examine costs and efficiency.

So, how did this "trench warfare" start? Where did the cards in the hands of the giants and the burdens on their backs come from?

A recent episode of the well - known tech podcast "Acquired" centered around Google's AI development history and strategy, and skillfully interwove the rise of other AI giants. The episode covered almost all the key figures in the current AI field, systematically presenting listeners with a concise history of AI development.

Podcast link: https://www.acquired.fm/episodes/google-the-ai-company

This article will follow the discussion thread of this "Acquired" episode. Meanwhile, it will deeply explore and cite relevant details from two key books mentioned in the episode, "In the Plex" and "Genius Makers" (Chinese version: The Deep Learning Revolution), and combine the current situation in the AI circle to conduct an in - depth review of the development history of AI and the strategic games of large companies.

The AI Goal Ingrained in Google's Genes

Different from many latecomers, artificial intelligence was not a trendy "turn" for Google at a certain stage. Instead, it has been the core concept of Google since its inception.

In 1998, Google was founded. One of its founders, Larry Page, regarded Google as an artificial intelligence company, which was largely influenced by his father, a computer science professor who had focused on machine learning and artificial intelligence in the early days.

Let's rewind the clock 42 years to 1956. At Dartmouth College in the United States, a group of passionate scientists officially proposed the term "Artificial Intelligence". They optimistically believed that machines with human intelligence would appear soon.

However, reality soon poured cold water on these overly optimistic predictions. Due to limitations in computing power, scarcity of data, and bottlenecks in theory, many promises could not be fulfilled. The funding and enthusiasm for AI research then plummeted, leading to two "AI winters" that lasted for decades.

In an era when AI was generally regarded as a "waste of time", Larry Page's father's perseverance was quite rebellious.

In 2000, Larry Page asserted, "Artificial intelligence will be the ultimate version of Google... If we have the ultimate search engine, it will understand everything on the web... This is obviously artificial intelligence... We are working towards this goal."

It can even be said that Google's PageRank algorithm, which uses statistical methods to rank web pages, already bears the mark of early AI thinking.

"Compression Means Understanding"

A key origin of Google's AI story began with a lunch conversation at the end of 2000 or the beginning of 2001. Early engineer George Herrick proposed a theory to his colleagues Ben Gomes and Noam Shazeer: Compressing data is technically equivalent to understanding data. The core idea is that the process of efficiently compressing and losslessly recovering information inherently contains a deep understanding of the information.

This idea attracted the genius engineer Noam Shazeer. Under Google's free - spirited engineer culture at that time, Herrick and Shazeer decided to fully explore language models and machine understanding. Although not everyone was optimistic, the support from Jeff Dean and others gave them confidence.

Their research delved into the field of probabilistic models of natural language, which aims to predict what the next most likely word sequence is given a sequence of words. This is an early manifestation of the modern LLM's "Next Token Prediction" concept.

The first direct result of this research was the highly useful "Did you mean" spelling correction feature in Google Search, which was led by Shazeer. It not only improved the user experience but also saved Google a large amount of ineffective computing resources by reducing incorrect queries.

Subsequently, they built a relatively "large" language model at that time and named it PHIL (Probabilistic Hierarchical Inferential Learner). This model quickly played a key role in Google's core business.

In 2003, Jeff Dean used PHIL to quickly implement the AdSense system, which understands web content to match ads. AdSense brought billions of dollars in new revenue to Google overnight.

By the mid - 2000s, it was estimated that PHIL consumed 15% of Google's overall data center resources, indicating its importance and computational intensity.

Machine Translation and Neural Networks

Google's pursuit of language understanding naturally extended to the field of machine translation.

Around 2007, the Google Translate team led by Franz Och built a language model based on massive N - grams (word combinations). The model was trained on a subset of Google's search index containing two trillion words. The team won the DARPA competition with the huge N - gram model, but the model was extremely inefficient, taking 12 hours to translate a single sentence.

Jeff Dean intervened again. He realized that the translation process could be parallelized. Using Google's powerful distributed computing platform, he worked with the team to reduce the translation time to 100 milliseconds within a few months and successfully put it into production. This became Google's first "large - scale" language model in a production environment, further inspiring the imagination of applying such technologies to more scenarios.

Meanwhile, another more revolutionary trend began to quietly knock on Google's door: Neural Networks and Deep Learning. This was thanks to the introduction of Sebastian Thrun.

This former director of the Stanford AI Laboratory (SAIL) joined Google in 2007. After successfully leading the "Ground Truth" map project, he convinced Larry Page and Sergey Brin to invite top scholars to participate in Google's research on a part - time basis.

In December 2007, Sebastian Thrun invited Geoff Hinton, a relatively unknown machine - learning professor at the University of Toronto at that time, to give a technical lecture at Google.

Hinton was a long - time advocate of neural network research. He and his students (including Yann LeCun) firmly believed that with the improvement of computing power, building deeper neural networks (i.e., "deep learning") would unleash their great potential.

Hinton's lecture had a strong impact within Google. In particular, it made Jeff Dean and others see new possibilities for the existing language model work. Subsequently, Hinton began to cooperate with Google as a consultant and even an "intern", officially bringing the spark of deep learning into this future AI giant.

It is worth noting that the neural networks advocated by Hinton and others were at the lowest ebb of being marginalized in the academic community at that time. As described in "Genius Makers", since Marvin Minsky's famous criticism of the "perceptron" in the 1970s, the mainstream in the AI field had shifted to "expert systems". However, expert systems often hit a wall in the face of the complexity of the real world, leading to the second AI winter.

Google's PageRank and the statistical methods relied on by machine translation are themselves a rebellion against rigid expert systems. Hinton's arrival signaled that a more profound paradigm based on data, statistics, and bionic computing was about to combine with Google's engineering capabilities.

By 2011, Google had not only accumulated profound strength in traditional machine learning and large - scale system engineering but also began to embrace deep learning, a new trend that was about to cause a huge wave. The introduction of talent, the success of internal projects, and an open attitude towards cutting - edge theories together laid a solid foundation for Google's next - stage AI explosion.

From "Brain" to "Cat"

The spark of deep learning brought by Hinton soon found a suitable growing environment within Google. Large - scale data and powerful computing infrastructure were exactly the key elements required for neural network research.

Against this backdrop, a core team within Google focused on pushing deep learning to new heights emerged and quickly achieved world - renowned breakthroughs.

The Birth of Google Brain

As Sebastian Thrun joined Google full - time and created the Google X department, he also invited Andrew Ng, another outstanding scholar who succeeded him at the Stanford AI Laboratory (SAIL), to work part - time at Google.

It is worth noting that recently, NVIDIA's market value exceeded $5 trillion, and Andrew Ng and his team pointed out the importance of GPUs for AI as early as a 2009 paper.

Paper title: Large - scale Deep Unsupervised Learning using Graphics Processors

Paper link: https://dl.acm.org/doi/10.1145/1553374.1553486

Back to the main topic. One day between 2010 and 2011, Andrew Ng met Jeff Dean on Google's campus, and they exchanged ideas on language models and deep learning. They quickly realized that by combining Hinton's theory with Google's unparalleled parallel computing power, they might be able to build a truly unprecedented large - scale deep - learning model.

This idea was quickly promoted. In 2011, Andrew Ng, Jeff Dean, and neuroscience doctor Greg Corrado jointly launched the second official project within Google X: Google Brain. Their goal was clear: to build a truly "deep" and "large" neural network on Google's infrastructure.

To support this huge computing task, Jeff Dean led the development of a new distributed computing system called DistBelief.

The design of DistBelief was quite controversial. It allowed different computing nodes to update model parameters asynchronously, which meant that the updates might be based on "outdated" information. This was contrary to the view of mainstream research at that time, which believed that synchronous updates were crucial for ensuring model convergence.

Many people, including experts inside and outside Google, were skeptical (Disbelief), which is also the pun in the system's name. However, Jeff Dean's engineering intuition was proven correct again. DistBelief was not only feasible but also highly efficient.

The Groundbreaking "Cat Paper"

With a powerful computing platform, the Google Brain team quickly launched a milestone experiment. They built a nine - layer deep neural network and used the DistBelief system to train it on 16,000 CPU cores across 1,000 machines. The training data consisted of 10 million frames randomly selected from unlabeled YouTube videos.

The experimental results shocked the world. Without being told what a "cat" was, this neural network autonomously formed a "cat neuron" in the highest - level network through unsupervised learning. This neuron would be strongly excited by images containing cat faces (especially from the front view) and have a weak response to other images.

This result was later published as a paper, but it is more widely known as the "Cat Paper".

Paper title: Building High - Level Features Using Large - Scale Unsupervised Learning

Paper link: https://arxiv.org/abs/1112.6209

The significance of the "Cat Paper" is extremely profound. First, it proved that large - scale deep neural networks have the ability to learn meaningful high - level features from massive amounts of raw data without human supervision. Second, it verified that Google's self - developed distributed system could effectively support this scale of training.

For Google internally, this success was highly persuasive. According to Sundar Pichai, the then - executive, seeing the "Cat Paper" was one of the key moments in his memory of Google's AI story. After presenting the results at a TGIF (Thank God It's Friday, Google's internal Friday meeting), many employees said that "everything had changed".

More importantly, the "Cat Paper" directly generated huge commercial value. At that time, YouTube faced the problem of insufficient understanding of video content. The titles and descriptions uploaded by users were often insufficient to support effective search and recommendation. Google Brain's technology enabled machines to "understand" video content, greatly improving YouTube's recommendation accuracy and user stickiness, and also laying the foundation for key functions such as subsequent content review and copyright recognition.

It can be said that the "Cat Paper" opened the "algorithm - recommendation era" for YouTube and even the entire social media and content platforms, indirectly driving hundreds of billions or even trillions of dollars in industrial value in the following decade.

AlexNet

Almost at the same time as the "Cat Paper", another breakthrough from the academic community completely changed the hardware foundation of deep learning.

In 2012, at the University of Toronto, two students, Alex Krizhevsky and Ilya Sutskever, guided by Geoff Hinton, achieved a "big - bang" success in the famous ImageNet image - recognition competition with their designed deep convolutional neural network, AlexNet.

The ImageNet competition required algorithms to identify objects in millions of labeled pictures. In the previous few years, the error rate of the best algorithms was still above 25%. However, AlexNet emerged and reduced the error rate to 15.3%, an improvement of more than 40% compared to the best result of the previous year.