Sam Altman singled him out: He is the most important researcher in the AI sector, yet almost no one knows him.

In a recent interview, Sam Altman gave a rare comment:

"He is probably the most important yet little-known researcher in the history of AI."

This person is Alec Radford, the real father of GPT.

You may not have heard of him, but you have most likely used something he invented.

He was the first author of papers on GPT-1, GPT-2, CLIP and Whisper.

In addition, he also participated in a series of key work including GPT-3, DALL·E, Scaling Law, GPT-4 and GPT-4o.

Oddly enough, such a person whose resume almost covers every key turn of large language models holds no doctoral degree and rarely gives interviews.

When ChatGPT became popular all over the world, he was barely seen in the spotlight.

But if you look back through AI papers from the past decade, you will find a more peculiar phenomenon:

Every time a model begins to acquire a new general capability that it did not have before, Alec Radford's name has often appeared in the author list in advance.

OpenAI's real take-off started when it hired Alec

In the latest episode of the show *Invest Like The Best*, the host asked Sam Altman:

"Who is the unsung hero you admire the most along the way of this company?"

Sam Altman blurted out that it was Alec Radford.

The work done by Alec later evolved into the GPT series.

Besides, he kept inspiring and guiding people around him to push the research to directions that were later proved to be extremely important.

*Wired* also put forward a highly influential judgment:

The rise of OpenAI truly began when it hired the then little-known Alec Radford.

In 2016, Alec joined OpenAI. He was only 23 years old that year.

At that time, he was a young guy who ate pineapple and onion pizza at midnight and competed in Kaggle tournaments with his classmates. He founded a small AI company in his dorm in Boston: Indico.

After accepting the invitation to join OpenAI, Alec did not think the job was that important. Instead, he felt the job "was a bit like being in graduate school".

There was not much short-term pressure, nor any product that had to be launched immediately. He was free to look for a question that no one knew the answer to at that time: What on earth can language models do?

His first attempt was to train a language model with 2 billion Reddit comments. The data scale was not small, but the result turned out to be useless.

This experiment, like many early explorations of OpenAI, ended in failure.

But OpenAI let him keep going. Greg Brockman, the then CTO, recalled:

"We just thought Alec was very capable, so we let him do what he wanted to do."

So this young man was allowed to continue his experiments. But soon, he was limited by the insufficient computing power of OpenAI.

Since he could not run larger experiments, Alec narrowed down the scope, collected about 100 million Amazon product reviews, and asked the model to do a simple, almost boring task: predict the next character based on the previous text.

An accident occurred during this process.

Although no one taught the model what a positive review was and what a negative review was, a neuron inside the model began to actively distinguish the positive and negative sentiment of reviews.

Adjust it to one direction, and the model is more likely to generate positive reviews; adjust it to the other direction, and it will start complaining about the products.

OpenAI later named it "Unsupervised Sentiment Neuron".

This experiment for the first time gave the answer that Alec was really looking for:

When the model is required to complete prediction tasks on sufficient data, it may spontaneously learn some capabilities that are never explicitly written in the training objectives.

Ilya Sutskever also encouraged him to go beyond Amazon reviews, and let the model learn text of larger scale and more diverse content, even the entire Internet.

The problem was that with the neural network architecture at that time, processing such a huge amount of data would take several years.

But soon, OpenAI got lucky. Because in 2017, Google's paper that rewrote the course of AI, *Attention Is All You Need*, was published.

Ilya took a fancy to the Transformer architecture at first sight. His first reaction was: "This is exactly what we have been waiting for!"

Alec immediately connected Transformer to his previous experimental line. He collected the BooksCorpus dataset, which included more than 7,000 unpublished English books covering romance, adventure and fantasy themes.

Compared with many datasets where sentences are scattered, books retain long continuous narratives, which is more suitable for the model to learn the long-distance relationship between contexts.

He did not use Transformer for machine translation as Google did, but let it predict the next most likely word.

The model responded: one word, then another, then another — each new word was inferred from the hidden patterns in those 7,000 books.

For Alec, "the progress made in these two weeks is more than the sum of the progress in the past two years". This set of experiments finally became the pre-training basis of GPT-1.

He and his colleagues began to talk about a direction called "Big Transformer": do not design more sophisticated rules for the model, directly scale up the model, data and computing power, and see what else it can learn.

In 2018, this line got its official name:

Generative Pre-trained Transformer, referred to as GPT.

The first GPT paper had four authors in total, and the first one was Alec Radford.

GPT-1 did not cause a sensation in the world immediately, but it brought OpenAI something more important: a clear direction.

After Alec made the breakthrough, the management of OpenAI made a series of key decisions. They began to withdraw research resources from scattered projects such as robotics, and focus them on language models.

Instead of continuing to design complex new architectures, the team preferred to collect more data, invest more computing power, and continue to scale up the methods that had already shown potential.

A year later, GPT-2 was released.

Its parameter count increased from 117 million of GPT-1 to 1.5 billion, and the training data expanded from more than 7,000 books to about 8 million web pages.

The GPT-2 paper had 6 authors in total, Alec was still the first author, and he was co-first author with Jeffrey Wu.

This time, the model no longer needs to be retrained for each task. Only by predicting the next word, it can already try translation, Q&A and summarization, and even achieve good results in multiple zero-shot tests.

In early 2020, OpenAI wrote this relationship into the paper *Scaling Laws for Neural Language Models*.

Alec was also among the 10 authors. The judgment accumulated through countless previous experiments began to be described as predictable mathematical rules.

In May of the same year, OpenAI further scaled up the model to 175 billion parameters and developed GPT-3.

At this point, the number of paper authors had increased from 4 of GPT-1 to 31.

The first author became Tom Brown, and Alec was ranked 29th, only ahead of Ilya Sutskever and Dario Amodei.

The change in the position of his name in the author list also implies the change of the nature of the GPT project.

It is no longer a small-scale experiment led by Alec with a few participants, but a large project completed by OpenAI with coordinated efforts of research, engineering and computing power.

The line that Alec first promoted has become the most important technical strategy of this company.

But just when GPT began to be known by the public and was about to produce huge influence, Alec had already turned his attention elsewhere.

Apart from language, he also accurately bet on the fields of image and speech one after another

After the release of GPT-3, Alec returned to his old field: computer vision.

Long before joining OpenAI, his most influential work was DCGAN.

He, together with Luke Metz and Soumith Chintala, introduced convolutional networks into GAN, solving the problem that this type of model was difficult to train stably.

They also found that when the model learned to generate bedroom images, it would spontaneously form visual representations of beds, windows and scene structures.

The intuition that appeared repeatedly in Alec's research later had already taken shape at this time:

As long as a sufficiently general training task is found, the model may learn more capabilities on its own in the process of completing the task.

Actually, there is an anecdote in between.

In 2016, Jensen Huang demonstrated the images generated by DCGAN at the NVIDIA conference, but attributed the main credit to the Facebook lab led by Yann LeCun at that time.

Alec and several of his friends sat in the Boston office watching the live broadcast and felt very hurt.

Soon after, Alec left Boston and joined OpenAI.

His friend Victoroff even believed that this neglect was one of the important reasons why Alec decided to go to OpenAI.

A few years later, he returned to the computer vision field with GPT technology.

In 2020, OpenAI released Image GPT: expand the image into a string of pixels, and then predict the next pixel just like predicting the next word.

Alec was ranked second in the author list of the paper.

It proves that GPT is not inherently limited to the language field. As long as the data can be represented as a sequence, the same method can also be used to learn images.

But generating pixels one by one is really too slow. Half a year later, DALL·E compresses the image into visual tokens, and then puts text and images into the same Transformer sequence.