OpenAI's approach is questioned. Meta researcher: It's simply impossible to build superintelligence.

Superintelligence is a higher - dimensional AI development direction above AGI, with general capabilities even surpassing those of humans.

Behind Zuckerberg's move to poach talents from competitors like OpenAI with an annual salary of up to $100 million lies the huge ambition of leading players like Meta to pursue "superintelligence".

So, how can superintelligence be achieved? Is the research path of existing large language models (LLMs) correct? Can Scaling Laws continue to work in this process?

As early as 2023, Sam Altman, the CEO of OpenAI, stated that building AGI is a scientific problem, while building superintelligence is an engineering problem. This seems to imply that they know a feasible path to build superintelligence.

However, in the view of Jack Morris, a researcher at Meta AI, the "engineering problem" of superintelligence mentioned by Altman lies in "building a large number of RL environments suitable for different tasks and training LLMs to handle all these tasks simultaneously". He believes that the current path vigorously promoted by companies like OpenAI - RL based on LLMs - cannot build superintelligence at all.

"My humble prediction is: LLMs will continue to perform better on tasks within the training distribution. As we collect and train on more types of tasks, this will result in LLMs that are increasingly useful across a wide range of tasks. But it won't become a single superintelligent model."

In a blog titled "Superintelligence, from First Principles", Morris explored three possible ways to build superintelligence: fully through supervised learning (SL), reinforcement learning from human verifiers (RL), and RL from automated verifiers.

Moreover, he also believes that integrating non - text data into the model does not improve the overall performance of the model. "Text written by real humans carries some intrinsic value that the pure sensory input from the world around us can never possess."

Academic Headlines has condensed the overall content without changing the original meaning, as follows:

Original article link:

https://blog.jxmo.io/p/superintelligence-from-first-principles

Many people are discussing how to achieve AGI (Artificial General Intelligence) or ASI (Artificial Superintelligence) using current technologies. Meta recently announced that they are establishing a secret "superintelligence" laboratory, investing billions of dollars. OpenAI, Anthropic, and Google DeepMind have all expressed their goals of building superintelligent machines in different ways.

Sam Altman specifically stated that superintelligence is merely an engineering problem:

This implies that OpenAI's researchers know how to build superintelligence and only need to invest time and effort to establish the required systems.

As an AI researcher, I'm not sure how to build superintelligence - I'm not even certain if it's possible. Therefore, in this article, I hope to delve into some details and speculate whether anyone can attempt to build superintelligence from first principles.

Let's assume that the basic building blocks for achieving this technology are already determined: using neural networks as the underlying architecture and training them through the backpropagation algorithm and some form of machine learning.

I believe that the architecture (the structure of the neural network) is not the most critical factor. So, let's skip the details about the architecture and make a bold assumption: superintelligence will be built using Transformers, the most popular architecture for training such systems on large datasets at present.

Well, we already know a lot: superintelligence will be a Transformers neural network, which will be trained through some machine - learning objective function and gradient - based backpropagation. There are still two main open questions here. Which learning algorithm do we use, and what data do we use?

Let's start with the data.

Data: It Must Be Text

Many of the major breakthroughs that led to ChatGPT stemmed largely from learning from the vast repository of human knowledge on the Internet. Although most of its complexity is cleverly hidden by modern engineering, let's take a moment to try to figure it all out.

Currently, the best systems rely on learning from text data on the Internet. As of the time of writing this article (June 2025), I believe that integrating non - text data into the model has not improved the overall performance. This includes images, videos, audio, and the supersensory data from robotics - we're not yet sure how to use these modalities to enhance ChatGPT's intelligence.

Why is this the case? It could just be a scientific or engineering challenge, and we might not be using the right approach; but it's also possible that text itself has some special characteristics. After all, every piece of text on the Internet (before the emergence of LLMs) is a reflection of the human thought process. In a sense, text written by humans is pre - processed and has a very high information content.

In contrast, images are just raw perspectives of the world around us without human intervention. It's quite possible that text written by real humans carries some intrinsic value that the pure sensory input from the world around us can never possess.

So, until someone proves otherwise, let's assume that only text data matters.

Then, how much text data do we have?

The next question is how large this dataset might be.

Many people have discussed how to deal with the situation if we run out of text data. This situation is called the "data wall" or the "token crisis", and people have explored what to do if we really run out of data and how to scale our models.

And it seems that this situation is really happening. Engineers in many large AI labs have spent countless hours scraping every useful piece of text from all corners of the web, even transcribing millions of hours of YouTube videos and purchasing a large number of news stories for training.

Fortunately, there might be another data source available here (verifiable environments!), but we'll discuss this later.

Learning Algorithm

In the above, we've discovered an important principle: the best path to superintelligence lies in text data. In other words, AGI is likely to be an LLM, or it simply doesn't exist. Some other promising areas include learning from videos and robotics, but these fields seem far from producing independent intelligent systems by 2030. They also require a large amount of data; learning from text is naturally very efficient.

Now we must face the most important question. What is the learning algorithm for superintelligence?

In the field of machine learning, there are two proven basic methods for learning from large datasets. One is SL, which involves training the model to increase the probability of certain example data. The other is RL, which involves generating data from the model and rewarding it for taking "good" actions (as defined by the user's "good" criteria).

Now that we understand this classification, it's clear that any potential superintelligent system must be trained through SL or RL (or a combination of both).

Image | Yann LeCun once said that he knows the secret to intelligence. In fact, intelligence is like a cake, and RL is just a small cherry on top.

Let's explore these two options separately.

1. Hypothesis 1: Superintelligence Comes from SL

Remember 2023? That's when people started getting excited about scaling laws; after the release of GPT - 4, people were worried that if the models continued to scale, they might become dangerous.

Image | Around 2023, many people started to worry that LLMs would soon evolve into superintelligence through simple supervised learning scaling.

For some time, it was widely believed that a large amount of SL, especially in the form of "next - token prediction", might lead to the emergence of superintelligent AI. Notably, Ilya Sutskeve gave a speech pointing out that next - token prediction is essentially about learning to compress the "(information) universe" because doing this well requires simulating all possible programs (or something similar).

I think his argument is roughly as follows:

Accurate next - token prediction requires modeling what anyone would write in any situation.

The more accurately you model a person, the closer you get to that person's intelligence.

Since the Internet contains text written by many people, training on a large text pre - training dataset requires accurately modeling the intelligence of many people.

Accurately modeling the intelligence of many people is superintelligence.

(1) The "Atmosphere" Theory: Can We Achieve Superintelligence by Simulating Humans?

Personally, I think there are some flaws in this logic. First of all, we seem to have created systems that far exceed human - level performance in next - token prediction, but these systems still cannot demonstrate human - level general intelligence. To some extent, the systems we've built have learned what we've asked them to learn (next - token prediction), but they still can't perform the tasks we expect them to (such as answering questions without fabricating and perfectly following instructions).

This might just be a failure of machine learning. We've been training a model to predict the average human result in each situation. This learning goal encourages the model to avoid assigning too low a probability to any possible result. This paradigm often leads to the so - called "mode collapse", where the model is very good at predicting the average result but fails to learn the tails of the distribution.

These problems might disappear after scaling. Models with billions of parameters, like Llama, generate hallucinations, but they only have 10^9 parameters. What will happen when we train a model with 10^19 parameters? Perhaps this will be enough for a single LLM to independently model the 8 billion humans on the planet and provide independent data - driven predictions for each person.

(2) The Infra Theory: We Can't Scale the Model and Data

But it turns out that this doesn't matter because we might never be able to scale to 10^19 parameters. This assumption basically stems from the deep - learning school around 2022, driven by the great success of the scaling laws of language models, believing that continuously scaling the model and data size would achieve perfect intelligence.

It's 2025 now. This theoretical argument has not been challenged, and the scaling laws have always been valid. But it turns out that when the scale exceeds a certain threshold, scaling the model becomes very difficult (and as early as 2022, we were already very close to the limit that we could effectively handle). Companies have far exceeded what we can do with a single machine - all the latest models are trained on giant networks composed of hundreds of machines.

Continuing to scale the model to trillions of parameters is causing shortages of hardware and electricity. Larger models will consume so much electricity that they can't be centrally deployed in a single location; companies are researching how to distribute model training across multiple distant data centers and even acquiring and rehabilitating abandoned nuclear power plants to train the next - generation larger - scale AI models. We're in a crazy era.

In addition to the model scale, we might also face a shortage of data. No one knows how much Internet data each model uses during training, but it's definitely a very large amount. In the past few years, large AI labs have made huge engineering efforts to squeeze the last bit of value from Internet text data: for example, OpenAI seems to have transcribed the entire YouTube, and high - quality information websites like Reddit have been scraped repeatedly.

It seems difficult to scale the model beyond 100 billion parameters, and similarly, it's very difficult to expand the data scale beyond 20T tokens. These factors seem to indicate that it's difficult to scale SL by more than 10 times in the next three to four years - so, the exploration of superintelligence might have to find a breakthrough elsewhere.

2. Hypothesis 2: Achieving Superintelligence by Combining SL and RL

Maybe you agree with one of the above views: either we won't be able to increase the pre - training scale by several orders of magnitude for a long time, or even if we do, being very good at predicting human tokens won't build a system smarter than humans.

There's another way. The field of RL offers a whole set of methods to learn through feedback rather than just relying on demonstrations.

Why do we need SL?

RL is very difficult. You might wonder why we can't use RL throughout. From a practical perspective, RL has many drawbacks. In short, SL is much more stable and efficient than RL. An easy - to - understand reason is that since RL works by having the model generate actions and then scoring them, a randomly initialized model is basically bad, all actions are useless, and it has to accidentally do something well to get any form of reward. This is the so - called cold - start problem, and it's just one of the many problems with RL. SL based on human data has proven to be an effective way to solve the cold - start problem.

Let's re - examine the RL paradigm: the model tries various actions, and then we tell the model how well these actions perform. This can be achieved in two ways: either by having human evaluators tell the model how well it performs (this is roughly how typical RLHF works), or by an automated system.

3. Hypothesis 2A: RL from Human Verifiers

In this first paradigm, we provide human - based rewards to the model. We want the model to have superintelligence, so we want to reward it for generating text that is closer to superintelligence (as judged by humans).

In fact, collecting this type of data is extremely costly. In a typical RLHF setup, a reward model needs to be trained to simulate the human feedback signal. Reward models are necessary because they allow us to provide far more feedback than the actual amount of human feedback. In other words, they are computational aids. Let's consider the reward model as an engineering detail and ignore them for now.

So, imagine a world where we have an infinite number of humans to label data for LLMs and provide arbitrary rewards, where a high reward means the model's output is closer to superintelligence.

Image | "A thousand monkeys at a thousand typewriters. Soon, they'll write the greatest novel in human history." - Mr. Burns, The Simpsons

Ignore all the programmatic complexity. Assume that this method can be applied on a large scale (although it might not be possible currently, but it might be feasible in the future). Will it work? Can a machine that only learns from human reward signals keep climbing the intelligence ladder and eventually surpass humans?

Put it another way: Can we "verify" the existence