The hottest embodied model in Silicon Valley today requires no post-training and can master the corresponding capability after only one observation.

The major breakthrough in embodied AI is coming!!!

Ever since Generalist released Gen 1.5, the embodied brain that can learn actions of 3 to 12 seconds last week~

Just now, North American embodied startup Skild AI has launched its brand-new foundational robot model S1, which directly extends the task length of robot in-context learning to over 10 minutes.

The new S1 is focused on in-context learning (ICL):

There is no need to specially learn new operation data in the post-training stage. It only needs to watch a human demonstration video to follow the example and directly complete a whole set of complex operation processes that it has never seen before.

In the official demonstration, the robot completed long-horizon tasks such as making pancakes, brewing coffee, repotting plants, and assembling equipment. For unseen tasks, it achieved a success rate of 66%, far exceeding the VLA based on language prompts (only 9%).

In addition, even if you follow the traditional post-training fine-tuning route, to match the performance of S1 that only watches the demonstration once, you probably need to feed in about 380 task demonstrations.

It is absolutely amazing!

It can be said that the recent wave of work focused on in-context learning has rekindled many people's hope for embodied intelligence.

Some Twitter users said that general-purpose robots may appear in two years instead of seven.

Other users said that from Rhoda, to Generalist last week, to the 10-minute task of Skild S1 now, the real GPT moment for robots seems to be emerging.

Because once this capability is truly generalized, developing robot skills in the future may really become as easy as writing prompts for ChatGPT.

Is it really that magical? Let's take a look together.

Moving from the BERT Era to the GPT Era

To understand how S1 implements in-context learning this time, we first need to see how traditional robots learn new skills.

In the traditional pipeline, robot learning is actually similar to that of large models:

First, pre-train on massive amounts of data to acquire some basic capabilities; then, for specific scenarios, collect data for a certain task, and let the robot learn proprietary skills through post-training.

The problem is that although the process of robots and large models is similar, the data cost is completely different.

Most of the pre-training data for large models comes from the Internet; even for specialized scenarios such as programming and Agents, a lot of post-training data can be quickly obtained online, or even automatically generated.

But robots are in a much tougher situation. Pre-training data has to be collected in the real world, and post-training data also has to be collected in the real world.

Therefore, in its technical blog, Skild AI stated directly:

If a foundational robot model still requires tens or hundreds of hours of real machine data for each new task it learns, and then undergoes re-post-training, then where is the "foundation" of this "foundational model"?

They even cited existing research pointing out that if there is already enough data for a new task, a model trained from scratch may even catch up with the foundational model that has undergone pre-training and then fine-tuning.

So what exactly is the meaning of embodied pre-training?

Skild's answer to this question is: in-context learning.

To have a reference coordinate, Skild takes the development path of large models as a comparison.

The early BERT was already very powerful, but every time it encountered a new task, it often still had to re-prepare data and fine-tune it.

What really changed the game rules was the in-context learning capability that gradually emerged after GPT-3:

Without changing the model weights, as long as a few examples are given in the prompt, the model can temporarily "learn" a new task.

Based on this, Skild believes that robots are actually still trapped in a similar BERT era, and what they really want is to allow robots to complete the same paradigm shift.

In other words, the real value of pre-training should not only be to reduce the amount of data collected in post-training, but to enable robots to finally acquire the ability to "learn directly from the context".

In-context Learning

So what exactly did S1 learn from the context this time?

Skild believes that there are only two dimensions that can truly test a robot's in-context learning capability:

One is whether it can learn new skills that it has never seen at all during training; the other is whether it can recombine existing skills to complete a very long and complex new task.

For example, a robot only needs to watch a video of flipping pancakes to learn how to make pancakes —

Even if this skill has never appeared in its training data before.

At the same time, compared with the second-level video demonstrated by Gen 1.5 last week, S1 this time directly extends the in-context learning to a maximum of 10 minutes.

In tasks such as repotting plants, each task requires completing dozens of consecutive operation steps. In these long-horizon task demonstrations, the robot not only needs to follow the example, but also needs to know:

Which step it is at now, what to do next, how to combine different skills, and how to continue if something goes wrong in the middle.

In other words, the robot needs to truly understand the intent of the task-action in the video demonstration, adapt to changing situations, instead of simply performing behavior cloning.

In addition, in the plant repotting task, it only took 11 minutes from starting to record the demonstration to the robot operating by itself, which completely outperformed the traditional post-training in terms of efficiency.

Finally, throughout the whole process, S1 did not use fine-tuning or post-training, the model weights remained completely unchanged, and it used the same set of weights to complete all the tasks shown in the blog.

It basically aligns with the leap of large models from BERT which requires fine-tuning for specific scenarios to GPT-3 which can complete OOD tasks only by watching demonstrations.

The only difference is that the prompt used is no longer language, but action videos that can better align with downstream tasks.

Experiment: Is ICL more scalable?

In the experimental section, Skild first compared ICL video prompts with traditional language prompt VLA on tasks that the model has seen (in training data) and tasks that it has not seen.

The test results show that when the training data is only 1000 hours, the language prompt works better, but as the data scale increases, ICL begins to surpass it.

At 100,000 hours, for tasks seen during training: ICL reaches 96%, and language prompt reaches 89%.

For the truly critical OOD tasks: ICL reaches 66%, while the language prompt only has 9%, directly creating a gap of more than 7 times.

To verify the generalization of S1, Skild conducted tests under different experimental conditions.

The results show that the performance degradation of the language prompt VLA is up to 3 times that of ICL.

In other words, what ICL learns is not a repetition of actions in a fixed scenario, but the ability to re-plan actions according to the current environment.

Finally, in terms of one-shot learning.

S1 does not perform any post-training on new tasks, and only watches one video demonstration, the success rate reaches 66%.

In contrast, traditional VLA probably needs to go through post-training of about 380 demonstrations to reach the same level.

Of course, after continuing to stack up to 2000 demonstrations, traditional post-training can eventually achieve 86% success rate.

So the conclusion may not be "robots don't need training anymore", but:

In the past, a new skill might have to be taught hundreds of times, but now after watching it once, the robot can directly achieve a score of 60 to 70 points.

This is what Skild calls the ICL scaling law:

The more data there is, the model will not only master more skills, but also become more and more capable of "learning skills from demonstrations".

Who is Skild AI?

Skild was founded in 2023, with two well-known veterans in the CMU robotics circle behind it: Deepak Pathak and Abhinav Gupta.

Both are professors at the Robotics Institute of Carnegie Mellon University. Deepak's research focuses on robot learning, reinforcement learning and computer vision, while Abhinav is a guru in the fields of computer vision and self-supervised learning.

As early as 2022, the two collaborated on WHIRL, which allows robots to perform one-shot imitation by watching human videos. It can be said that the development path of S1 did not emerge out of nowhere.

As a star enterprise in the North American embodied intelligence circle, Skild just came out of stealth mode in 2024, and won a 300 million USD Series A round with a valuation of 1.5 billion USD;

By January this year, it completed a 1.4 billion USD Series C financing, and its valuation directly exceeded 14 billion USD, led by SoftBank, with Nvidia, Bezos and other investors continuing to participate. In more than two years, its valuation has nearly increased by 10 times.

In a horizontal comparison, Skild's positioning in the North American embodied intelligence circle is also quite special.

Compared with PI which is more focused on building "robot GPT", Generalist which has recently emphasized one-shot learner, and Genesis AI and Sunday which have shown full-stack momentum recently, Skild has always emphasized one sentence: Any robot, any task, one brain.

It emphasizes more cross-embodiment, hoping that the same Skild Brain can run on different bodies such as robotic arms, quadrupeds, and humanoid robots.

To put it in layman's terms, it wants to become Android in the robot era: an intelligent layer that can be embedded in various different bodies.

This also determines Skild's data strategy: all-inclusive.

Skild views robot data from three dimensions: hardware proximity, diversity, and scalability:

Real machine teleoperation is the closest to the hardware, but expensive; first-person human videos are the easiest to scale,