Will General Large Models Deliver a "Dimensionality Reduction Strike" to Embodied Intelligence? The First AA-Robotics Ranking Has Been Launched

Is the "robot brain" painstakingly trained by embodied intelligence companies about to be outcompeted by GPT, Claude and Gemini?

On October 14, Artificial Analysis, a well-known large model evaluation institution, put these large models with outstanding performance in mathematics, programming and other fields into a real robotics exam.

In the newly released  AA-Robotics v0.1  evaluation, cutting-edge models such as GPT, Claude, and Gemini need to control real robotic arms to complete eight tasks including pouring water, writing, and building blocks, to compete for who has stronger "hands-on ability".

As a result, the large models made quite a few blunders as soon as they got started. One robotic arm spent more than ten minutes finally stacking several foam blocks into a qualified tall tower. GPT-6.1 Sol, which was directing it, had clearly completed the task, but still refused to stop and continued to adjust the blocks, which eventually led to the failure of the task.

This is just a dramatic scene in this robotics exam.

The first batch of results is also full of contrasts: GPT-6 Astra ranks first, and general large models have been able to manipulate real robotic arms with the help of programming Agents; but they take an average of about half an hour to complete a task, which is nearly two orders of magnitude slower than the human baseline.

This report card also provides new basis for discussion on a recent debate in the robotics industry: as general large models become more and more powerful, how much irreplaceable value does the "robot brain" specially trained by embodied intelligence companies still have?

Will the "robot brain" be replaced by general large models?

Last month, videos of GPT-6 Astra controlling robotic arms to draw the Golden Gate Bridge, grab blocks and imitate human operations went viral one after another. A robotic arm priced at about 150 US dollars, under the control of Astra, can even independently plan movements based on camera footage and draw the Golden Gate Bridge stroke by stroke.

A more convincing experiment comes from RoboCurve. In its real-machine test, Astra controlled a dual robotic arm to put red blocks into a bowl, succeeding 19 times out of 20. Such results have led many people to ask: since general large models already understand vision, space, reasoning, and can write control codes, why do robotics companies still invest a lot of time and money to train their own VLA models from scratch? Some people even proposed that the OpenAI in the robotics field may be OpenAI itself.

However, people in the embodied intelligence industry do not fully buy this view.

At the Bund Summit on September 10, Wang Qian, founder of Zili Robotics, talked about the "dimensionality reduction strike" argument and judged that the possibility is not high at present. His reason is that the capability of models comes from data. The improvement of programming models is backed by massive code data and a mature verification system; the physical world faced by robots also requires dedicated data, evaluation, hardware and infrastructure, which will not be automatically solved just because language models are powerful. Wang Qian also mentioned that Astra is not completely isolated from robotics data. Therefore, the fact that general models can control robotic arms cannot be simply understood as the natural transfer of language capabilities to the physical world.

Xu Huazhe, founder of Poke Robotics, also tried Astra himself. It can recognize items on the desktop and even pick up a chopstick; but once it is required to perform more complex contact operations with chopsticks, its performance drops significantly. In his view, the progress in semantic understanding and spatial understanding is amazing, but complex physical interaction is still a completely different problem.

The focus of the debate between the two sides has gradually become clear: whether general large models can "do things with their hands" is no longer the only question; how much they can actually do, and whether they can do it stably and quickly, is more critical.

AA-Robotics just provides a new ruler for this measurement.

GPT-6 ranks first, but the robotics exam is far from simple

Artificial Analysis has been tracking the reasoning, coding capability, speed and price of large models before. This time, it moved the exam venue from the screen to the desktop.

Each model was given control of a Hiwonder NexArm robotic arm. This is a six-axis robotic arm with five joints and one gripper, and a total of four cameras are arranged around the desktop and on the wrist of the robotic arm.

At first, the model only knows the name of the robotic arm, how to communicate with it through the HTTP API, and how to submit tasks. It has to look at the cameras by itself, understand the position of objects, write and run programs to make the robotic arm complete the movements. The GPT model runs in Codex, Claude uses Claude Code, and Gemini uses Antigravity. All eight tests are zero-shot attempts, and no additional training will be conducted for the current task. This evaluation examines how general large models, with the help of programming Agents and robot control interfaces, understand the environment, plan movements, and finally complete real-world operation tasks.

The comprehensive scores of the first five models have been announced: GPT-6 Astra scored 69.4 points, GPT-6.1 Sol 64.6 points, Gemini 4 Argon 55.3 points, Claude Opus 5.5 and Claude Fable 5.1 scored 48.8 points and 46.3 points respectively.

However, the score is not the success rate. Each full score of a task is 100 points, and you can get process points even if you don't fully complete it. Therefore, although Astra has the highest total score, it got full marks in four out of eight tasks; Sol, which ranks second, got full marks in five tasks instead.

The five models all passed the two simplest tasks: pressing the red button with the gripper, and putting a crocodile toy completely into the designated box. These operations are no longer difficult for today's cutting-edge large models.

After that, the gap widens rapidly.

Astra can pour the drink from the bottle into the cup, and can even hold a pen to write "hello world" on the paper, but got zero points in the task of measuring the length, width and height of the box. Sol completed the tasks of storing blocks, shape matching and writing, but failed to pour water properly. None of the models got full marks in the task of building a block tower: Sol even once built a qualified tower, but all its previous efforts were wasted due to subsequent operations.

This scene is very illustrative. In the computer, the model only needs to stop generating after finding the answer; in the physical world, it must also judge whether the object in front of it has been placed stably, whether the task is really completed, and whether the next movement will destroy the results.

A more striking contrast lies in time and cost.

Among the five models, except that Gemini 4 Argon takes an average of about 52 minutes, the other four take about half an hour to complete one task. The manual direct operation baseline given by Artificial Analysis only takes an average of about 20 seconds. For trivial tasks like pressing the red button, Astra spent 2.2 minutes; Claude Fable 5.1 even spent about 10 minutes understanding the scene and planning movements before starting the first operation.

The cost is also high. The average model invocation cost of Astra for each task reaches 16.36 US dollars, while Sol only costs 1.93 US dollars, with a score difference of only 4.8 points. For robots that need to work continuously in homes and warehouses in the future, being able to complete a task once is a completely different thing from being able to complete hundreds of tasks a day at a reasonable cost.

Being able to figure out the next step does not mean you can do it well

Why do large models take so long just to move a robotic arm?

Artificial Analysis disassembled the whole task process. Taking Astra as an example, about 20% of its time is spent understanding the environment and the robotic arm, about 59% is spent planning movements, and the actual movement of the robotic arm only accounts for about 13%. The situation of Sol is similar: 61% of the time is spent on planning, and the movement of the robotic arm only accounts for about 13%.

They look like a person who walks into the laboratory for the first time: first observe how the robotic arm moves, then write programs, test positions, check the footage, and issue the next command after confirming that everything is correct. The movement itself often only takes a few seconds, but the thinking before and after may last for several minutes.

Slowness is not the only problem.

Among the six tasks involving grasping, Astra's grasping success rate is 45%, Claude Fable 5.1's is 44%, and Sol's is only 28%. The model has clearly found the target, but the gripper may close next to the object; sometimes as soon as it is picked up, the bottle slips from between the fingers.

In the failed clips sorted out by the official, there are many impressive scenes: the robotic arm, in order to adjust the position of the crocodile toy, accidentally pushes the toy that has been put into the box out; when grabbing the bottle, the wrist hits the tray, making the bottle more and more difficult to reach; at the end of pouring water, it does not consider the change of the water flow position, and the last part of the drink spills out of the cup.

These errors cannot be completely solved simply by "thinking for a little longer". Objects in the real world will slide, tip over and deform, and friction and collision will occur when fingers contact objects. Robots need to correct errors in time according to motion and contact feedback, instead of waiting for the large model to re-plan after each movement.

This is also where professional embodied models still have room to play. VLA, low-level motion strategies and controllers have long been dealing with how to quickly convert vision and task targets into continuous and stable movements.

Of course, this list does not yet put general Agents and dedicated VLA to a direct contest under the same set of conditions. The eight desktop tasks cannot represent all robotics tasks, let alone two-handed collaboration, flexible object manipulation and long-term autonomous work.

But it at least takes the debate a step forward: there is systematic evidence that general large models can complete real-world operations; how far they are from performing real work efficiently and reliably has been specifically measured in this set of tasks for the first time.

There are no Chinese models in the first batch of AA-Robotics v0.1 list.

So, will Qwen, Kimi, GLM or DeepSeek appear in the next batch? If placed in the same robotic arm and the same tasks, can domestic models catch up with or even surpass these top international models at a lower cost?

This is perhaps more worth looking forward to than simply competing for a ranking on the list. What the robotics industry will compete for in the next stage is no longer just "who can make the robotic arm move", but who can make it move faster, grasp more accurately, and know when to stop after success.

Reference links:

https://artificialanalysis.ai/articles/aa-robotics

https://x.com/ArtificialAnlys/status/2110500262628806787

This article is from the WeChat official account "Machine Heart" (ID: almosthuman2014), author: Embodied Intelligence Focus, published with authorization from 36Kr.