How capable are GPT and Claude at directing robots to perform tasks? The first ever evaluation ranking list has been released.

According to Robot Forward Outlook report on October 15, yesterday, the US AI benchmark testing company Artificial Analysis released a new benchmark test, AA-Robotics v0.1.

The company stated that the current evaluation is still a research preview version, which is mainly used to assess the understanding, planning and execution capabilities of cutting-edge large language models in the physical world.

The team designed 8 desktop operation tasks including grasping, object placement, button pressing and other operations in v0.1, allowing cutting-edge large language models such as GPT-6 Astra, GPT-6.1 Sol, and Gemini 4 Argon to directly control a 6-axis robotic arm by writing code to complete the operations.

Desktop operation tasks

The research team released the test results of the 8 tasks. In AA-Robotics v0.1, GPT-6 Astra ranks first with a score of 69.

It is followed by GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 in sequence.

GPT-6 Astra takes an average of 30 minutes to complete each task, with a cost of $16.36 (approximately RMB 109.49).

Although the model is capable of controlling the robotic arm, the test results show that its current speed and cost still cannot meet the requirements of production applications, with a gap of nearly two orders of magnitude compared with the human benchmark.

01. 8 Desktop Operation Tasks to Test the Control Capability of Large Models on Robotic Arms

Since the beginning of 2026, the capability of cutting-edge large language models to control robotic arms has made tremendous progress.

The research team believes that future robots will most likely be driven by powerful general-purpose agents. Therefore, the team designed the AA-Robotics v0.1 benchmark test to verify this direction.

The evaluation allows large language models running in the coding agent framework to directly control the robotic arm.

This test only provides the model with brief operation instructions, including how to connect the robotic arm through an HTTP API, which robotic arm it is, and how to submit tasks and declare completion.

The robotic arm used in the evaluation is Hiwonder NexArm, a 6-axis robotic arm with 5 joints and 1 gripper.

The model can observe the scene through 4 cameras, 3 of which are installed around the table, and the other one is installed on the wrist of the robotic arm.

After observing the scene, the model can also write and run code in an isolated code running environment for motion planning, and send motion instructions to the robotic arm.

AA-Robotics v0.1 task flow

The AA-Robotics v0.1 evaluation mainly includes 8 desktop operation tasks, including

• Control and dexterous operation, to grasp and operate objects precisely, such as organizing building blocks;
• Spatial reasoning, to understand how different objects interact with each other, such as pouring drinks into cups;
• Adaptability, to assess the model's ability to cope with unexpected situations.

Desktop operation tasks

After the model submits the task, the system will score the final state according to the passing criteria set for the task.

You can get 100 points for completing the task. If the task is not fully completed, you will get corresponding partial scores according to the completion degree.

During the evaluation process, all tasks are attempted to be carried out in a zero-shot manner, the model is not specially trained for each task, and only completes the operation according to the task requirements.

02. The Model Can Control the Robotic Arm, But the Problem Is That the Operation Takes Too Long

From the perspective of the test process, the model has been able to control the robotic arm. The current problem is that it takes a lot of time and cost for the model to control the robotic arm to complete the task.

The evaluation results show that the speed of current language models to complete AA-Robotics v0.1 tasks is only about 1/100 of that of humans. The average completion time of each task for the model is about 30 minutes, while the average for humans only takes 20 seconds.

On the whole, 4 of the 5 test models perform relatively close, and each task takes an average of about 30 minutes, of which Gemini 4 Argon is the slowest, taking an average of 52 minutes.

Average task completion time

The reason is that at the initial stage of the task, the model will spend a lot of time calibrating the robotic arm and understanding the scene, and this process will continue until the model takes the first action.

After the first action, most of the time is spent on motion planning. In contrast, the actual time the robotic arm spends executing actions is relatively short.

However, there are differences between different models. For example, GPT-6 Astra only took 2.2 minutes to press the red button.

Press the red button task

In contrast, Claude Fable 5.1 spent about 10 minutes understanding the scene and planning before taking the first action.

In the task of placing a crocodile toy, this gap is even more obvious.

Place the crocodile toy task

GPT-6.1 Sol declared completion 5 minutes after the start of the task, but Claude Opus 5.5 spent 45 minutes. The completion time of the two differs by 9 times.

Among all tasks, Gemini 4 Argon is the slowest model to complete tasks. It takes an average of 21 minutes to observe the scene and reason before starting to take the first action, which is almost twice the completion time of other models.

At the same time, the research team also found in the test that after the model has reached the task passing state, it often continues to optimize the operation, even for a very long time. For example, GPT-6.1 Sol had already built a tower that met the requirements at about 17 minutes. But then it continued to operate, which eventually led to the failure of the task.

This example also reveals that to make the robot complete the task, it is not only necessary to solve the problem of "how to do it", but also the problem of "to what extent it can be regarded as finished".

Whether the model can further improve its ability to judge the task completion state in physical space in the future remains to be seen.

03. GPT-6 Astra Takes the First Place, But With the Highest Cost

Grasping and fine operation are the key capabilities tested in AA-Robotics v0.1. In the grasping and fine operation test, GPT-6 Astra and Claude Fable 5.1 performed best, with the highest proportion of successfully completed tasks, about 45%. This shows that they have a certain degree of fine control capability.

Performance of each model in grasping and fine operation test

GPT-6.1 Sol performed the worst, with a success rate of only 28%. The success rate of Gemini 4 Argon is 36%, which is between the two.

At the same time, speed and cost are still one of the biggest obstacles for models to enter production scenarios. Compared with the time gap, the cost gap between humans and models is smaller.

In terms of cost, the model with the lowest cost costs about 10 times the human cost calculated according to the average hourly wage in the United States to complete the task. In terms of time, the gap between the model and humans is close to 100 times.

In addition, there are differences in cost between different models. Although GPT-6 Astra performs best, it also has the highest cost.

Average task completion cost

GPT-6 Astra costs an average of $16.36 per task (approximately RMB 109.49). The lowest-cost model, GPT-6.1 Sol, costs an average of only $1.93 per task (approximately RMB 12.92).

In terms of cost, GPT-6 Astra is 8 times that of Sol, but the final score is only 4 points higher, which are 69 points and 65 points respectively.

It is worth noting that most of the current cost of the model comes from input. Considering that the robotic arm and test scenarios are relatively novel, and the model needs to process a large amount of visual information, about 70% of the cost is used to process context information.

04. Conclusion: Time, Cost, Judgment, Three Thresholds for Large Models to Control Robots

In the AA-Robotics v0.1 test, the model observes the environment through the camera, combines visual information for reasoning, and controls the robotic arm to complete the task.

But for models and robots, time and cost are still hard shortcomings in actual operation and implementation, and there is still an obvious gap compared with human operation.

At the same time, the model's ability to judge the task completion state in physical space still needs to be enhanced.

Among the failure cases announced by the research team, some robotic arms place objects in corresponding positions, and the tasks are clearly completed, but they repeatedly grasp the items, cannot judge whether the tasks have been completed, and also extend the operation time.

More importantly, when robots enter the real environment, they need to deal with more uncertainties, including problems such as object position changes, grasping failures, and task state judgment.

Therefore, how to improve the operation efficiency of the model, reduce the use cost, and enhance its ability to judge the task completion state are still problems that need to be solved in the follow-up.

This article is from the WeChat official account "Robot Forward Outlook", author: Liu Lishan, editor: Xu Lisi, authorized for release by 36Kr.