Can the model perceive and adjust the pose of 3D objects? The first interactive projection alignment spatial reasoning benchmark
After recognizing an object, you can still identify it when viewing it from another angle, and further imagine what it looks like after rotation. All these processes involve spatial perception. Humans need to represent the shape and orientation of objects in their minds, understand the relationships between different perspectives, and infer 3D geometry from 2D projections.
When vision-language models enter interactive scenarios, this understanding needs to be translated into actions. A single rotation will change the contour, proportion and occlusion relationship of an object; the model needs to predict the changes in advance, observe the execution result, and decide how to adjust in the next step. Can correctly answering a spatial relationship question further support such continuous pose control?
Existing benchmarks have different focuses on this. Static benchmarks usually evaluate spatial reasoning through question answering on fixed images, lacking action execution and closed-loop feedback; many interactive benchmarks focus on task planning, tool calling or path searching. These tasks can test interactive capabilities, but it is difficult to specifically verify whether the model can understand 3D orientation from visual cues and convert this understanding into precise rotation.
Therefore, can vision-language models convert 2D observations into 3D understanding that supports precise actions?
This requires the model to connect the geometry and pose of the object, predict the impact of rotation on the projection, and continuously make corrections using new observations after the action. For agents that need to explore the environment, understand geometry and take actions, these judgments connect visual perception and actual control, and are also the capabilities that the industry hopes to further observe through evaluation.
To this end, the team from Xiamen University and Nanyang Technological University proposed RoTA-Bench to evaluate this capability through interactive 3D-to-2D projection alignment.
Paper Link: https://arxiv.org/abs/2610.17665
Code and Dataset: https://github.com/MAC-AutoML/RoTA-Bench
Online Demo: https://www.modelscope.cn/studios/san2333333/rotabench-anon?mode=full&app_path=%2Fportal
Hugging Face: https://huggingface.co/papers/2610.17665
The following is the interactive interface of a real task in RoTA-Bench. The upper left of the picture is the T-shirt-shaped target contour, the lower left is the current projection of the object, and the right side is the object view with depth shading. The task requires selecting rotation actions to gradually align the current projection with the target contour.
A real task example of RoTA-Bench. The upper left of the picture is the target contour, the lower left is the current projection, and the right is the depth shading auxiliary view; the buttons at the bottom are used to select the rotation direction and angle.
Turn a little to the left first, or flip up first? Should you rotate 30° at this step, or only adjust 5°? The rules are intuitive, but every step requires connecting the 2D picture with the 3D pose.
In the experiment, GPT-5.6-Sol achieved a success rate of 17% on 100 test tasks; 400 task runs completed jointly by 20 undergraduates were all successful.
This gap is worth further analysis: at which step did the model encounter difficulties? If the contour looks more similar to the target, does it mean that the 3D pose is also rotated more correctly?
After seeing this picture, how should you rotate next?
Take the T-shirt example above, rotating left and right may change the contours on both sides, and flipping up may change the overall proportion; the same object facing a different direction may also have occlusions. The model needs to connect the projection in front of it with the 3D pose behind it, and judge which axis to rotate along, which direction to go, and what angle to turn.
In RoTA-Bench, the model sees three pieces of information at each step: the target contour, the current projection, and the depth shading auxiliary view from the same perspective as the current projection. The auxiliary view supplements the depth and occlusion cues missing in the binary contour.
Then, the model selects the next step from 24 discrete rotation actions: three rotation axes, two directions (positive and negative), paired with four step sizes of 1°, 5°, 10°, 30°. After each execution, the environment re-renders the picture, and the model continues to adjust using the updated observations and interaction history.
This requires both large-amplitude rotation and reserves space for fine adjustment. When the orientation is still far from the target, a larger step size can be selected; when approaching the target, a smaller angle can be tried.
The task is successful when the Intersection over Union (IoU) between the current projection and the target contour reaches 0.90. IoU measures the overlap degree of the two contours, and the higher the value, the closer the projection is. The evaluation allows a maximum of 48 decisions, and there are also rules for active stop and stagnation termination.
The researchers limited the interaction to pure rotation, in order to separate 3D orientation reasoning from factors such as translation, grasping, and contact friction, so that the problem focuses on "how to control the pose based on visual evidence".
Where do these objects come from, and why can they be guaranteed to be rotated back?
If common 3D objects are used directly, the model may make judgments with the prior knowledge of familiar shapes.
In order to reduce such semantic shortcuts, the researchers constructed 3D objects within the constraints of a spherical volume starting from 2D vector icons. The generated objects can restore the original image under the standard front projection and retain details.
Next, use a series of reversible discrete rotations to rotate the object to the initial pose. Since each step can be reversed, the target pose can be restored by rotating in reverse order, which ensures the solvability of the task from the construction process.
On this basis, the researchers further searched and verified the reference paths, and selected test tasks in layers according to the length of the reference paths. The reference paths here are used for task screening and analysis, which are not guaranteed to be the shortest solutions, and will not be provided to the tested model.
Finally, RoTA-Bench contains 674 tasks, of which 100 tasks form a frozen test set, and the remaining 574 are reserved for subsequent training and research.
Task construction and closed-loop evaluation process (Figure 2 in the paper). The upper part of the picture shows the process of generating 3D objects from 2D icons, initializing through reversible rotation, verifying reference paths and selecting test tasks; the lower part shows the closed-loop evaluation of "observe - select rotation - update picture".
For the same task, what is the gap between humans and models?
The researchers evaluated 9 vision-language models, 2 agent configurations, and invited 20 undergraduates to participate in human experiments.
On 100 test tasks, the success rate of GPT-5.6-Sol is 17%, which is the highest model success rate observed in this evaluation. In the human experiment, each participant completed a block containing 20 tasks; 20 participants covered the entire test set four times in total, with 400 task runs all successful.
Humans and models use the same tasks, visual observations, rotation action space, decision budget and success criteria. Humans submit actions through buttons, and models submit actions through structured outputs, which are finally mapped to the same rotation operation.
RoTA-Bench Overview (Figure 1 in the paper). The left side of the picture is the reference path length distribution, the middle is the T-shirt task example, and the right side is the test set success rate and final IoU. The human results summarize four runs of each test task.
The gap can also be seen from the rotation process. The average IoU of human trajectories increases as a whole with the advancement of actions, and finally crosses the success threshold; the model trajectories enter a plateau period earlier, and most stay around 0.50-0.63.
Variation of average projection IoU with the number of performed rotation steps (Figure 3 in the paper). The results are based on the same set of 100 test tasks, including 400 human runs; the dashed line marks the success threshold of 0.90. After the task ends early, the final IoU is retained until the 48th step.
But the success rate only indicates whether the task is finally completed. To understand the gap, we need to go back to each step: whether the direction is selected correctly, whether the angle is appropriate, and whether the change of projection is consistent with the change of 3D pose?
Which direction to turn, how many degrees to turn, what are the respective difficulties?
Is the reason why the model failed to complete the task only that the surface of the object it saw is not enough? The paper first checked the cumulative surface observation coverage: GPT-5.6-Sol is 79.58%, and humans are 83.07%. The total coverage of the two is close, but the specific areas that have been seen and how to use these observations may still be different.
Therefore, the researchers further split each rotation into two choices: the direction, that is, the rotation axis and the positive/negative direction; the angle, that is, how large the amplitude of this step is.
First look at the direction. If along a certain direction, there is at least one legal angle that can reduce the 3D pose error, this direction is recorded as an "effective direction".
The proportion of GPT-5.6-Sol selecting the effective direction is 58.75%, and that of humans is 83.91%; the corresponding value of uniform random selection is 53.78%. This shows that selecting a rotation direction that can improve the pose is already a difficult problem for the model.
Then look at the angle. Even if the effective direction is selected, turning too much or too little may affect the alignment. The researchers compared the angle selected by the model with the legal angle that can best reduce the pose error within the selected effective direction. The average angle difference of GPT-5.6-Sol is 6.49°, and that of humans is 2.99°; the proportions of selecting the optimal legal angle are 61.80% and 71.82% respectively.
Direction and angle selection diagnosis, data extracted from Table 1 in the paper. Angle-related indicators are calculated only when the selected direction is effective; "effective direction" does not guarantee that the actually selected angle can also improve the pose.
The errors in direction and angle together constitute the gap at the action level. By splitting the two, we can further judge: whether the model has found a suitable rotation direction, and whether it can make fine adjustments along this direction.
The researchers also analyzed the correlation between the rotation angle and the pre-action projection IoU, 3D pose error. For GPT-5.6-Sol, the correlation between the angle and the projection IoU is stronger; the angle selected by humans has a strong correlation with both. This is consistent with the relative pattern of most models in the paper, suggesting that the angle selection of the model may be more related to the visible projection overlap.
Absolute value of Spearman correlation coefficient between rotation angle and pre-action projection IoU, 3D pose error (Figure 4 in the paper). Blue represents IoU-angle correlation, orange represents pose error-angle correlation; the error bar is the result of mapping the original 95% bootstrap interval to the absolute value. This figure shows the strength of the correlation, and does not directly reveal the internal judgment mechanism of the model.
The contour looks more similar, but the pose may be more deviated
After rotation, the current contour overlaps more with the target, which seems to be a good signal. However, different poses of a 3D object may produce similar projections, and the local improvement on the 2D picture is not always synchronized with the reduction of 3D pose error.
In order to compare this phenomenon, the researchers found 211 pairs of state pairs with the same pose in the trajectories of humans and GPT-5.6-Sol, and observed the actions they chose next.
Among these pairs, 19.4% of the model's actions improved contour overlap but made pose alignment worse; the corresponding proportion for humans is 10.9%. Conversely, humans more often choose actions that temporarily reduce projection overlap but reduce 3D pose error, with a proportion of 33.2%, compared with 23.7% for the model.
Action results under 21