Is Scaling Law a One-Trick Pony? The First Benchmark for Crystal Structure Manipulation Exposes Top Large Models' Collective Failures

Large language models have hit a bottleneck in atomic manipulation tasks. Despite their ability to parse materials science knowledge, they struggle to precisely control atomic structures. Research points out that the Scaling Law shows limited effectiveness in spatial logic tasks, emphasizing that AI for Science must shift toward Action Scaling to enhance models' capabilities in real scientific research operations.

Over the past few years, the most successful takeaway in the large model field has undoubtedly been the "Scaling Law". An almost universally accepted consensus across the industry is that as long as the model is large enough and the dataset is sufficiently extensive, capabilities will keep emerging, even generalizing automatically to uncharted domains.

However, a newly released materials science benchmark has offered a different perspective on this optimistic "bigger is better" mindset.

Developed collaboratively by institutions including the Suzhou Advanced Research Institute of the University of Science and Technology of China and the University of New South Wales, and presented at ICML2026, AtomWorld draws conclusions through a series of real atomic manipulation tasks: the Scaling Law, which performs reliably in scenarios like text comprehension and knowledge induction, often fails to meet expected performance when applied to hands-on atomic tasks constrained by physical rules.

Paper Link: https://arxiv.org/abs/2510.04704

Project Homepage: https://masterai-eam.github.io/atomworld/

Code Repository: https://github.com/MasterAI-EAM/atomworld

Comprehension Does Not Equal Manipulation

In the scientific domain, large models have already demonstrated remarkable "comprehension capabilities": reading academic literature, predicting material properties, analyzing crystal structures, and even making scientific discoveries.

For example, Anthropic launched Claude Science, an AI scientific research workstation that breaks research work into an auditable step-by-step pipeline, boosting efficiency by 10 times in specific tasks such as literature review writing and genetic analysis. Google DeepMind's GNoME uses graph neural networks to predict the stability of inorganic crystals, producing approximately 2.2 million structures through a closed loop of "generating candidates → DFT verification → data feedback".

This has led to a widespread industry belief: since models can understand materials-related knowledge, completing hands-on tasks like atomic construction and structural adjustment should logically follow.

However, real materials computing research is not a simple multiple-choice quiz. Daily scientific work is filled with highly concrete operational instructions: constructing the (001) surface of a specific material to simulate the "real-world" boundaries; replacing atoms at specific lattice sites to dope or modify the material; embedding new atoms at designated interstitial positions to design "energy storage" and "transport" channels.

Such tasks demand entirely different capabilities from the model: 3D spatial manipulation that complies with the laws of physics.

To objectively quantify this capability, the research team built the AtomWorld evaluation framework, which enables automated assessment based on the universal crystallographic information in the materials field. It does not test problems related to material recognition or theoretical analysis, but only focuses on fundamental spatial operation tasks: can the model precisely adjust atomic arrangements according to instructions?

Figure 1: Schematic diagram of the AtomWorld benchmark workflow. AtomWorld generator process: 1. The random sampler retrieves pre-defined atomic structures; 2. The random initializer configures atomic indices and position parameters; 3. Structural operators compute to obtain the target structure; 4. The prompt module generates the corresponding natural language description. The produced structure-text paired data is fed into the large model agent, and the model's output structure is compared with the standard target structure via the StructureMatcher tool from pymatgen to quantitatively evaluate the model's performance.

The Scaling Law Encounters Its Capability Boundary

Figure 2: Overall performance of different models on AtomWorld. a and c show success rates; b and d show the mean max_dist geometric error. The left panel compares different mainstream models, while the right panel compares Qwen models of varying sizes. Scaling up model size can improve performance on some tasks with clear rules, such as atomic replacement, deletion, and movement; however, improvements are unstable for operations that require 3D spatial understanding and geometric planning, like rotation, region deletion, and supercell expansion. Even highly capable general models such as Claude perform poorly on tasks like "rotating around an atom".

The results from AtomWorld indicate that the Scaling Law cannot be simply interpreted as "the larger the model, the stronger the capability" for atomic manipulation tasks.

This test covers mainstream models including Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, Gemini 2.5 Pro, Qwen3-32B, GPT o3, GPT-4o-mini, DeepSeek Chat, and Llama3-70B. Figure 2 demonstrates that scaling up model size does improve some templatable operations with clear rules, but this improvement is unstable for tasks that rely on 3D spatial relationships.

Taking the Qwen model series as an example, from 4B to 32B, the success rate of tasks like atomic replacement, removal, and movement has significantly improved, proving that scaling still delivers value. However, this improvement is mostly limited to tasks with well-defined rules and relatively fixed workflows, and cannot automatically generalize to all atomic manipulation tasks.

More challenging tasks reveal obvious bottlenecks. A typical example is "rotation around an atom": this task consistently shows extremely low performance across all sizes of Qwen models, and even on powerful models like Claude Opus 4.6, the success rate is only around 12%. This indicates that the problem is not just that a certain model is not large or capable enough, but that current general large models universally lack stable 3D spatial action capabilities.

Similarly, tasks like "deleting atoms below a target" and "expanding the supercell" still yield unstable results even when using larger models; geometric errors do not necessarily decrease as model size increases.

Therefore, AtomWorld does not simply negate the Scaling Law, but points out its applicable boundaries: scaling up can bring partial capability gains, but cannot automatically fill the core gaps in 3D physical space manipulation. For materials modeling, there is no direct equivalence between language reasoning ability, textual knowledge reserves, and atomic-level structural action capabilities.

In this sense, AtomWorld also points to a new direction: beyond pursuing parameter scale and text data scale, AI for Science also needs to focus on "Action Scaling".

This refers to systematically scaling up executable action data generation, action primitive decomposition, simulator feedback, physical constraint verification, and tool call error correction, so that models can become stronger not just in language, but also in verifiable scientific actions.

A New Track for Scientific Agents

The core value of AtomWorld goes far beyond identifying model failures: it breaks down the vague pain point of "materials agents cannot perform modeling" into a series of measurable, traceable atomic manipulation capabilities — ranging from basic element replacement to spatial region determination, and further to continuous geometric understanding, clarifying the type, severity, and scaling gain patterns of failures layer by layer.

This also exposes the root cause why pure parameter scaling struggles to deliver practical results: the existing Scaling Law focuses on fitting language and knowledge from massive text corpora, but the spatial understanding, geometric planning, and physical constraint action capabilities required for materials atomic modeling are extremely scarce in public datasets, where high-quality paired training samples of "operation instruction - coordinate change" are rarely found, making it nearly impossible to fill these gaps naturally through language scaling alone.

To address the poor 3D manipulation performance of large models, the industry commonly compensates by integrating professional tool libraries such as pymatgen. Controlled tests from AtomWorld show that external tools can only improve performance on tasks that heavily rely on coordinate calculation, like atomic insertion, while delivering very limited gains for complex scenarios requiring atomic relationship judgment and spatial region recognition.

Essentially, tools can only output precise coordinates, but cannot replace the model to make core decisions such as "where should the atom be placed" and "which atoms belong to the target region". If the model itself lacks 3D spatial awareness, tools will only execute incorrect intentions more precisely, ultimately leading to results with "modeling logic errors".

AtomWorld does not directly negate the Scaling Law, but reminds scientific agents to rethink "what exactly to scale". Language Scaling based on text corpora is the foundation of knowledge, but strongly operational tasks like materials modeling prioritize Action Scaling oriented toward action capabilities — turning the full "action - feedback - error correction" loop into an object for scalable learning.

The true significance of AtomWorld lies precisely in providing a foundation for action data and training loops for materials modeling by automatically generating tasks, standard structures, and matching feedback, driving AI for Science from pursuing larger general-purpose models to iterating real action capabilities in verifiable scientific operations.

Conclusion

AtomWorld is more than a standardized evaluation benchmark; it acts as an observation lens that intuitively reveals a key issue in the current development of AI for Science: a large model that can explain material structure and properties does not necessarily mean it can reliably modify material structures; a model that can read the periodic table of elements does not necessarily mean it can stably perform an atomic-level operation in 3D space.

This problem is not limited to materials modeling. Real scientific research is never purely text-based work, but consists of a series of actions including proposing hypotheses, designing experiments, invoking tools, adjusting parameters, observing results, troubleshooting errors, and continuous iteration. Whether it is materials modeling, molecular design, automated experiments, or broader scientific discovery workflows, if AI wants to truly participate in scientific research, it cannot only "explain knowledge" — it must also learn to "execute actions".

Therefore, AtomWorld reminds us to re-understand the applicable scope of the Scaling Law in scientific scenarios: Language Scaling based on web text corpora remains important, but it is only the starting point.

In the future, AI for Science will prioritize Action Scaling oriented toward action capabilities, allowing models to learn how to complete real scientific research tasks through executable operations, tool invocation, environmental feedback, and physical verification.

Only when models possess both knowledge comprehension and action capabilities can scientific agents evolve from "knowledgeable encyclopedias" that can answer questions into "practical experimental assistants" that can accomplish tasks.

References:

https://arxiv.org/abs/2510.04704

This article is from the WeChat public account "Xinzhiyuan", authored by LRST, and published by 36Kr with authorization.