The U.S. Argonne National Laboratory has proposed ChemGraph, which uses 13 benchmark tests to evaluate the value of Agent in the field of computational chemistry

A research team from the U.S. Argonne National Laboratory has proposed an LLM-driven agent system called ChemGraph, designed to execute molecular simulation workflows in the field of computational chemistry.

In recent years, the rapid development of artificial intelligence, especially the advances in Large Language Models (LLMs), has opened up new possibilities for automated scientific research. Multiple LLM-based agents have been developed to assist users in completing various chemistry-related tasks — for example, Bran et al. developed ChemCrow, an LLM-driven chemistry agent capable of performing various molecular chemical synthesis tasks; McNaughton et al. developed CACTUS, an LLM agent integrated with cheminformatics tools, which can help researchers complete tasks such as molecular property prediction, similarity search, and drug-likeness assessment; White and his colleagues proposed MDCrow, an LLM agent assistant that can execute molecular dynamics (MD) workflows. 

Against this background, the research team from the U.S. Argonne National Laboratory proposed ChemGraph, an LLM-driven agent system, which aims to implement molecular simulation workflows in computational chemistry. 

The researchers evaluated ChemGraph across 13 benchmark tasks. The results show that for simple tasks that only require a small number of tool calls, smaller LLMs (such as GPT-4o-mini and Claude-3.5-haiku) can achieve high accuracy and stability. However, as task complexity increases, the performance of these models drops significantly, while the large model (GPT-4o) maintains strong performance. By strategically breaking down complex tasks into smaller, more manageable subtasks, even using smaller-scale models can improve the performance of ChemGraph, bringing it to a level comparable to GPT-4o, and even exceeding the performance of GPT-4o in some cases. 

The related research, titled "ChemGraph as an agentic framework for computational chemistry workflows", has been published in Nature journal's Communications Chemistry. 

Research Highlights:

* ChemGraph leverages graph neural network-based foundational models to achieve accurate and efficient computations, while combining the LLM's capabilities in natural language understanding, task planning, and scientific reasoning to provide an intuitive and interactive operating interface

* ChemGraph can complete a series of tasks ranging from SMILES string and molecular structure generation to geometry optimization, vibrational analysis, and thermochemical calculation

* ChemGraph supports users in using multiple molecular simulation methods, including semi-empirical methods, machine learning potentials, and density functional theory (DFT)

Paper Link: https://www.nature.com/articles/s42004-025-01776-9

13 Benchmark Experiments to Establish an Evaluation System for Computational Chemistry Agents

Although some agent benchmark testing frameworks already exist, there is no unified benchmark for agent evaluation in the field of computational chemistry. Therefore, the researchers designed 13 benchmark experiments to evaluate the capabilities of ChemGraph. These experiments are designed to test ChemGraph's ability to complete tasks using six integrated tools, including single tool calls and chained tool calls. 

According to different user input types, these tasks are divided into three categories: (1) Molecular names; (2) SMILES strings; (3) Chemical reactions. The table below summarizes the 360 independent evaluations conducted across the 13 experiments. 

*Summary table of benchmark experiments for evaluating ChemGraph*

In the first 11 experiments, tasks mainly revolve around molecular names or SMILES strings, with each task requiring up to 4 tool calls or completing 4 subtasks. In contrast, the last two experiments focus on the calculation of thermochemical properties of chemical reactions, with significantly increased complexity, requiring 9 to 12 tool calls depending on the number of reactants and products. These tasks require performing thermochemical calculations for each substance separately, and further constructing reaction-level properties based on previously obtained results.

ChemGraph: The LLM-Driven Agent Framework

ChemGraph is an LLM-driven agent framework designed to automate molecular simulation workflows through structured tool invocation and reasoning capabilities. It is implemented based on LangGraph and follows the ReAct framework.

LangGraph introduces a graph structure-based execution model, aiming to achieve more robust multi-agent collaboration. A typical LangGraph workflow consists of three components:

• State: A state is a shared data structure that represents the current state of the system.
• Nodes: Nodes are Python functions that define agent logic. They take the current state as input and return an updated state; nodes can be LLMs or functions that perform specific operations.
• Edges: Edges are functions that control the flow of messages, determining which node to execute next. Edges can be directed edges or conditional edges — directed edges represent a fixed one-way flow of messages from one node to another; in contrast, conditional edges provide greater flexibility, allowing messages to flow to different nodes based on specific conditions.

The figure below illustrates the general structure of ChemGraph — first, the LLM agent is provided with a set of predefined tools. Based on the user prompt, the agent decides which tool to call; after each tool call is completed, the LLM receives the tool output and determines whether another tool call is needed; when all tool calls are finished, the message can continue along one of two paths.

*ChemGraph Overview*

In the first path, ChemGraph is similar to a traditional LLM, returning a human-language response containing the results generated from tool calls; in the second mode, the message is forwarded to a second LLM agent, which is responsible for generating structured outputs (in a predefined JSON format) to directly respond to user requests. This dual-mode design supports both natural language interaction and systematic evaluation, enabling ChemGraph to not only meet typical usage scenarios but also be applicable to evaluation workflows.

The figure below further demonstrates how ChemGraph calls tools and coordinates tool outputs to complete computational chemistry tasks:

*Example of human interaction with ChemGraph (single agent) while performing the react2enthalpy task using GPT-4o-mini* 

In this example, the user asks ChemGraph to calculate the reaction enthalpy of methane combustion at 400 K using the GFN2-xTB method. ChemGraph (powered by GPT-4o-mini) first converts the chemical name of each molecule in the reaction to a SMILES string (tool calls 1 to 4). Then, based on these SMILES strings, it generates atomic coordinates in the AtomsData data structure (tool calls 5 to 8). Finally, ChemGraph performs thermodynamic calculations using the generated atomic coordinates and user-specified parameters (such as calculator type, temperature, etc.).

Multi-Agent Architecture Can Significantly Improve Model Performance

Through 13 benchmark task tests, the research team systematically evaluated ChemGraph's performance in computational chemistry tasks of varying complexity: 

Single-Agent Evaluation

First, the researchers evaluated the performance of single-agent ChemGraph across 13 experimental tasks for three LLMs — GPT-4o-mini, Claude-3.5-haiku, and Qwen-2.5-14B; for GPT-4o, only its performance on the last two problems (react2enthalpy and react2gibbs) was evaluated. The figure below shows the accuracy of different models on different tasks:

*Accuracy heatmap for single-agent ChemGraph* 

In the name2smi task — Claude-3.5-haiku does not consistently call the molecule_name_to_smiles tool, but sometimes tries to manually construct SMILES strings using its internal chemical knowledge, or refuses to use the tool. As a result, Claude-3.5-haiku has the lowest average number of tool calls per molecule among the three models in this task; GPT-4o-mini can accurately complete this experiment and generate correct answers; Qwen-2.5-14B usually generates accurate tool calls, including correct tool names and parameters, but its errors mainly occur in the process of extracting results from tool outputs.

In the name2xyz, name2opt, and name2vib tasks — LLMs are required not only to call tools accurately but also to generate correct and structured outputs. In this regard, both GPT-4o-mini and Claude-3.5-haiku performed well, with accuracy exceeding 83%.

In the smi2xyz to smi2file tasks — Claude-3.5-haiku and GPT-4o-mini maintain strong and stable performance, with both achieving accuracy above 83%; Qwen-2.5-14B also shows improved performance in this group of tasks, with a minimum accuracy of 77%.

In the react2enthalpy and react2gibbs tasks — Qwen-2.5-14B has the weakest performance, with accuracy below 20%; GPT-4o-mini achieved moderate results, with accuracies of 40% and 49% respectively; Claude-3.5-haiku outperformed other small-scale LLMs, with accuracies reaching 67% and 69% respectively; GPT-4o achieved the highest performance, with an average accuracy of over 83% on both tasks.

Multi-Agent Evaluation

The researchers further evaluated the performance of multi-agent ChemGraph using GPT-4o-mini, Claude-3.5-haiku, Qwen-2.5-14B, and GPT-4o (as shown in the figure below), and compared the results with the single-agent performance.

*Average accuracy of multi-agent and single-agent ChemGraph on the react2enthalpy and react2gibbs tasks when using different LLMs*

The results show that the multi-agent system significantly improves the accuracy of different models. In the react2enthalpy task, the accuracy of GPT-4o-mini increased from 40% to 87%, and that of Claude-3.5-haiku increased from 67% to 87%, both results being the average of three independent runs. Notably, both models exceeded the 83% baseline performance of single-agent GPT-4o. A similar trend was observed in the final task react2gibbs: the accuracy of GPT-4o-mini increased from 49% to 87%, and that of Claude-3.5-haiku increased from 69% to 93%. In contrast, Qwen-2.5-14B only achieved limited improvement, mainly due to its frequent tool call errors, which limited the ability of the aggregation agent to generate accurate answers.

Ultimately, in the multi-agent mode, GPT-4o achieved perfect accuracy, completing both tasks (react2enthalpy and react2gibbs) with a 100% success rate.

Closing Remarks

As an LLM agent system for computational chemistry and materials science, ChemGraph demonstrates the potential of AI-automated scientific research workflows. The research team stated that in the future, they will further expand ChemGraph's application capabilities in large-scale simulation tasks by integrating more tools, optimizing the agent architecture, and enhancing high-performance computing support. At the same time, the system ensures operation security through Docker containers and continuously improves the reliability of the agents.

It is worth noting that ChemGraph is not intended to replace traditional computational chemistry software, but rather acts as an intelligent collaboration layer connecting researchers and simulation tools through natural language interaction. With the development of open-source large models, ChemGraph is also expected to reduce the cost of AI scientific research applications and promote intelligent scientific agents to play a greater value in fields such as materials discovery and molecular design.

References:

https://phys.org/news/2026-07-ai-framework-battery-combustion-materials.html

https://www.nature.com/articles/s42004-025-01776-9

This article is from the WeChat public account"HyperAI HyperNeuron", author: HyperAI, published with authorization from 36Kr.