AI Scientists "Ultimate Test", the world's first Physical AI Verifier for real wet experiments is here

Not long ago, Anthropic demonstrated an AI for Science research achievement: Claude generated 318,000 protein design schemes within 64 hours. After screening, 211 of them were sent to wet laboratory for testing, and finally 17 met the preset protein binding standards, showing the huge potential of AI scientific design moving towards real experimental verification. However, Anthropic also pointed out that from binding hits to truly usable anti-toxin therapies, there are still a series of challenges such as complex activity verification and druggability assessment.

This reveals one of the most acute realities of current AI for Science: Simulating schemes in the virtual world is completely different from conducting experiments in the real physical world.

So, how deep is the gap between the scheme proposed by the model and the real physical experiment? No one could explain it clearly in the past, because the industry lacked a set of physical verification benchmarks that can be executed repeatedly and compared horizontally. Code has unit tests, mathematics has standard answers, but real experiments are restricted by equipment differences, execution errors and costs, and results from different laboratories are also difficult to compare directly.

Recently, AI Scientists' Last Exam (AISLE), jointly launched by Rhythm Creation and many top global scientific research institutions, attempts to establish a new "ruler" for the real experimental capability of AI. As the world's first public benchmark for closed-loop scientific discovery of dry and wet experiments, AISLE hopes to bring different models into real laboratories and accept the test of the physical world under unified standards.

Project address: https://aisle.science/

Github repo: https://github.com/MasterAI-EAM/ai-scientists-last-exam

Paper link: https://github.com/MasterAI-EAM/ai-scientists-last-exam/blob/main/paper/AI_Scientists_Last_Exam.pdf

The result is that among the first round of 4079 real wet experiment tests, the top participating models such as GPT, Claude and Gemini handed in a warning-filled report card: Among the models that completed the wet experiment list evaluation, GPT 5.5, which ranked first overall, only scored 41.19 out of a full score of 100; in the experiment optimization task with safety constraints, the full pass rate of all participating models was 0%, and the score of GPT 6 Astra even decreased.

This result completely breaks the "test-taking aura" of large models on the leaderboards.

In the computer, even if the model calculates "1+1" 100 million times, the answer is still 2. But in a real wet laboratory, if an automatic pipetting device repeats the operation 100 million times, the results will never be exactly the same. The equipment will wear out, the accuracy will drift, the measurement data will have noise, and the experimental environment may change at any time...

What is more alarming is that the enhancement of scientific capabilities does not mean the simultaneous improvement of safety capabilities. When the model directly calls experimental equipment, a decision that violates safety constraints may turn from an error in the digital world into equipment damage, personal injury or even environmental pollution in reality. Even worse, if the model cannot identify the dangerous intentions gradually revealed in multi-round interactions, its scientific reasoning capabilities may be maliciously exploited.

This is another significance of AISLE. When global tech giants are competing to depict the technical vision of "AI Scientist", this physical exam reveals a cruel reality: The capability boundary of AI Scientist cannot only be judged by whether it can put forward correct hypotheses and complete experimental goals, but also by whether it can abide by safety boundaries throughout the whole experimental process.

It is worth noting that this milestone work in the industry brings together a number of cutting-edge research forces in the fields of automated laboratories, computational chemistry and AI for Science. The project was initiated by Dr. Xie Tong, founder of Rhythm Creation, and co-led by the University of New South Wales (UNSW), the National University of Singapore (NUS), École Polytechnique Fédérale de Lausanne (EPFL) and the University of California, Berkeley (UC Berkeley). The participants include 13 global cutting-edge AI4Science laboratories such as materials scientist Gerbrand Ceder from UC Berkeley, Philippe Schwaller from EPFL, pioneer of chemical Agent, and the Acceleration Consortium team, a pioneer in the field of chemical embodied and self-driven laboratories (SDL), as well as researchers from NVIDIA, Microsoft Research, Eli Lilly and other institutions.

Among them, Gerbrand Ceder's team once led the construction of the autonomous materials laboratory A-Lab, and cooperated with Google DeepMind's GNoME material discovery results. In 2023, relevant research was published in *Nature*, demonstrating the potential of combining AI material prediction with robotic automated synthesis.

The "Last Mile" of AI for Science: From Theory and Computation to Experiment

From the perspective of scientific discovery methodology, a complete scientific research cycle usually consists of three stages: theoretical deduction, numerical calculation and simulation, and the final physical experiment execution.

In recent years, models from AI giants such as OpenAI and Anthropic have shown increasingly amazing generalization capabilities, making continuous new breakthroughs in mathematical reasoning and scientific problem solving, and AI has also entered scientific research processes such as protein structure prediction, molecular design and material discovery.

But the ultimate closed loop of scientific discovery must eventually fall into real experiments. Only experiments can give the final judgment on whether the hypotheses proposed by the model can be supported by real observations and whether the optimal scheme in calculation can be established in the physical world.

However, the problem is that the feedback of physical experiments is far more complex than digital tasks. One experiment not only requires equipment, reagents and time, but the results may also be affected by measurement noise, execution errors and equipment status. Even if the experiment fails, it is difficult to immediately judge whether the problem lies in the scheme, operation or the equipment itself.

This has made automated laboratories an important development direction of AI for Science, hoping to connect robots, experimental equipment and AI Agents, so that models can not only analyze existing data, but also actively design experiments, perform operations, read results, and then decide what to do next.

For example, in October this year, Google Research introduced the VERID framework, which tracks the status of robotic arms, reagent volumes and consumables through the semantic digital twin of the laboratory, allowing AI Agents to conduct virtual rehearsals before performing real operations, and reducing operational risks through safety checks.

However, enabling AI to operate experimental equipment only solves part of the problem, and AISLE further focuses on: how to measure the model's performance throughout the experimental process and convert experimental results into feedback signals that can be used continuously.

For a long time, real experimental feedback has been difficult to be efficiently and scalably used for model evaluation and training like code test results. How to convert sparse, delayed and noisy physical feedback into a reliable evaluation basis has become a problem that AI for Science must solve.

AISLE hopes to fill this gap: Let real experiments become not only the final examination room for AI, with the answers placed in the physical world. The model can memorize papers and brush through all question banks, but it cannot memorize the results of the next real experiment in advance. Equipment will drift, materials will change, and new experimental conditions will keep emerging — this is not an examination paper waiting to be fully solved, but an examination room that continuously produces questions, and also becomes a feedback source for its continuous learning.

AISLE: The "Final Exam" for AI Scientists, Nature as the Verifier?

The core idea of AISLE is to establish a distributed physical verifier that connects model reasoning, computational simulation and real experiments. It takes the automated laboratory as the execution carrier, forms a dry-wet closed loop of "proposing scheme — performing experiment — obtaining observation — adjusting decision", and converts the feedback of the physical world into signals that can be used for model evaluation and continuous improvement.

But unlike previous explorations relying on a single laboratory, AISLE connects various cutting-edge large models with automated laboratories in different locations and with different equipment through a unified task, action and observation protocol, and the team spent nine months to fully run through the whole process.

Among them, the unified harness allows results from different sites to be compared under the same standard, while equipment differences between sites are retained to test the robustness of the model under different conditions.

Figure a — The four-level ladder of scientific evaluation: HLE and GPQA only focus on static answers, BioBasicsBench released by OpenAI on November 6 tests the ability to read raw wet experiment data (the third level "experimental knowledge"), AISLE is at the top, allowing the model to conduct experiments in person on real equipment. Figure b — The dry-wet closed loop framework of AISLE: the model puts forward hypotheses, the dry laboratory (simulation and digital twin) screens candidates, distributed automated wet experiment sites execute, and the characterization results are fed back to the model; the complete interaction track is scored from four dimensions: optimization, safety, adaptive robustness and multi-modality.

Therefore, AISLE is not only a set of common examination questions for comparing the experimental capabilities of models, but also can provide repeatable real feedback for model improvement just like unit testing in the code field.

The first round of AISLE evaluation focuses on the dry-wet closed-loop capability of large models in real wet experiments, where the model proposes schemes on the dry end, and the wet end executes and feeds back real observations.

In order to test the basic decision-making ability of the model in the closed loop of dry and wet experiments, the team designed basic tasks such as optical target reverse design, pH-sensitive formulation optimization, irreversible operation planning of polymer compounds, active pKa inference based on spectral measurement under different pH conditions, and spectrum-image multi-modal reasoning, taking into account cost, feedback speed and repeatability, while retaining real operation errors and measurement noise.

In addition, the team introduced that complex cutting-edge scientific research tasks such as Buchwald–Hartwig coupling reaction and CdS-MPA quantum dot synthesis are currently being accessed.

More notably, AISLE introduces "digital twin" into this closed loop.

Rhythm Creation told *Machine Heart* that each experimental site has a corresponding digital twin environment. The model can first conduct a large number of deductions and trial and error in the digital environment, and then send the screened schemes to the real experimental equipment for execution; the observation results obtained from real experiments will in turn calibrate the digital twin, so that the simulation gradually approaches the actual behavior of the equipment.

This is equivalent to establishing two layers of mutually cooperative feedback mechanisms: the digital twin provides low-cost, high-density trial and error opportunities; real experiments provide the final physical verification basis.

In this way, the model does not need to consume real reagents and equipment resources for every exploration, and can continuously use real observations to correct the simulation environment.

Even GPT 5.5 Only Scores 41 Points, What Does the "First Ruler" Measure?

Previously, no one in the industry could definitely assert whether large models can replace humans to conduct experiments, because there was a lack of a set of standards that can systematically measure the real experimental capabilities of AI. Code has unit tests, mathematics has standard answers, but wet experiments have long been in a rule vacuum.

Now, AISLE acts as this "ruler", and the measured results are not only sobering, but also reveal the cliff gap between the physical world and the digital world...

The first round of AISLE evaluation covers 2 automated experimental sites, 4079 real experiments, 46626 minutes of running time, involving models such as GPT, Claude and Gemini, and evaluates model performance from four dimensions: "exploration efficiency, experimental safety, adaptive robustness and multi-modal reasoning", respectively investigating whether the model can approach the target within a limited budget, abide by safety constraints, cope with equipment changes, and infer scientific parameters from observation data such as spectra and images.

AISLE v0.1 cooperative experimental sites distributed around the world, from automatic pipetting to glove box synthesis, covering the core operation links of wet experiments

The results show that among the models that completed the evaluation of all dimensions of wet experiments, the highest score is GPT 5.5, but it only gets 41.19 points (full score 100). More notably, according to the safety full-pass standard adopted by the team, none of the participating models met all safety requirements, and the pass rate was 0%.

This report card first proves that this "ruler" is extremely accurate: it can quickly strip off the score foam of large models at the pure language level, and clearly stratify the hands-on ability of each model in the real physical coordinate system. Secondly, it proves that the actual gap between large models and real physical experiments is much deeper than expected by academia and industry, which can be seen from the problems exposed in the experiments. Reaching the end point may cover up violations in the process; one success may cover up continuous errors after equipment misalignment. What AISLE records is not just the right or wrong of the end point, but the complete decision-making link: when to cross the boundary, how to respond after discovering abnormalities, and when to stop for calibration. These signals that can only be generated by real execution cannot be obtained only by tests in the digital environment.

First, there is a "safety blind zone" where the model knowingly violates the rules. In a set of safety tests, the research team artificially set up restricted areas and informed the model of the constraints in advance. For example, in the color matching task, certain RGB value ranges are marked as unsafe areas, and the model must actively avoid them when looking for the optimal experimental path. But the fact is that in the experiment, almost all participating models touched the restricted areas in order to get close to the target without exception.

This shows that the fact that a model can understand safety rules does not mean that it can always abide by the rules in actual actions. The safety evaluation of AI Scientist in the future cannot only focus on whether it can complete tasks, but also needs to further cover complex intention recognition and risk prevention.

Second, there is "self-deception" that the model will not stop for calibration. Real experimental equipment will not always maintain the same accuracy. For example, the model requires the pipetting device to add a fixed volume of liquid each time, but as the equipment runs, the actual pipetting volume may gradually deviate from the expected value. Human experimenters usually detect abnormalities, check the equipment and perform calibration. However, in the AISLE test, although the model can adjust subsequent operations according to the measurement results and try to compensate for the deviation, it does not fully demonstrate the ability to actively identify equipment problems and propose calibration requirements.

In addition, there is no so-called "hexagonal warrior" in the evaluation results. No model leads in all four dimensions. Models that are good at experimental optimization may not be able to abide by safety constraints or cope with changing conditions; models that perform well in robustness may not be good at multi-modal reasoning.

This shows that experimental optimization, safety codes and adaptive anti-interference are completely different dimensions of intelligence, and a single end-point result can never prove that an AI can independently take over the laboratory.

Not Just an "Exam", But Also a Model Training Environment

On the whole, the first round of AISLE evaluation gives a common measurement standard for the experimental capabilities of models, indicating that current cutting-edge large models are far from being ready to become qualified independent "AI Scientists". But the ultimate goal of AISLE is not just scoring, the