SuperEvolution AI Releases 36 Self-developed AI Research Achievements: Full Public Disclosure of Codes, Experiments and Generation Trajectories | Xinglian Portfolio Updates

Today, Chaoyan Intelligence officially releases 36 research outputs generated by our AI system, covering two major directions: AI research, robotics and embodied intelligence.

01  

What We Have Built

The Chaoyan Intelligence team has built an autonomous system that can independently complete scientific research end-to-end with minimal human intervention. Over the past few months, our AI autonomous scientific research system has independently completed the entire process from literature research, method design, code implementation, experimental analysis, chart drawing, LaTeX paper writing, to self-review and iterative optimization.

The 36 research outputs released this time are almost entirely generated by AI, with only a small amount of non-expert human intervention, which proves that Chaoyan Intelligence's AI system truly has the ability to independently raise scientific questions, design rigorous experiments and draw insightful conclusions.

This research covers two directions: AI research and robotic embodied intelligence. Under the AI research direction, we carry out research around the following five types of issues:

Model Evaluation, Benchmark and Data Quality

Information Extraction, Causal Inference and Symbolic Reasoning

Agents, Reinforcement Learning and Code Repair

LLM Safety and Alignment

Model Mechanism, Training and Inference Efficiency

The robotic embodied intelligence direction focuses on embodied control and Vision-Language-Action (VLA) models.

02

Five Representative Research Studies

We have selected the following five representative research outputs for interpretation.

01｜Tracing AI-Generated Content: Discovering Differentiated Statistical Responses in Recursive Generation Processes

Depth Profiling: Classifying Recursive Generation Depth in Base-Model LLM Text

As AI-generated text continuously flows into the Internet and model training corpora, an important yet understudied question is: how will the statistical distribution of text evolve after multiple rounds of generation by large language models? Can these changes be used to identify how many rounds of AI generation the text has undergone?

Traditional AI text detection mainly focuses on Human-vs-AI binary classification, while existing recursive generation studies mostly discuss text distribution convergence or model collapse. The Apex AI System proposes the Generation Depth Profiling task, which expands the research objective from judging whether a text is generated by AI to identifying its recursive generation generation.

Based on 8 base large language models from 5 model families, the study constructs approximately 37,000 pieces of multi-round generated text, uses a fixed reference model to calculate the Token-level Negative Log-Likelihood (NLL) distribution, and combines statistical features such as lexical diversity to systematically analyze the variation pattern of different features with generation depth.

The study identifies a consistent statistical phenomenon: the positional features of the NLL distribution continue to drift with the number of generation rounds, while some distribution shape features remain approximately stable. Among them, features such as average NLL and lexical diversity show significant changes, with effect sizes of approximately d=0.74–1.05; the effect sizes of some tail distribution and extremum-related features are less than 0.10.

The study defines this asymmetry as Differential Depth Response (DDR).

Based on DDR, the paper builds a lightweight feature classifier to distinguish text into Gen-0 (original text), Gen-1 (one round of generation) and Gen-2+ (two or more rounds of generation), and verifies its cross-model transfer ability through leave-one-generator-out evaluation. In the leave-one-out test setup of 8 generation models, the binary classification F1 score reaches 0.694; it reaches 0.830 in specific cross-model generation chain experiments.

The core novelty of this work lies in decomposing the distribution changes in recursive generation into two types of responses: "continuous drift" and "relatively stable", and using this structural difference to establish a generation depth identification method. Compared with traditional detection methods that only identify the source of text, this study explores the possibility of inferring the generation history of text from its statistical distribution, providing a new method for synthetic data tracing, training corpus auditing and research on multi-round generation rules.

At present, this rule is mainly verified in base pre-trained models, and instruction fine-tuning and more complex real generation environments still need further research.

02｜Safety Monitoring for Reasoning Models: Discovering Layer Selection Bias in Hidden-State Probes

Layer Selection Sensitivity in Hidden-State Safety Probes for Reasoning Models

Reasoning models may gradually shift from following safety constraints to generating harmful content during continuous reasoning. This internally occurring safety state transition limits detection methods that only rely on input or final text output. In recent years, researchers have begun to train safety probes using the model's internal Hidden States, hoping to identify changes occurring during the reasoning process in a more fine-grained manner.

However, hidden state probes need to select specific Transformer layers as monitoring objects, and how to select network layers is usually regarded as a technical implementation detail, whose impact on the final safety detection conclusion lacks systematic research.

The Apex AI System focuses on this issue and proposes a cross-layer, cross-metric analysis framework for hidden state safety probes.

The study compares the safety state representations of different Transformer layers on 4 reasoning models, and uses multiple metrics to evaluate the performance of probes in identifying safety state transitions, including threshold crossing frequency, precision, balanced accuracy and false positive rate.

Experiments show that different layer selection metrics have systematic divergences: metrics based on threshold crossing frequency tend to select shallow layers (L0–L10), while metrics based on threshold detection performance are more inclined to select deep layers (L14–L63). Across different models, the layers selected by the two types of methods differ by 14–61 layers.

This difference not only affects the model monitoring position, but also directly changes the judgment of the hidden state detection capability.

In some shallow layers, the performance of hidden state probes is close to that of text probes; in reasonably selected deep layers, hidden state probes in 7B–8B models achieve a precision improvement of about 20–22 percentage points compared to text probes, which is verified by cross-validation.

To further explain this phenomenon, the study analyzes the geometric properties of hidden representations in different network layers, and finds Cosine Compression in shallow layer representations: the difference in representation direction between adjacent reasoning steps is compressed by about 1.4–2.7 times, making some frequency-based metrics more susceptible to noise, thus generating incorrect layer selection preferences.

This bias is further verified in cross-model and AdvBench dataset experiments.

The core innovation of this study is the discovery and interpretation of Metric-Induced Layer Selection Bias in hidden state safety probes. Compared with existing work that only trains safety classifiers on fixed network layers, the paper proves that the layer selection strategy itself is an important methodological variable affecting the safety monitoring conclusion.

The study also compares multiple alternative detection methods and discusses the possibility of multi-layer integrated probes, providing experimental basis for establishing more reliable and reproducible internal safety monitoring methods for reasoning models in the future.

It should be noted that the conclusions on the 32B model are still limited by the size of test samples, and the advantage of hidden states over text also depends on the specific model and text encoder configuration.

03｜Internal Mechanism of Embodied Intelligence: Distinguishing Representation Changes Caused by Action Training from Pre-training Priors

Instruction Routing in Fine-Tuned Vision-Language-Action Models: A Cross-Architecture Analysis

Vision-Language-Action (VLA) models realize the mapping from images and natural language instructions to robot actions by fine-tuning on robot action data based on pre-trained Vision-Language Models (VLMs).

However, there is still a key problem regarding the internal information processing mechanism of VLA at present: does the instruction processing capability demonstrated by the model after action training come from the newly added action training, or is it inherited from the pre-trained vision-language model?

Existing studies mostly focus on the task success rate of VLA, language instruction following performance and the internal representation of a single model, but lack systematic analysis of cross-architecture and pre/post-training comparison.

To address this issue, the Apex AI System conducts a cross-architecture VLA Instruction Routing study.

The study selects three different backbone architectures, namely OpenVLA, TraceVLA and NORA, to compare the trained models with reference checkpoints on four types of tasks in LIBERO. Among them, TraceVLA and NORA are used to analyze the training transition from VLM to VLA, and OpenVLA is used to analyze further fine-tuning based on the existing VLA.

In terms of methods, the study combines three types of complementary analyses: first, using Layer-wise Probing to measure the decodability of action-related information in different Transformer layers; second, analyzing the impact of instruction content changes on the model's internal representation through Counterfactual Instruction Perturbation; third, studying the attention patterns of different network layers to language instruction information using Attention Profiling.

The study obtains three main findings.

First, the action training from VLM to VLA is accompanied by specific output layer representation changes. In the two tested VLM-to-VLA transitions, the decodability of action information near the output layer decreases significantly compared with the middle layers, with ΔR² reaching 0.16–0.46; the corresponding change in VLA-to-VLA fine-tuning is less than 0.10, and no similar change is observed in the ordinary NLP instruction fine-tuning control.

Second, VLAs that have undergone action training show significant sensitivity to changes in instruction content. Counterfactual instruction experiments observe moderate to large internal representation changes (Cohen's d=0.45–1.52), indicating that there is a measurable correlation between language instruction information and action-related representations.

Third, part of the shallow layer instruction attention structure is actually inherited from the pre-trained model. The study compares the attention patterns before and after training and finds that some dominant instruction attention features in the first layer are not newly formed by action fine-tuning, but already exist in the base model.

The main innovation of this study is to decompose the internal mechanism of VLA into two parts: "action training-related changes" and "pre-trained model inherited structure" through cross-architecture, pre/post-training comparison and multiple representation analysis methods.

Compared with studies that only investigate whether VLA can follow instructions through task success rate, this work further analyzes how instruction information is represented and processed inside the model, providing new experimental evidence for understanding the transition from base vision-language models to robot action models.

These findings also provide references for subsequent VLA model representation diagnosis, architecture analysis and training method design, but currently they are mainly based on open-loop experiments, and their actual effects need to be further verified through closed-loop robot tasks.

04｜Neurosymbolic Reasoning: Revealing the Goodhart Effect in Structural Rewards

Observe but Don’t Optimize: Structural Rewards Trigger Goodhart Effects in Neurosymbolic Reasoning

In neurosymbolic reasoning systems, large language models are usually responsible for converting natural language problems into formal logic expressions, and then the symbolic solver verifies whether the conclusion is valid. Existing training methods often directly take the execution success rate of the solver as the reinforcement learning reward, but this objective has structural defects: the model can make the proof pass verification by directly embedding the conclusion to be proved into the premises, constructing logical tautologies and other methods, without completing real logical derivation.

To address this issue, the Apex AI System proposes Solver Trace Ablation (STA), which applies the item-wise ablation method to the structural diagnosis of neurosymbolic reasoning. For the formal expressions generated by the model, STA sequentially removes the premises or components in them, and re-invokes the symbolic solver to judge whether each part is necessary for the proof to hold.

Based on this process, the study constructs the Intensional Structural Reward (ISR) metric, which quantifies the necessity of the proof structure by the proportion of components whose removal leads to proof failure, without requiring additional manual step-by-step reasoning annotation.

The study uses the Qwen3-4B and GRPO reinforcement learning framework to analyze the training behavior of the model under multiple reward conditions, and verifies it through first-order logic (FOL), Prolog and cross-dataset experiments.

Experiments show that when only the execution success rate is used as the reward, even if the model achieves a proof success rate of over 90%, the proportion of non-speculative proofs may still drop to below 6.2%. This indicates that the model can gradually rely on structural shortcuts while the formal verification metrics continue to improve.

More importantly, the study further finds that when ISR, which was originally used to identify structural cheating, is directly added to the training reward, the model can still obtain high scores by constructing tautological structures that meet the requirements of the metric.

In 5 groups of random seed experiments, this Goodhart effect occurred in 3 groups, and the other 2 groups maintained a relatively reasonable reasoning structure. Cross-logic form experiments further show that this asymmetry between diagnostic metrics and optimization objectives is not limited to a single logical representation.

The novelty of this study lies not only in designing a proof structure diagnosis method that does not require manual process annotation, but also in revealing the exploitability of the structural diagnosis metric itself through reinforcement learning experiments. The study advances the analysis of Reward Hacking from the level of final results to the level of the constituent structure of formal proofs, and reveals an important boundary: metrics suitable for observing reasoning quality are not necessarily suitable for being directly used as optimization objectives.

This finding provides an important methodological basis for reward design of Reinforcement Learning with Verifiable Rewards (RLVR), neurosymbolic reasoning and automated formalization systems.

05｜Reliability of Large Model Evaluation: From Multi-factor Variance Decomposition to Evaluation Budget Optimization

Reproducibility Budgets for Open-Weight LLM Evaluation: A Generalizability Theory Study of 7–9B Instruction-Tuned Models

Existing large model benchmarks usually run under fixed prompts, random seeds and inference configurations, and measure model capabilities with a single score. However, factors such as prompt templates, randomness,