GPT-6 Astra Overhauls Qwen Brain: Only 0.016% of Parameters Tuned, Math Performance Skyrockets

By modifying only 0.016% of the weights, the math performance of a large model with 800 million parameters skyrocketed from 7.9% to 48.1%.

Even more surprisingly, it was another large model, GPT-6 Astra, that made these weight modifications.

Recently, the research team from MIT CSAIL demonstrated a rather bold AI optimization method. They let GPT-6 Astra directly analyze the weights of the neural network, locate the model's problems, and modify the parameters inside, just like a programmer reading and modifying code.

Figure ｜ Research blog address: https://rikiyat.github.io/blog/weight-space-programming/

The entire process did not use traditional gradient training. The researchers modified only 131072 parameters of Qwen3.5-0.8B, boosting its performance on MATH-500 from 7.9% to 48.1%, its GSM8K score from 13.8% to 67.1%, and its instruction following capability was also significantly improved.

This research is called Weight-Space Programming.

After the news spread, some people believe that this is a "step-change capability" that has just emerged in the latest generation of models. If large models can directly read and write the weights of other models, in the future when AI optimizes AI, retraining may not even be required.

This is a completely new, step-change emerging capability that has only just appeared in the latest batch of models.

However, doubts soon emerged. Some pointed out that Qwen already had a serious problem of repetitive reasoning, and the sharp rise in math performance may only be because Astra fixed this Bug, which does not mean that the model has gained new mathematical capabilities out of thin air.

From 7.9% to 48.1%, what exactly did GPT-6 Astra do to this model?

From Writing Code to Directly Modifying Model Weights

Generally speaking, if we want AI to generate better responses, we usually can only modify the input prompts. If developers want to modify the output of the model, they can use data for fine-tuning.

Weight-space programming attempts to directly modify the weights of the trained model, that is, directly modify the numbers used in neural network calculations.

The so-called weights can be understood as a large number of adjustable knobs in the neural network model, each of which is a numerical value. Different values determine whether the data is amplified, weakened, or involved in calculations in other ways when passing through the network.

The model training that we often hear about is to a large extent constantly adjusting these knobs based on massive amounts of data, so that it gradually learns to produce correct outputs from inputs.

After training is completed, the model's knowledge, habits and behavioral tendencies are encoded in these values in some complex way; a model can have millions, billions or even more weights.

In the general perception, the part that large language models can participate in model iteration is limited to writing training code, adjusting hyperparameters, optimizing prompts, and even designing the complete training process. But in these tasks, the model weights themselves, which can reach hundreds of billions or trillions of parameters, are still mainly updated by optimization algorithms.

Now, the object of large model optimization has begun to sink from "how to train a model" to "what numerical values make up the model itself". Weight-space programming enables general large language models to directly read and modify neural network weights according to natural language goals; in some experiments, the model does not even require environmental feedback.

After Understanding Weights, AI Starts Writing Weights on Its Own

To prove the rationality of this breakthrough, the research team first tested whether large models can understand exactly what a neural network has learned only through its weights.

In the experiment, GPT-6 Astra could infer from the original weights of the model which game-specific optimized model each of several small networks with exactly the same structure corresponds to, and could also judge from the weights whether the diffusion model would generate circles, squares or triangles.

Faced with several small language models, it could also judge whether the training corpus belongs to Python, English or Japanese romaji only based on the weights of the middle layers, and find the attention heads responsible for copying previous content.

After proving that large models can read weights, the research team continued to test whether they can "write", that is, modify weights.

The research team used the reinforcement learning algorithm PPO to train the car control network for about 1 billion steps. As a result, the car drove straight up the slope and got stuck there; in 256 rounds of inspection, it never succeeded once.

Astra received the network weights, description of the scenario, inputs and outputs, as well as a record of a failed process. It submitted a parameter patch in one response, modifying only 3.5% of the weights, adding a control path of "back up first to build momentum, then move forward at full speed" to the network, increasing the success rate from 0 to 100%.

In another set of experiments, Astra generated the weights of the game network from scratch based on the game rules and network structure. These untrained networks outperformed the PPO network trained for 100 million steps in all three MinAtar games. In the 9×9 Go task, after limited forward testing and modification, the generated network also outperformed the PPO network of the same scale trained for 20 million games.

These results point to a fairly imaginative possibility: large models may be able to directly "compile" rules and strategies in natural language into neural networks.

Modifying Only 0.016% of Parameters, Math Performance Increased by 40 Percentage Points

What really made this research go viral was the modification to Qwen3.5-0.8B.

The small models for the previous mini-games could all convert their weights into text and then be processed by GPT-6 Astra, but a model with 800 million parameters cannot be directly stuffed into the context. The researchers gave Astra a Python environment that can read and modify weights, as well as practice questions and answers from the GSM8K and MATH training sets, requiring it to make this small model better at mathematics.

It could analyze the weights with the help of Python, test the model through dedicated tools, and also view the hidden states, neuron activations and attention information during model operation, but it could not use gradient information or carry out training.

Astra finally found that a large number of the model's responses fell into repetitive reasoning. Among the first 96 practice answers it checked, 85 kept generating up to the token limit of 32768 and still did not give the final answer. GPT-6 Astra located this problem to a specific direction written by some MLP layers, and suppressed the loop by deleting this direction.

Under the setting of allowing 50 forward tests, the modified model not only improved its math performance, with MATH-500 rising from 7.9% to 48.1%, and GSM8K rising from 13.8% to 67.1%.

Even the IFEval instruction following score rose from 21.0% to 62.3%; the proportion of responses that hit the token limit on MATH-500 dropped from 93.0% to 33.1%. The entire modification only involved 131072 parameters.

However, the research team also conducted corresponding supplementary experiments. For Gemma 3 270M, which rarely has reasoning loops, Astra's modifications did not bring clear improvements. Its baseline MATH-500 score was 6.8%, and the version after 50 forward tests scored 7.0%.

The research blog finally mentioned that Astra mainly removed obstacles that prevented the model from completing responses, rather than injecting new mathematical knowledge into the model, and it is unlikely to add capabilities that the model did not originally have.

Significant Capability Improvement, or Just Fixing a Bug?

On X, many discussions summarized this research as "let AI directly repair AI". If this method is reliable enough, in the future developers may not need to re-collect data and fine-tune for each model defect, but only need to let a stronger model check the weights, find faults and perform partial repairs.

Some netizens posted that they have reproduced the results in the blog, and believe that this is a "step-change capability" that has just emerged in the latest generation of models.

Other netizens said that their team had done similar experiments and found that Claude models can also write complete, runnable neural network weights from scratch.

The main voices of doubt come from some netizens who believe that the score improvement is mainly related to reducing loops, the idea used by Astra is similar to existing activation-guided methods, and the long prompts provided to Astra have clearly implied the loop problem and guided it to find neurons that can fix the problem.

Some people proposed that a simple control test should be conducted first: can directly turning off Qwen's thinking mode achieve similar improvements?

If adjusting the reasoning settings can greatly reduce loops, then how much additional value weight editing brings needs to be re-evaluated. Secondly, this method should also be compared with normal gradient training, not just with the original model that performs very poorly.

The controversy over reading and writing weights is another point of contention. Xingyu Dang, a PhD student at Princeton University, reported after running a Writing Weights Index test that open-source models have difficulty performing well on such tasks. This also means that reading and writing weights may be a capability that has just been demonstrated by the latest cutting-edge models, rather than a general skill that all large language models already possess.

The original blog itself also recorded an important failure: in the experiment of identifying the "sycophant model", Astra could not make a judgment when only comparing weights, and did not identify the difference until it was allowed to observe the model's attention patterns on specific texts. This shows that "directly understanding weights" is not yet a stable and universal capability.

Everyone Can Release Large Models

This work has not changed my judgment on how far old-school mechanism interpretability can go; what it has changed is my