The 535B large model has completed three months of "live-streamed" training, with all its code, data and loss fully disclosed, and Andrew Ng has publicly voiced his strong support for it.
When most model companies are still debating whether to open model weights, a large language model with a total parameter count of 535 billion has chosen to publish its training curves, data recipes, model configurations and technical discussions directly online, even before its training is completed and when there is still a risk of mid-training failure.
Recently, Percy Liang, Associate Professor of Computer Science at Stanford University and Director of the Center for Research on Foundation Models (CRFM), posted on X announcing that the open foundation model project Marin launched the training of Marin 535B-A23B last week.
According to the currently announced plan, Marin 535B-A23B will process 18.75 trillion Tokens, of which about 80% will be used for pre-training and 20% for mid-training; the training runs on 11 sets of NVIDIA GB200 NVL72 systems, which is expected to last for about 3 months with a total computation volume of approximately 2.7×10²⁴ FLOPs, and will enter the post-training phase afterwards.
According to the launch announcement of the Marin project, the project was first born at the Center for Research on Foundation Models (CRFM) of Stanford University, and was officially announced to the public in May 2025. The authors of the launch announcement include David Hall, Percy Liang, and a number of researchers from Stanford, Open Athena and the open community.
David Hall
Percy Liang
Percy Liang previously worked at conversational AI company Semantic Machines as Chief Scientist. The company was acquired by Microsoft in 2018, and its relevant team later participated in the technical development of Microsoft's conversational system and voice assistant.
In addition, he is also one of the co-founders of Together AI, a large model inference cloud and open source AI company. The business of Together AI includes model training, inference infrastructure and open model services.
The reason for launching the Marin project is that they want to explore a core issue in the AI field: in a scenario where computing power is highly concentrated and training recipes are increasingly closed, can foundation models still be publicly researched and co-built like open source software?
Previously, some of the major open weights on the market, such as models like Llama and Gemma, are open source, but the code and data used to generate these models, commonly known as "recipes", are not publicly available. Projects including BLOOM, Pythia, OLMo and LLM 360 have further opened up data, code, logs or intermediate checkpoints. Marin acknowledges the pioneering contributions of these projects, but argues that open models still lack a collaborative mechanism similar to that of software open source.
Software developers can view Issues, submit code, conduct Reviews and reproduce Bugs on GitHub, but foundation model experiments usually run in closed clusters.
What the outside world sees is often the model weights and technical reports after training is completed, with no visibility into why researchers made certain decisions, nor any access to failed experiments.
Therefore, Marin proposes an "open laboratory" mechanism: each experiment declares its goals and hypotheses in advance through GitHub Issues, the specific configuration is submitted as code and Pull Requests, external researchers can participate in Reviews, and W&B training metrics are made public after the experiment starts. More importantly, all traces of success, failure and mid-process modifications are recorded during this process, and data, code, recipes and the final model remain open.
As of now, Marin has trained 8B and 32B models, and has begun to scale up the experiment to the 535B-A23B MoE model.
Percy Liang's post on X has exceeded 800,000 views, and the news was quickly reposted by Andrew Ng.
Andrew Ng called Marin a "precious demonstration" of defending AI openness at present, emphasizing that the project not only opens up model code, but also makes public data, training recipes and experimental processes.
He continued to write: "Publicly publishing AI research results was once the industry norm; I am grateful to @percyliang for adhering to the open lab philosophy."
However, Marin is not yet a completed new model ready for comparison.
The real reason why this experiment has attracted attention is not the 535 billion parameters itself, but that it turns the training process, which is usually strictly protected by model companies, into a public research object that can be observed and reviewed in real time.
535B does not mean that 535 billion parameters have to run every time
Judging from the naming, Marin 535B-A23B is a Mixture-of-Experts model, that is, an MoE model.
"535B" means the model has a total of about 535 billion parameters, and "A23B" means that for each Token processed, about 23 billion parameters are actually involved in the calculation. It does not make all 535 billion parameters work at the same time. Instead, the routing module first judges the input content, and then assigns Tokens to part of the expert networks.
This is also an important reason why MoE has re-emerged as the mainstream technical route for large models in recent years: the model can continue to expand its total capacity, but the computational cost of a single Token does not need to grow synchronously with the total parameter count.
However, "23 billion activated parameters" cannot be simply equated with a 23B dense model.
In addition to the experts selected by routing, MoE also includes structures such as attention layers, embedding layers, shared experts and routing modules that are always or partially involved in calculation. Different teams may have different statistical calibers for "activated parameters". Therefore, when comparing two MoE models, it is not enough to only look at "A23B", but also to compare the number of training Tokens, the number of experts, the routing method, the proportion of shared experts and the actual FLOPs.
The public technical description of Marin shows that this model adopts a design where shared experts and routed experts coexist: each layer retains 2 shared experts, and activates 8 routed experts at the same time; both types of experts adopt a half-width structure. Since the routed experts also use 2x compression, the team equivalently describes it as: shared experts provide neurons with about one hidden layer width, and routed experts provide neurons with about two hidden layer widths.
In other words, about one-third of the expert computation comes from the shared experts that are always working. This design is not to make the model parameter number look better, but to reduce the risk brought by "Token Dropping" in MoE training.
Training MoE, the difficulty is not just splitting the model onto more GPUs
The advantage of MoE models is sparse computation, while the engineering cost is complex communication.
When a training batch enters the model, different Tokens may be routed to different experts, and these experts are often scattered across different GPUs or even different racks. The system first needs to perform an All-to-All communication to send Tokens to the corresponding experts; after the experts complete the calculation, the results need to be sent back to the original calculation path again.
Therefore, the bottleneck of MoE training is not necessarily GPU computing power, but also cross-card communication, uneven expert load and memory access.
If a few experts receive too many Tokens while other experts are relatively idle, the system will have "hotspot experts". To avoid individual GPU overflow, the training framework usually sets a capacity upper limit for experts; Tokens exceeding the capacity may be directly discarded, which is called Token Dropping.
An excessively high Token Dropping ratio means that some Tokens do not completely pass through the selected expert network, which may weaken the training effect. Increasing expert capacity can reduce dropping, but it will bring more video memory overhead, computational redundancy and communication waiting. This is a typical trade-off between system and model performance.
The Marin team disclosed that in previous experiments, when the context length was extended from 4K to 65K, the Token Dropping ratio once rose from about 7% to about 40%.
One of the reasons is: when the total Token batch size is relatively fixed, the longer the context, the fewer independent sequences contained in a batch, the more unbalanced the Token distribution is, and the more difficult it is to achieve load balancing between experts.
Therefore, Marin 535B does not pursue ultra-long context from the very beginning, but rolls back to 4K context to start pre-training.
Compared with 8K, a Token batch of the same size can accommodate about twice as many independent sequences, which helps different experts to get more even input.
The new pooled/wave expert parallel scheme tested by the team reduced Token Dropping to about 3% under 4K context. But Marin also clearly acknowledges that this implementation is still experimental in nature, and after extending to 65K, the dropping ratio may become too high again.
This is also one of the technical issues worth observing in this public training: Marin is not only training a large model, but also testing whether a large-scale MoE communication system can run continuously and stably for about 100 days.
A set of expert parallel implementations is handwritten exclusively for JAX
Marin's training stack is mainly built on JAX, XLA and Levanter.
JAX is responsible for numerical calculation and automatic differentiation, XLA is responsible for compiling the computation graph to the underlying accelerator, and Levanter is the large model training framework used by Marin. Previously, Marin 8B and 32B mainly ran on Google TPU; this time the 535B model shifts to NVIDIA GB200 NVL72, which means the team needs to re-address the expert parallelism and communication efficiency issues on GPU clusters.
Marin stated in public records that since no off-the-shelf expert parallel scheme with sufficient performance in the JAX/XLA GPU environment was found, the team implemented EP, that is, Expert Parallelism, on its own.
Expert parallelism places different experts on different devices. If each GPU only stores part of the experts, the total model parameters can span the entire cluster, but each Token must move between devices according to the routing result. Its performance is highly dependent on whether All-to-All communication can be effectively overlapped with expert computation.
GB200 NVL72 organizes 72 Blackwell GPUs and 36 Grace CPUs in one rack-level NVLink domain, which is suitable for running MoE models that require a large amount of cross-card communication.
Judging from public materials, Marin does not regard the hardware configuration as a guarantee that training will necessarily succeed.
The project has specially reserved response plans for hardware failures, model computing utilization decline and schedule delays: if the problem occurs within the first 25% of the Token budget, the team may shorten the final Token training volume and readjust the linear learning rate decay and data ratio, instead of rigidly maintaining the original plan.
Build a "Scaling Ladder" before the 535B officially starts running
Marin did not put all its computing power into the 535B model directly, but first trained a set of small MoE models with increasing scales to form a four-level Scaling Ladder.
According to the information disclosed by Percy Liang, these models cover:
- 1.6B total parameters, 61M activated parameters;
- Intermediate models that expand step by step;
- A maximum of 27.7B total parameters, with about 1.2B activated parameters.
The purpose of the Scaling Ladder is not to prove the capability of the 535B model in advance, but to check at low cost whether the training recipe can scale stably with the model size, and fit the relationship between loss, computation volume and model size.
Marin says this set of scaling experiments only accounts for about 1% of the final computation volume, but can undertake several key tasks.
First, it can predict the loss level that the 535B model should reach at different training stages. If the main training curve deviates significantly from the predicted value, the team can judge early whether there is a problem with the data, routing or optimizer, instead of discovering training failure after trillions of Tokens.
Second, it can expose training stability issues in advance. Marin previously found through scaling experiments that as the Token training cycle extended, the gradient norm once rose to above 4. Subsequent ablation experiments showed that under conditions such as high Batch Size, if this problem was not handled, the training might diverge completely midway.