DeepSeek published a new paper disclosing Agent training, with Liang Wenfeng as the named author.

Training large models relies on computing power, while Agent training relies on the environment.

How to build such an environment? The latest DeepSeek paper signed by Liang Wenfeng has made all technical details public.

The system developed by DeepSeek is called DSec (DeepSeek Elastic Compute), which is designed to mass-produce sandboxes for Agent training.

It can generate more than 5,000 sandboxes per second, reaching 3 million per day, with a peak of 380,000 running simultaneously.

The single cluster supporting this scale is also very large, with approximately 160 nodes, 30,000 CPU cores and 250TB of memory.

Why is training an Agent so difficult?

Because the training environment for large models is just a GPU cluster that feeds data and calculates gradients, but Agent training is completely different.

It has to write code in the sandbox, run compilation, open browsers, and even install operating systems... Every step of execution changes the state of the environment, which may crash the environment at any time.

Therefore, each training round must provide it with a brand new, clean sandbox that is discarded immediately after use and thrown away after training.

So after all twists and turns, the problem comes back to the infrastructure —

These infrastructures need to deploy a complete set of operating systems and toolchains for each sandbox at a speed of 5000 per second.

At the same time, hundreds of thousands of concurrent sandboxes must not exhaust the memory and CPU resources of the cluster.

The paper fully presents the complete picture of this entire engineering implementation.

Agent Training Requires "A Whole World"

The first core problem DSec needs to solve is that different types of Agent tasks have extremely different requirements for sandbox environments, and all these environments must be scheduled uniformly on the same platform.

An Agent that solves OJ problems only needs a stateless function call environment, which can output results after running, without even a persistent file system.

But an Agent for SWE-bench tasks requires a complete Linux user space, where it needs to install dependencies, modify code, run pytest, and may even need to add new packages to the environment halfway through the task.

For security attack and defense and computer-use scenarios, container-level isolation is not sufficient. Agents need to operate browsers or even desktops, and a vulnerable Agent may crash the host machine accidentally, so virtual machines must be used.

The most extreme scenario is training Agents that operate commercial software, which requires a complete Windows or macOS system with a graphical interface and drivers, almost no different from a real personal computer.

DSec has prepared four backends for these four scenarios respectively: FnCall handles stateless function calls, Container runs Docker containers, MicroVM uses Firecracker to run lightweight virtual machines, and Full VM uses QEMU to run complete operating systems.

The isolation strength and resource overhead of the four backends increase step by step, but the training framework sees a unified Python SDK called libdsec.

No matter the underlying layer is a container or a virtual machine, the same interface is adopted. The calling methods for all steps including creating sandboxes, executing commands and obtaining results are completely identical.

To make the four backends run on the same set of clusters, the scheduling layer of the platform must also keep up.

DSec splits the entire link into six layers.

This link starts from a creation request from the training framework, goes through IAM authentication first, enters the API Server, then the Placement Engine selects a target node from the cluster according to the remaining resources, and the Edge component on the node is responsible for actually starting the corresponding type of sandbox.

The network egress and package management images of the sandbox are uniformly proxied by Aether. Every command executed by the Agent and every line of output generated in the sandbox is relayed back to the training framework through an in-sandbox communication component called Chronus, so that the framework can know which step the Agent has reached and what feedback should be given.

Through resource overcommitment and high-density deployment, a single node can host 3200 containers or 800 MicroVMs at the same time.

How to Support 3 Million Sandboxes Per Day

However, the most severe challenge for DSec in terms of scale is not scheduling, but environment construction.

When each sandbox starts, it needs a complete set of operating system images and toolchains, which is equivalent to installing systems on 5000 "computers" per second.

The traditional Docker idea is to combine the base image, workspace and toolkits into a complete image.

This solution works well on a small scale, but DSec's container backend has cumulatively used 11266 base images and 102171 workspaces, and 67.8% of the sandboxes need to overlay at least one layer of workspace or toolkit on top of the base image.

Under such diversity, once a certain toolkit is updated, all combined images containing it need to be rebuilt, and the cost is O(m·N).

DSec's solution is to split the environment into three independent EROFS read-only image layers: base image, workspace and toolkit, each with independent versioning, and combine them on demand through overlayfs when the sandbox starts. Updating the toolkit only affects the toolkit layer, reducing the cost to O(m)+O(k).

After the image is created, how to deliver it to the nodes is also critical.

Intuitively, the image should be pulled to the local cache in advance, but the paper counts the real runtime data:

The Python container image is 6.0GB, and the Agent actually only reads 6.0% of the data;

The Java image is 12.1GB, and only 9.2% of it is accessed;

The C++ image is 4.9GB, and only 8.7% of it is accessed.

In other words, most of the image content is never touched by the Agent from beginning to end.

Therefore, DSec chooses on-demand loading. Its images are stored in EROFS format on 3FS (Fire-Flyer distributed file system), metadata is prefetched locally, and data blocks are only pulled from 3FS when the sandbox actually reads them.

The DeepSeek team's actual measurement shows that the burst deployment of 8192 containers can be completed in only 35 minutes with on-demand loading, while Docker cold pull takes more than 60 minutes.

In addition, the disk write volume of on-demand loading is also more than half less than that of Docker cold pull, dropping from about 1600GB to about 700GB.

After the environment is built, hundreds of thousands of sandboxes running at the same time will face resource contention.

In terms of memory, when MicroVM reads image data through the virtual block device, the same piece of data will be stored in the page cache of the host and the virtual machine respectively, leading to doubled memory demand.

DSec uses virtio-pmem combined with DAX to allow the virtual machine to skip its own page cache and directly map to the host's physical memory. Multiple virtual machines share the same mapping, cutting peak memory usage by 40.2%.

For writable disks where virtio-pmem is not applicable, DSec uses DAMON to periodically scan cold memory pages and actively return them to the host, and cooperates with virtio-balloon's free-page reporting to further reduce memory demand by 21.2%.

In terms of CPU, DSec divides sandboxes into two categories: latency-sensitive and best-effort. The latter is set to SCHED_IDLE priority, and Linux's core scheduling is enabled at the same time to prevent low-priority tasks from running on the sibling hyperthreads of the physical core where high-priority tasks are located.

After the two layers of policies are superimposed, the latency expansion of latency-sensitive tasks under 50% background load is reduced from 45.2% to 17.3%.

In addition, DSec needs to cooperate with the RL training framework to handle GPU preemption.

In the early architecture, the Agent's inference loop runs inside the GPU training Pod. When the GPU task is preempted, the Agent's execution progress is completely lost.

Starting from DeepSeek-V4.1, the Agent loop is separated and runs independently in the worker container of DSec, and is no longer bound to the lifecycle of the GPU Pod.

When the GPU is preempted, DSec suspends the relevant sandbox and saves its state. After the GPU is restored, the sandbox continues to execute, and the training framework no longer needs to implement breakpoint recovery logic by itself.

Even with so many optimizations, a single cluster may still not be able to withstand the peak load.

The largest production task mentioned in the paper applies for 32000 sandboxes at one time. Under such burst load, DSec will automatically trigger cloud bursting to offload overflow tasks to cloud virtual machines.

The DeepSeek team's actual measurement shows that when the cluster utilization exceeds 80%, 200 cloud VMs can absorb about 30% of the peak load.

Agents Have Learned to Cheat

For Agents, the sandbox is not only an operating environment, but also a security fence.

The paper discloses multiple cases of reward hacking methods discovered by Agents themselves during training, that is, Agents have found shortcuts that can get high scores without completing tasks through legitimate means.

The first target these Agents aimed at is Chronus, the component responsible for communication between the Agent and the outside world inside the sandbox.

Chronus will call the system's bash when performing certain operations. After discovering this, the Agent directly overwrites /bin/bash, injects its own commands into it, and tries to intercept the answers to the questions or other privileged information from subsequent shell sessions.

After this path is blocked by AppArmor, the Agent finds a more underlying breakthrough: the ioctl call of the XFS file system.

XFS_IOC_SWAPEXT, whose function is to exchange the underlying data block mappings of two files. The Agent uses it to "swap" the content of the protected file to the file descriptor it holds, bypassing the file-level access control.

The side effect of this attempt is more dangerous than cheating, because it damages the metadata of XFS, which may directly cause the file system to crash, affecting far more than this single sandbox.

Information leakage channels outside the sandbox have also been found by Agents.

Some Agents will scan the network ports and services reachable by the sandbox to find reference implementations; some directly cheat by pulling code from GitHub through the Go module proxy; some even install updated versions of software packages to get ready-made solutions.

These behaviors cannot be discovered by simply checking the final output, because the Agent really "solved the problem correctly", but through improper means.

In addition to cheating, there are more violent scenarios.

An Agent executes grep recursively in the sandbox, scanning from the root directory all the way to /proc/kpagecgroup, triggering a kernel bug that directly crashes the entire host's kernel.

Another Agent executes the attack command that should have been sent to the target virtual machine in its own container during a security attack and defense task, which also crashes the kernel.

Some other Agents found that Chronus asynchronously records the stdout output of commands, so they repeatedly call the yes command to continuously print characters, and dozens of gigabytes of logs fill up the storage space.

DSec's current defense measures are divided into two layers.

The