The Anthropic team started with zero Rust experience, completed the cross-language refactoring of the core API with the help of Claude, achieving a 40% reduction in CPU consumption and a 20% drop in latency.

Anthropic has recently completed a representative engineering transformation: a team that initially had no Rust developers migrated the company's Messages API from Python to Rust with the help of Claude. According to Katelyn Lesse, Head of Platform Engineering at Anthropic, after the transformation, the CPU resource consumption of related services was reduced by approximately 40% to 50%, and the time-to-first-token latency was also improved by around 10% to 20%.

In this project, Claude took on a large volume of code writing work, while human engineers were mainly responsible for system architecture, migration strategies and verification solutions. A complex project that originally required massive manual coding can now be completed with the aid of AI. Then, when AI is already capable of undertaking such complex development tasks, what else can software engineers do? And how can they demonstrate their own value?

Katelyn is quite optimistic about this. On October 26, Katelyn published a long post on X titled "The new set of problems for engineers to solve". In the post, she mentioned that this is a fantastic era for software engineers. While AI is changing the way engineers work, it also brings a batch of engineering problems that have never been faced in the past.

This optimism is not unfounded. When Agents no longer only answer questions and perform simple tasks, but start to run autonomously for a long time, collaborate with each other, and even participate in complex scientific research and engineering projects, the existing software infrastructure, identity authentication and authorization, security isolation and monitoring mechanisms are all facing new challenges.

In Katelyn's view, these challenges also mean new engineering demands, which are exactly the directions worthy of software engineers' efforts in the next stage. For engineers who are thinking about their own career value, this may provide another perspective to look at the future.

1 Tens of thousands of Agents running at the same time, can Kubernetes still handle the load?

An Agent in charge of overall work coordination can assign tasks to multiple sub-Agents; different Agents can be responsible for execution, inspection, scoring and other work respectively, and then aggregate the results. As tasks become more complex, the number of Agents participating in collaboration may continue to increase, and some tasks even need to run continuously for several hours or days. When tens of thousands of Agents run simultaneously, can the existing infrastructure efficiently manage these workloads?

Katelyn mentioned Kubernetes. As a widely used container orchestration platform, Kubernetes can manage application deployment, scaling and fault recovery. It can run both stateless applications and stateful workloads, but the operating characteristics of Agents put forward new requirements for existing management methods.

Katelyn describes Agent as a type of workload that requires isolation, has independent state, and usually runs in a single instance. Compared with traditional online services, the operating mode of Agent has obvious differences. It may continuously perform complex tasks, or wait for model inference, tool calls or results returned by other Agents for a long time. If each Agent always occupies an independent operating environment, the resource utilization rate may be very low; but if resources are released when idle, problems such as state preservation, fast recovery and rescheduling need to be solved. How to strike a balance between startup speed, state persistence, isolation strength and resource cost has become a new challenge for Agent infrastructure.

Around these problems, new technical solutions are also emerging continuously. Katelyn mentioned that some AI infrastructure companies have begun to develop new file systems and sandbox services, and the Google team is also building a new runtime environment called Agent Substrate. However, in her opinion, there are still many problems for engineers to solve on how to provide an efficient and reliable operating environment for large-scale Agents.

2 When Agents start to do things on behalf of people, who should the permissions be delegated to?

Making Agents run stably is only the first step.

When they start to access code repositories, call external services, modify files, and even assign tasks to other Agents, another problem arises: how much permissions should these Agents have exactly?

Katelyn clearly distinguished two concepts in her article: Authentication and Authorization. The former answers "who you are", while the latter determines "what you can do". For the former, the software industry already has relatively mature technologies such as Workload Identity and short-term certificates for reference; the truly complex part is how to distribute and transfer permissions when Agents start to perform tasks on behalf of users for a long time, and even create sub-Agents autonomously.

At present, many Agents actually work on behalf of human users. They use the user's identity and credentials to access different systems and perform operations on behalf of the user. The problem is that when a file is modified or an operation is performed, how can the system distinguish whether it is done by the user himself/herself or completed by the Agent on behalf of the user?

When Agents start to run autonomously for a long time, the authorization problem will become more tricky. For example, a user authorizes an Agent to handle development tasks for several consecutive weeks, and the Agent delegates part of the work to sub-Agents. At this time, whether the sub-Agents can inherit the original permissions, how much to inherit, and how to determine the validity period of authorization all need to be redesigned. Especially when the Agent is going to perform operations beyond the user's original expectation, the system must also judge whether human reconfirmation is required.

Katelyn specifically mentioned the On-Behalf-Of (OBO) mode. The traditional OBO authorization mode allows services to access other resources on behalf of users, but the autonomy and task duration of Agents make permission delegation more complex. The system not only needs to confirm who the Agent is performing operations on behalf of, but also needs to clarify the authorization scope, delegation chain and the handling method after permissions are revoked.

The more complex situation is that if future Agents no longer rely on the identity of human users, but have their own independent identities, how should permissions be assigned to them? When multiple Agents need to collaborate with each other, and one Agent sends a request to another, how should the latter confirm the identity of the other party and judge whether this request is authorized? These problems still lack mature solutions at present, and are still new problems that engineers need to solve.

3 Security Isolation: How to limit the spread of risks after an Agent is attacked?

In the past few months, especially in recent weeks, software vulnerabilities have been exposed one after another, appearing even faster than the industry can fix them, many of which involve key fundamental software that a large number of systems rely on to run.

Katelyn pointed out that when tens of thousands of Agents run continuously, constantly write and execute untrusted code, and read content from untrusted sources, the unrepaired vulnerabilities in fundamental software may bring greater security risks. Therefore, how to harden the operating system kernel, browsers, runtime environments and common software libraries has become an important problem for software engineers to solve.

In addition, multi-agent systems will also introduce another type of risk.

Assume that an Agent reads a web page or file containing malicious instructions while performing a task, and is attacked by prompt injection. If it only runs independently, the risk may be limited to the current task; but if it needs to collaborate with other Agents, the malicious content may be transmitted to other Agents through communication, and even enter the shared memory, affecting subsequent tasks.

At this time, the Blast Radius of the attack becomes a problem that must be carefully considered: if the first Agent is compromised, how many parts of the entire multi-Agent system may be implicated? In her opinion, a strongly isolated execution environment is an important foundation.

In fact, as early as September 13, Katelyn wrote a special article to discuss the security architecture of Agents, proposing to separate the "brain" of the Agent from its "hands".

The so-called "brain" includes the Agent Harness responsible for model calls, tool calls and user interaction, as well as the session state and access credentials that need to be persisted. These components should run in trusted infrastructure outside the sandbox. The sandbox as the "hands" is responsible for executing the code and commands generated by the Agent, and processing external data that may contain malicious content.

The key to this architecture is to make untrusted code always run in an environment with limited permissions. When the Agent needs to access external services, the trusted proxy outside the sandbox completes credential injection and access control, preventing sensitive keys from being directly exposed to the code inside the sandbox. Even if the Agent encounters prompt injection or executes malicious code, the resources that attackers can access are restricted.

4 How to identify whether an Agent is trying to complete the task or has gone out of control?

Even with isolation measures, engineers still face another problem: how to detect that an Agent has become abnormal in time?

Assume that an Agent keeps trying to access a domain name that is not in the allowlist in order to complete a task. Is it trying to find a way to complete the task, or has it been affected by malicious instructions and is trying to break through the restrictions?

This is a scenario proposed by Katelyn in her long article. The two behaviors may look very similar from the outside, and in a system where multiple Agents collaborate with each other, it is more difficult to judge abnormal behaviors. Engineers not only need to know what each Agent has done, but also need to understand how tasks are transferred between different Agents to troubleshoot problems.

To this end, Katelyn mentioned cross-Agent distributed tracing based on OpenTelemetry (OTel), so that engineers can track the execution process of the entire task. At the same time, the trace records must be stored in a trusted system that the Agent has no right to tamper with, preventing the Agent from modifying its own or other Agents' operation records and interfering with subsequent investigations.

In addition to detecting anomalies, the system must also have the ability to stop risks in time. Katelyn specifically emphasized the Kill Switch mechanism: once dangerous behavior of the Agent is detected, engineers should be able to stop the relevant Agents from the outside, terminate the entire system if necessary, and revoke access credentials to prevent the risk from continuing to expand.

For Agents that can run autonomously for a long time and collaborate with each other, how to track their behaviors and stop them in time when necessary is also a problem that software engineers need to solve.

5 After code is handed over to Claude, what are engineers doing?

Back to the beginning of the article, Anthropic's Messages API refactoring project provides a specific case for observing how AI programming changes software engineering practices.

In the Latent Space podcast released on October 27, Katelyn introduced the specific process of this migration. As the scale of API usage expanded, Anthropic's original Python monolithic service faced pressure on CPU resource efficiency, so the team decided to re-implement the Messages API using Rust.

However, when the project started, the team had no Rust developers at all. According to the traditional development method, such cross-language refactoring usually requires supplementing relevant talents first, and then completing the migration step by step. Anthropic chose to use Claude to undertake a large amount of code implementation work.

However, this is not simply letting Claude translate Python code into Rust. Katelyn introduced that the team needs to first determine the system architecture, formulate a migration plan, and then use Claude to complete the specific implementation. More importantly than letting Claude write Rust code is how to prove that the new system can safely replace the old one. The team also used methods such as Shadow Traffic to let the old and new implementations process the same or equivalent requests, compare the output results, and then gradually verify and switch traffic.

Anthropic's practice demonstrates an emerging engineering division of labor: AI undertakes more and more code implementation work, while human engineers devote more energy to system architecture, verification, reliability and security control. But this division of labor also means that the capabilities that engineers need to possess are changing.

The four types of problems proposed by Katelyn point exactly to this change. Agents are getting better and better at performing tasks, but to make tens of thousands of Agents run stably and collaborate safely in the real production environment, and always stay within the human controllable range, there are still a large number of fundamental engineering problems that have not been solved.

In the Latent Space interview, Katelyn also shared Anthropic's specific experience in API refactoring, Agent architecture design, security isolation and engineering evaluation. These practices also allow us to see more intuitively how the work focus of engineers has changed when AI begins to undertake a large amount of coding work.

The following six groups of Q&A most relevant to the above topics are selected, and InfoQ has edited and organized them without changing the original meaning.

6 Selected Interview Content: How Does Anthropic Make Claude Truly Enter the Production Environment?

Q1: Your team originally had no Rust engineers. Why did you decide to use Claude to rewrite the entire Messages API? How was it completed specifically?

Katelyn Lesse: In the past, if the budget allowed, you could basically expand CPU resources on demand. But the situation is different now, CPU capacity also needs to be planned in advance, which is a new challenge for the whole industry.

We are solving this problem from several aspects. First of all, of course, to obtain as many CPU resources as possible. The second is to improve the operating efficiency at the infrastructure layer, for example, to make the sandbox run more efficiently without sacrificing security requirements such as tenant isolation. The third direction is the application layer, thinking about how to make our applications and APIs themselves run more efficiently.

Last week, we just completed the full launch of the Rust version of the Messages API. This is a very bold engineering project that can only be completed in the era of Agent programming. Earlier this year, we realized that CPU resources might become a bottleneck. Before that, we had completed multi-region deployment and other work, but what was running at that time was still a set of Python monolithic services, which had very low CPU efficiency, so we began to think about how to make the whole system run more efficiently.

Eventually, we chose Rust. From a technical point of view, this decision is easy to understand, but it was quite bold at that time, because there was not even a single Rust developer in the team. However, we decided to directly try to let Claude be responsible for writing code, and human engineers be responsible for designing the system, after all, it is essentially still a set of distributed system architecture.

Claude can not only write code, but also help engineers discuss design plans and learn new knowledge they need to master. But the two most important things in the whole project are actually formulating the implementation plan and verifying the final result. The specific implementation work is mainly completed by Claude.

We finally directly built a complete Messages API with Rust, then started to import shadow traffic to the new system, verify whether the outputs of the Python and Rust systems are consistent, and then gradually switch the real traffic over.

If it were in the past, I would never let a team adopt this migration method. Such a project usually takes several years, you will split the whole system into many parts, migrate part by part, instead of deciding to rewrite the whole system at once and then launch it gradually. But this time we really did it, and it succeeded.

Host: Do you have Rust engineers now?

Katelyn Lesse: Now the team members probably all consider themselves Rust engineers. They have gradually mastered Rust. In this process, we have also recruited several engineers with more Rust experience.

Host: How much have you saved in terms of CPU resources?

Katelyn Lesse: About 40% to 50%, which is a pretty amazing number. The latency has also improved, specifically, the time-to-first-token latency has improved by about 10% to 20%. For some workloads that run for a longer time, the improvement will be even greater.

Host: When doing large-scale refactoring, people can't help but want to add some features that were not there before. How do you control this impulse? Is this a pure rewrite?

Katelyn Lesse: Yes, we deliberately restrained the urge to add new features. I prefer to call it a Port. We specially formed a small team to complete this matter, and at the same time, other teams continued to develop new features for the original API. The team responsible for migration must keep up with these changes. They use methods such as static analysis to ensure that when other engineers add features to the Python version, they also consider the corresponding implementation of the Rust version. This is also part of the complexity that the whole project needs to handle.

Q2: Why did Anthropic separate the "brain" and "hands" of Agents? How did this architecture evolve?

Host: Another problem related to computing resources is the sandbox. Now everyone is discussing the problem of Agents breaking through sandbox restrictions. How do you deal with it? What is the most advanced solution at present?

Katelyn Lesse: When we designed Claude Managed Agents, we adopted an architecture that separates the "brain" from the "hands". The so-called "brain" is the Agent Harness, which runs on our own persistent servers; the "hands" are the execution environment, such as sandboxes or containers.

The initial adoption of this architecture was largely out of consideration for latency and reliability. If the Harness runs on a persistent server, and the sandbox is only started when code needs to be executed, then after the user initiates a request, we can start returning tokens very quickly. At the same time, even if the sandbox terminates, the entire Agent will not terminate accordingly, because the Harness is still running, and the state of the Agent is stored elsewhere.

In fact, we have gone through a complete product architecture evolution process. At the very beginning, the way we built the Agent API was very straightforward: put Claude Code into a sandbox, and then wrap a layer of API on the outside. But this solution has obvious problems. Every time