TikTok SRE Technical Lead: At the end of the day, AI Agents are essentially distributed systems.

When your AI Agent calls the refund interface and encounters a network timeout, has the refund actually succeeded or not?

Salman Munaf, Site Reliability Engineer at TikTok, uses this example to illustrate his point: A timeout never means failure, it means "unknown". When an Agent encounters any failure, its first reaction is often to retry. Without request identifiers, idempotency keys, and status lookup, this instinct of "retry first and think later" may cause the same refund to be executed twice.

Salman works in Site Reliability Engineering at TikTok. He believes that once the model starts calling external services, the problem is no longer just a model problem, but a distributed system problem. This means it will encounter various failure modes that have been repeatedly studied and named in this field for decades.

Recently, he systematically broke down in a speech why AI Agents have evolved from simple LLM calls to real distributed systems, and what "weapons" developers must bring from the distributed system toolbox to prevent Agents from causing irreversible consequences when making mistakes. Based on the speech video, InfoQ has organized the content.

Core viewpoints are as follows:

AI Agents have evolved from models that only output text to external system coordinators that perform side effects, and this transformation has completely changed the system boundaries and risk surface.

When context can influence actions, it is state. And state will become stale, state will conflict with authoritative data, and state will contaminate the Agent's subsequent actions.

We should treat memory as cache: it should be invalidable, and it should carry source information.

A harmless model can become dangerous when it is capable of performing unsafe operations.

Logs are no longer sufficient. You must record what the Agent sees at each step, what it does based on that, and why it thinks that is correct.

From Chatbots to Production Systems

Today I want to talk about why AI Agents are also distributed systems.

In fact, LLM models were very simple at the beginning: text in, text out, no actions performed, and the only impact it could have was a wrong output. At that time, the system was closed, and if the model made a mistake, the error only stayed in the response and would not spread outward. But with the rise of Agent capabilities, current Agents can interact with external systems and have become distributed systems. Therefore, it is very important to integrate distributed system thinking and concepts when building AI Agents.

You may have heard of accidents caused by AI Agents. Replicate's AI Agent deleted the production database, and Air Canada's chatbot made a wrong refund promise. Behind these accidents is the lack of systematic thinking.

For example, in the Replicate case, if we had robust backups and scoped permissions that would never allow an AI Agent to delete the production database, this incident would never have happened. The same goes for the Air Canada case: if it had an authoritative source of truth to ensure it would not make decisions based on outdated or wrong policies, that wrong refund promise would never have occurred.

Initially, when LLMs were just chatbots, we entered prompts and got text output, no side effects, and the Agent did not interact with any other systems. However, in the AI Agent era, these Agents can now run Agent Loops, call external services and tools, and perform state changes by receiving prompts. The architectural boundary has far exceeded the LLM model itself, and can produce side effects on the external world. So when building an AI Agent, it is important to recognize which external systems it is communicating with, which states it is interacting with, what credentials it has, and which operations it can perform.

I want to emphasize a very key comparison. In distributed systems, we used to have services coordinating multi-step workflows, but they were deterministic: what to do at each step and how to handle errors were predefined. But AI Agents are different: they are essentially probabilistic coordinators. The types of actions they can perform and what they will do next vary so widely that a traditional decision tree cannot cover all cases. If these operations are not constrained by deterministic controls, they may have serious consequences. Therefore, it is very important to ensure that we have deterministic control measures in place to ensure that AI Agents do not perform any potentially problematic operations.

Every Step of the Agent Loop Crosses Boundaries

What does a typical Agent loop look like? First it plans, then executes an action based on the plan, then observes the result of this action, possibly persists the result to some data store, and then decides what to do next. Every step in this loop crosses system boundaries.

In the planning phase, it interacts with data sources to retrieve information. In the action phase, it calls external APIs, tools, databases, and performs real operations. In the observation phase, it gets partial results, and then decides subsequent actions based on these partial results. It can persist incorrect data, choose to perform an incorrect action when making decisions, or worse, trigger a "retry storm".

Therefore, when building an Agent loop, one principle is particularly important: every step must be persisted. Every action the Agent takes and every piece of context it retrieves must be recorded. Why? Because if a step fails, the Agent needs to know at which position it failed, so that it can perform reversible operations and rollbacks. At the same time, every step must clearly define the transaction boundary. For example, if the Agent makes a call that fails, what should it do? For irreversible or unsafe operations, what is the compensation? If the Agent sends an email to the wrong customer, how can it make up for it? These questions must be answered in the design phase, not after an accident occurs.

Retries Are Not a Virtue

Tool calls essentially wrap external APIs, databases, queues and so on. When you make a remote call, you have to face the failure modes that distributed systems have encountered long ago: network latency, timeouts, duplicate requests, and the most subtle one — the server may have succeeded, but the client receives an error report.

We have seen many such scenarios: the data has been written to the database, but due to some other error, the server returns an error to the client. When a human is involved, we can check the real state in the database and take the correct corrective action. But the Agent does not have this intuition: it sees an error and retries, without knowing that it knows nothing about the real state of the system.

Take a very specific example. The Agent calls a "customer refund" tool, the tool executes the refund operation, and then the request times out. Did the refund actually happen? How will the Agent infer it? Will it issue a second refund? The key insight here is: timeout does not mean failure, timeout means unknown.

When designing these tools, there are several things that must be done. First, there must be request IDs and idempotency keys. When duplicate requests come in, the downstream system can recognize that they are the same request and will not produce duplicate side effects. Second, the system must support status lookup to query the status of the previous request and figure out whether it succeeded or failed.

The AI Agent's first reaction to failure is to retry, which is its default behavior, so you must hardcode idempotency at the tool layer. If the same request comes in again, the tool must recognize it as a duplicate request and ensure that no side effects occur.

But idempotency only solves the problem of "duplicates do not produce side effects", it does not solve the "retry storm" problem. If the Agent keeps retrying nonstop, it will create tsunami-level call pressure on your external APIs, and this pressure will propagate downstream, leading to cascading failures. So we need to set the maximum number of rounds, budgets, spending limits, and maximum number of parallel calls to ensure that its fan-out is not too large. Exponential backoff should also be implemented to give downstream dependencies breathing room. For operations with side effects, compensation operations must be predefined.

Your Agent Has "Memory" That Goes Stale

Many teams building AI Agents treat the context held by the Agent as nothing more than "context". But I want to make it clear: when context can influence actions, it is state. And state will become stale, state will conflict with authoritative data, and state will contaminate the Agent's subsequent actions.

I divide the Agent's memory into two categories. Short-term memory is the context in this execution thread. Long-term memory includes project files, system prompts, the databases it interacts with, the cache layer, and so on. When information conflicts between these different data sources, you must decide what the source of truth is. Moreover, we should treat memory as cache: it should be invalidable, and it should carry source information. For example, when the database is updated and the source of truth is updated, the old context held by the Agent should be invalidated to ensure that it does not make decisions based on stale data.

Agents usually perform multi-step actions: it is normal for the first few steps to succeed and then fail at a certain step. But the question is, what to do with those previous steps that have already been executed? You need to roll back the entire transaction, and these steps may cross system boundaries.

For example, the Agent can update an internal work order, send an email to the customer, and then fail when updating the CRM. At this point, you need to define what the correct compensation operation is. Another example, as I mentioned earlier, if the Agent sends a wrong email to the customer, the compensation operation is to send an apology email or a correction email.

The Agent runs in a loop, and it will enter a retry loop when it fails. So when it calls external dependencies, a circuit breaker is mandatory. If the downstream system is unhealthy, the circuit breaker should stop the Agent from continuing to call that dependency. This not only protects downstream services, but also prevents cascading failures.

There is another dimension that many teams ignore: budget. If an Agent has no budget and rate limits, it will keep running, keep retrying, and keep spending money. Therefore, the maximum number of rounds, maximum parallelism, and maximum overhead must be set to ensure that the model does not exceed the budget boundaries we set.

A Smart Model Does Not Equal a Safe System

There is another very common anti-pattern. When building an Agent, we always tend to grant all possible permissions, thinking that this will allow it to complete the task. Interacting with a database? Give it read access to the entire table directly. However, it is very important to provide it with scoped credentials. There should be separate read and write permissions, and an allowlist to restrict the tools it can call. A harmless model can become dangerous when it is capable of performing unsafe operations.

The same goes for human approval mechanisms. Approval cannot be a blanket "agree" button. Approval must be bound to specific actions, timestamps, executors, and expiration times. If a user approves a $30 refund, this approval cannot become authorization to approve a $300 refund next. The approval must be tightly bound to the specific parameters of the original request at that time.

Observability is a rigid requirement for building Agents, but many people still only understand it at the level of "adding logs". I want to make it clear: logs are no longer sufficient. When an Agent fails, the team needs to reconstruct what happened at that time, what information it reacted based on, and why it failed. Logs alone cannot achieve all this.

You need to track: which model was called, what prompts were given, which tool calls were executed, what requests were sent, what responses the tools returned, what errors were encountered, what context was retrieved, what decisions it made based on this context, what write operations were performed, and what approvals were obtained.

Finally, I want to say that yes, model capabilities are very important. The better the model, the higher the probability of performing correct operations, and a smarter model can reduce errors. But model capabilities cannot eliminate network failures, cannot eliminate stale data, and cannot eliminate adversarial inputs.

When building this architecture, we need to ask ourselves: Can I constrain the Agent's behavior? Can I observe its behavior? Can I recover from its mistakes? The tool contract must clearly define the types and schemas of requests and responses, and hardcode idempotency into it. This way, when duplicate requests are sent in, unsafe operations will not be retried. In addition, when there are conflicting memory states, decisions should be made based on the real data source. Retry policies must have rate limits to ensure that the Agent does not retry aggressively. Permissions should be configured, and there should be tracking and recovery paths.

When building an AI Agent, the most important question to ask is: When it makes a mistake, to what extent does the system allow it to go?

Original speech video link:

https://www.youtube.com/watch?v=hD9-V56FNRI

This article is from the WeChat Official Account "InfoQ" (ID: infoqchina), written by Tina, and authorized for release by 36Kr.