OpenAI treats crash debugging as epidemiological research and has fixed the 18-year-old vulnerability that existed in GNU libunwind.

OpenAI's engineers spent weeks trying to explain the mysterious crashes in Rockset. Rockset is a C++ data infrastructure service that powers ChatGPT's search and data plugins. Functions seemed to return incorrect memory addresses, and the stack pointer appeared to shift by 8 bytes during execution. Every hypothesis the team proposed was met with strong counter-evidence. This bug seemed impossible to exist at all.

What they thought was a single bug turned out to be two unrelated bugs that just happened to be discovered at the same time by coincidence. This breakthrough did not come from deep investigation of a single crash event, but from shifting to what they called "epidemiological debugging": building a pipeline to automatically analyze every core dump file in the production environment over the past year, and then looking for overall patterns instead of inferring from individual cases.

The team had ChatGPT write a script to download the beginning of each core file, extract register data, filter known false positives, and label each crash as "return null pointer", "stack alignment error" or other types. They applied this script in parallel to all Rockset core dump files from the past year. They quickly spotted correlations. Problems that originally appeared to be the same type based on symptoms actually corresponded to two sets of crash events with completely different characteristics.

All these crashes caused by stack alignment errors originated from the same Azure region, had a clear start date, and never occurred on long-running nodes. The team traced that these crashes came from a physical host whose CPU was silently producing incorrect results. It was neither overheating nor throwing machine check exceptions, but simply making silent errors in mathematical operations. After removing this host from service, the crashes caused by stack alignment errors completely disappeared.

After ruling out the hardware crash issue, the remaining crashes caused by "return null pointer" became manageable. Previously, the team had ruled out C++ exception unwinding as the cause because they thought they had found counterexamples: crashes occurring in code paths that did not use exceptions. But all those counterexamples came from the faulty cluster with hardware damage. Once these interfering factors were eliminated, all the remaining crashes occurred during exception unwinding.

The root cause of the problem was an 18-year-old race condition in the _Ux86_64_setcontext function of GNU libunwind. During C++ exception unwinding, libunwind synthesizes a ucontext_t structure on the stack, fills in the required register state, and then calls _Ux86_64_setcontext to transfer control to the cleanup handler. The problem is: _Ux86_64_setcontext updates the stack pointer (%rsp) to point to the new stack frame before it finishes reading the instruction pointer from the old structure. Once %rsp changes, the structure is no longer part of the active stack and is no longer protected by the kernel red zone. If a signal arrives exactly in this time window between the update of %rsp and the read of %rip, the kernel will build its signal frame on top of this structure, the instruction pointer will be corrupted, and the function will jump to NULL or a garbage address.

The width of the race window is exactly one instruction. Calculated at the clock frequency of modern processors, this is about 100 picoseconds. In most programs, this scenario will never be triggered. OpenAI's Rockset uses the timer_create function to send a SIGUSR2 signal every few milliseconds of CPU time, in order to implement lightweight per-query accounting, which generates far more signal events than traditional applications. It is this high-frequency signal transmission that turned a theoretically possible race condition into a real production environment crash.

The team submitted a fix and a self-contained reproduction example to GNU libunwind, and verified through testing that other unwinders (such as libgcc) do not have this problem. This fix completely eliminates this time window by reordering the instructions to ensure that %rip is read before updating %rsp.

The team's summary of this lesson is worth quoting in full:

The most important step was not cleverly interpreting assembly code, nor was it deep understanding of the details, but building a high-quality dataset. Without this dataset, we would have confused two completely different phenomena and tried to reason our way out of that confusion. Once we had accurate and complete full-volume data, the structure of the problem became obvious.

If your team is troubleshooting unexplained production environment crashes, check to see if you are conflating multiple bugs. Symptoms that seem inconsistent with every hypothesis may not actually be contradictory; they may be consistent with two different hypotheses that you have inadvertently mixed up. The fastest way to gain insight into the structure of the problem is not to dig deeper into individual cases, but to obtain complete, labeled data covering all failure cases.

This full engineering blog post includes detailed stack memory diagrams, the vulnerable assembly instructions, and a crash rate visualization chart that reveals the two distinct types of failures.

Original link:https://www.infoq.com/news/2026/07/openai-libunwind-core-dumps/

This article is from the WeChat official account "AI Front", author: Steef-Jan Wiggers; Translator: Pingchuan, published with authorization from 36Kr.