Microsoft and Apple have suffered successive service outages, and more large-scale service disruptions are coming our way.
The global wave of widespread outages that has upended the tech world has finally reached Apple.
In the early hours of November 18 Beijing time, due to an internal system update failure, core services including iCloud, Apple Pay, App Store, and Apple Music experienced successive large-scale disruptions.
Over the past week, similar "failure" incidents have been too numerous to count:
In the early hours of November 13, Microsoft services including Outlook, Microsoft 365, Teams, and Copilot went down one after another. The fault lasted for 23 hours and 50 minutes, affecting 74 countries and regions;
On November 13, massive requests sent by a faulty router overloaded the communication network of the U.S. Federal Aviation Administration (FAA), grounding all flights across the U.S. for nearly 5 hours;
On November 16, an internal software bug at JPMorgan Chase slowed down transaction processing at multiple banks
……
Tech giants have been crashing one after another in a queue, but in this wave of dense outages, the one that made the biggest and most noticeable impact is still Microsoft.
As infrastructure that supports countless government and enterprise departments, the ultra-long outage of Microsoft's services a few days ago forced many organizations to return to the old-school office era, struggling to get by with phones, Bluetooth, and paper documents; of course, many office workers, full of passion for work, made themselves coffee and browsed their phones, enjoying the hard-earned legitimate slack time at work.
Third-party agencies estimate that the productivity loss caused by the shutdown reached as high as 70 to 16 billion U.S. dollars.
It seems unimaginable that a service used by hundreds of millions of people could be down for 24 hours, but when it happens to Microsoft... it somehow feels a little plausible.
Two days ago, Microsoft officially released the "Preliminary Incident Report on Microsoft 365". This report provides us with a perfect opportunity to peek behind the scenes of the successive outages of these modern tech giants.
The Interlocking, Bizarre Perfect Storm
It is no easy task to collect all the "dragon balls", but according to this preliminary investigation report, on the eve of the incident, Microsoft did gather three rare risk factors —
First, a hidden bug deployed more than three months ago.
Specifically, this incident occurred on the network gateway servers of Microsoft 365. You can think of it as the information desk in a hospital, responsible for establishing connections with users, forwarding requests, and managing traffic. It is the only way for most of Microsoft's cloud service traffic to pass through.
Since the gateway server may face a large number of requests at the same time, in order to prevent multiple threads from competing for access to the same internal resource, it has set up a coordination access mechanism called "Lock", which is like a registration desk, ensuring that different threads access resources in order of priority.
Everything seemed to be going very smoothly until Microsoft installed the patch.
Starting from July this year, Microsoft began to push a patch to fix an existing issue to gateway servers around the world. In the more than three months of operation that followed, no one noticed that this patch actually hid a serious malicious bug —
For undisclosed technical reasons, it would trigger "lock contention", meaning different threads would pile up while fighting for the "registration slots", eventually causing system resources to be exhausted and the gateway to gradually lose responsiveness.
As for why the problem was not discovered in pre-launch tests, Microsoft's explanation is that the tests used servers with weaker performance and fewer cores, so requests were easily cleared; but on the high-performance servers in the production environment, the larger "registration desk" amplified the intensity of contention, making it impossible to clear requests in time when problems occurred.
This time bomb buried in the system was just waiting for an opportunity to detonate.
The second factor is the switch between the primary and secondary gateways.
For system reliability, Microsoft 365 actually has two independent network gateway systems with different technical details. If the primary gateway fails, it can switch to the backup gateway.
But 15 hours before the incident, due to "daily maintenance", Microsoft switched the Teams traffic that was originally running on the backup gateway to the primary gateway that was already carrying traffic for services such as Outlook and Office, causing the pressure on the primary gateway to increase sharply.
To make matters worse, the primary gateway itself was already in a "weakened" state.
Coincidentally, the day before, another failure occurred on the primary gateway, causing 14% of the server capacity to go offline temporarily.
Theoretically, the remaining capacity is still sufficient to maintain operation; but with the access of Teams traffic, the primary gateway suddenly entered a high-load state, and the high load triggered the previously latent lock contention bug, which eventually caused the server load to further increase and crash.
What is more fatal is that the gateway server also has an automatic scheduling system. Theoretically, when one server fails, the scheduling system will transfer the traffic it originally carried to other servers.
However, since all servers have a similar bug that causes outage under high load, automatic scheduling eventually turned into a "contagion", where servers fell like dominoes and went down one by one.
At its worst, more than 80% of the gateway servers lost responsiveness.
Let me translate that for you: what do you call a situation where your house is on fire, and the fire hydrant spits out gasoline instead of water?
In other words, Microsoft already had redundant disaster recovery mechanisms. If any single link had not failed, a major outage would not have occurred; but the rare and bizarre part of the entire incident is that every link went wrong at just the right time, eventually leading to this "perfect storm".
Or, as the Heinrich Law puts it, behind one major accident there are 300 hidden safety hazards. Perhaps in places unknown to the public, the system has narrowly escaped danger many times, and this time luck simply stopped favoring it.
Considering the complexity of the incident, Microsoft engineers' post-incident handling is actually quite commendable —
The faulty patch had been online for too long to roll back, so they wrote a new patch on site; the automatic scheduling system would keep crashing the newly restored servers, so they decisively suspended scheduling and manually pulled up the entire server cluster step by step.
Before the 24-hour deadline, various services were gradually restored and brought back online.
However, during this period, hospitals suspending outpatient services, courts adjourning hearings, and exchanges and insurance companies being forced to fall back on phone contact were all established facts.
Gergely Orosz, former engineer at Microsoft and Uber, and founder of the well-known tech newsletter The Pragmatic Engineer, stated bluntly, This is likely the most destructive business email outage event in history.
That's why Microsoft CEO Satya Nadella had to publicly apologize:
Last week's incident reminds us once again that digital infrastructure is critical to the functioning of the world. We know you place your trust in our services, and we take this responsibility extremely seriously. In this context, I know we still have a lot of work to do to deliver the resilience that you and the entire world can count on.
Nadella's promises can be observed and listened to, but his words also reveal a basic fact: the more successful Microsoft is, and the more deeply its services are embedded in the infrastructure of the modern world, the more severe the shocks caused by outages will be.
In today's world where a few giants highly control communications and networks, this Sword of Damocles will probably hang over our heads for a long time to come.
AI Is Not the Culprit, But Systemic Risks Cannot Be Underestimated
Humans have an instinct to seek causal connections. Faced with this sudden major outage, it is hard not to associate it with the frequent AI "jailbreak" cyberattacks that have occurred this year, and the previous news that Microsoft opened a "voluntary separation program" to 60,000 employees.
Perhaps realizing the direction of public opinion, Microsoft specifically distanced the incident from AI and cyberattacks in its official announcement:
Information obtained by tech media from Microsoft's engineering team also shows that the patch that went online with hidden risks was completely written manually.
In other words, this is a pure, old-school manually crafted bug.
We mentioned in our previous commentary on the PHP2Shell vulnerability that AI programming is certainly not perfect, but it is most likely better than the average level of human programming. Instead of saying that AI has written a lot of vulnerable code, it is more accurate to say that it makes us notice that long before that, humans had already written much worse code.
Related reading: OpenAI discovered a century-old PHP vulnerability, putting 70% of websites worldwide at risk
However, AI vendors have repeatedly hyped the topic of "AI security" to demonstrate the capabilities of their models, which objectively pushed AI to the forefront of public scrutiny.
Just as in recent years, gamers tend to attribute the flop of a big game to its embedded DEI elements, when AI becomes the most prominent label in the tech industry, any systemic failure will inevitably make AI the target of public criticism; in turn, public pressure will force more regulation, further restricting the development of AI, and creating the highly defensive "sensitive skin" form of today's ChatGPT and Claude models.
On the other hand, although Microsoft's major outage this time seems to be just a coincidence, the similar "accidental incidents" in just the past week are far too frequent.
Figure | Inc. Magazine
A single accident can be attributed to the negligence of an unprofessional team; but when giants are staging crash events one after another, it is hard to use a single accidental factor as a shield —
Behind this, a certain systemic structural risk is emerging.
We always take the smooth operation of modern technology for granted, forgetting that the normal operation of such a huge system is itself an engineering miracle.
Most of the time, the maintainers of each module of the system cannot see the full picture of the system at all; everyone only does their own part in their respective scope according to pre-set rules, and the complex system runs through precise coordination.
This "emergent" collaboration mechanism is the cornerstone of modern society.