HomeArticle

Microsoft's patch rolled out three months ago triggered widespread system crashes across 74 countries and regions.

差评2026-11-17 09:42
Humanity will have no margin for error.

I believe everyone has heard the news: Microsoft suffered a massive global outage in the past two days!

The incident started in the early hours of Beijing time on the 13th, when a large number of US users suddenly found that they could not connect to their Outlook mailboxes no matter what they tried.

Outlook is Microsoft's office software, which includes not only email, but also functions such as schedule management and meeting invitations. Most companies and government agencies abroad use Outlook for communication during work.

At first, people were posting online asking what went wrong.

However, before Outlook was restored, Microsoft's Teams, Copilot, and SharePoint also began to experience connection issues one after another.

Later, people checked Downdetector, the website that monitors service failures, and found that reports of Microsoft's outage had come from 74 countries and regions.

What? Microsoft, the third largest company by market capitalization in the world, crashed??

Normally speaking, the core business of such a large technology giant cannot go down. Even if it does, it will definitely be fixed quickly, right?

But no one expected that Microsoft's services would not be fully restored until the early hours of the 14th.

So how much did the global workforce get affected during these 24 hours?

Work efficiency has regressed by 20 years.

For example, hundreds of hospitals in the UK have gone back to using paper medical records, and even many surgeries have been temporarily canceled.

Because more than 1 million employees of the UK's National Health Service (NHS) rely on Microsoft services for the NHS.net Connect system they use to send and receive emails, hold meetings, and transfer files on a daily basis.

After the NHS system failed to log in, all patient data could not be sent out or received, and even emergency referrals for critically ill patients could not be notified.

The UK court system, which normally uses Outlook to send and receive emails, even sends civil case judgments via email.

As a result, when the mailbox went down, the judgment results could only be postponed to be announced next Monday...

One of the affected courts ↓

Japan was also affected in the same incident.

The GSS, the common government platform built by Japan's Digital Agency for all departments, runs on Microsoft 365.

Now, Outlook in all departments could not send or receive emails, and people had to work via phone and SMS. Perfect, everyone went back to the 90s and started the retro Y2K trend.

According to estimates by US research institutions, the losses caused by this Microsoft outage are around 7 billion US dollars.

With so many people affected, netizens have created a lot of meme pictures, as everyone loves to watch the excitement no matter how big the problem is.

Someone posted a picture of "Microsoft 364", saying that it is no longer Microsoft 365, we have to subtract one day from the total this year.

Microsoft has to change its name! Don't ask, it's the rule.

Another netizen posted a photo of NASA's control center, where a room full of staff were hugging and cheering: Microsoft Outlook is down, and it's Friday today...

It can only be said that for many workers, not being able to open the mailbox on Friday morning is indeed equivalent to the weekend starting early.

Speaking of which, here comes the question: what on earth caused such a large-scale failure by Microsoft?

At first, Microsoft suspected that a component responsible for checking server health had an error, but it was not until yesterday (the 16th) that it released a preliminary incident report, and people finally figured out the cause.

After reading the report, all I want to say is: It was indeed not caused by hackers, Microsoft messed it up all by itself.

When users around the world connect to services such as email, Teams, and cloud drives in Microsoft 365, they have to go through a layer of "network gateway" first. It's like going through the turnstile when entering a subway station.

Microsoft prepared two sets of such gateways: one primary gateway and one backup gateway.

Each set of gateways has a large number of servers, which are equivalent to rows of turnstiles. Users pass through the turnstiles first, and then are directed to the corresponding services behind.

Before the incident, emails went through the primary gateway, and Teams went through the backup gateway. The two sets are backups of each other, so if one set encounters a problem, the service can be switched to the other.

This time, the one that went wrong was exactly the primary gateway.

According to Microsoft's official statement, there were three hidden factors that led to the accident.

The first hidden factor was that in July, Microsoft installed a patch on the gateway servers, which contained a defect. It would cause an internal program on the server to experience "lock contention".

Lock contention is like fighting for a single resource. For example, there is only one official seal in the business hall, and every clerk has to queue up to use it to handle business. The more clerks there are, the longer the queue will be, and the work will never be finished.

Therefore, this defect is even more serious on servers with more CPU cores.

But the tricky part is that Microsoft mostly used servers with fewer cores for stress tests, so naturally it could not find this problem. As a result, this patch ran safely on gateways around the world for more than three months...

The second hidden factor was a switch 15 hours before the incident.

Teams was moved from the backup gateway to the primary gateway, and was squeezed together with Exchange Online, the email service behind Outlook. Microsoft said this kind of switch is very normal, but the pressure on the primary gateway became a little higher.

The third hidden factor was that the day before the incident, due to another unrelated issue, 14% of the servers on the primary gateway were still shut down and being restored slowly.

When the three hidden factors came together:

14% of the servers on the primary gateway were already down, all Teams traffic was crowded over, and it happened to coincide with the peak traffic of US working hours.

As a result, the remaining servers on the primary gateway got busier and busier, directly triggering lock contention, and started to freeze one after another.

Then, Microsoft's protection mechanism backfired:

When a server freezes, the system automatically transfers its users to other still normal servers on the primary gateway. But since all servers have the same problem, wherever the traffic goes, it freezes immediately.

As a result, in less than two hours, more than 80% of the primary gateway servers were down.

What's worse, there were originally partitions in the gateway to prevent failures in one place from spreading to other places. But these partitions failed to block the spread due to another problem, and the failure spread all the way to the whole world...

At this point, the two sets of gateways came into play.

Services such as Teams, OneDrive, and SharePoint were successively switched to the backup gateway and recovered earlier; but the email service was too large, the backup gateway could not handle it, so it could only stay on the primary gateway and struggle, which is why Outlook mailbox was down the longest this time.

To be honest, I understand all these small hidden factors explained by Microsoft. But I want to say, isn't there the biggest hidden factor that you didn't mention...

OK, the story is not over yet.

After Microsoft found the problem, it immediately started restarting the servers. But as soon as the servers restarted, they were crushed by the large amount of backlogged traffic.

As a result, users were tortured over and over again, again and again.

This reminds me of when there was a power outage when I was a kid, the power bureau worked hard to fix it, but the moment the power came back, everyone turned on their air conditioners, and the power tripped again.

Because this defect existed for several months, a quick rollback didn't work. In the end, Microsoft could only revert that part of the code to the old version, and restart the servers in batches, to restore the services little by little.

Microsoft has not yet said how it will compensate everyone, but it will most likely pay according to the contract.

According to the Service Level Agreement (SLA) signed by Microsoft with enterprises, if the availability of a service is less than 99.9% in a month, 25% of the monthly fee for this service will be refunded; if it is less than 99%, 50% will be refunded; if it is less than 95%, the full fee will be refunded.

This time, the email service was down for nearly 24 hours. Calculated based on 30 days a month, the availability is 96.7%, which means half a month of email service fee will be refunded.

If you run a company with 100 employees, using Microsoft 365 Business Standard, the total monthly fee for the whole company is 1400 US dollars. If the whole set of services is refunded at half price, the maximum refund is only 700 US dollars. If only the email part is counted, the refund will be even less.

Such a small amount of