This story is currently available in англійська only — a translation isn't ready yet.
Microsoft blames July gateway defect for global 365 email outage
A preliminary post-incident report says the flaw left more than 80 percent of network gateway servers unresponsive during the November 12 disruption.
ℹ️ Озвучка браузером · студійний голос ШІ незабаром

Microsoft said a software defect introduced in a July 2024 update to its Microsoft 365 network gateway servers caused the outage that took Exchange Online and other services offline for a day last week, in a preliminary post-incident report the company published on November 15.
The update passed load testing and was deployed without impact at the time, Microsoft said. The report states that its load testing "tended to focus on weaker hardware with fewer cores," while the defect — lock contention inside an internal Windows process that exhausted CPU and memory — worsened on machines with more cores and newer hardware.
Two events combined to trigger it. Teams failed over to the primary gateway that supports Exchange Online at 0107 UTC on November 12, creating a surge in load, while almost 14 percent of server capacity for the networking gateway was out of service. Microsoft said the service had sufficient capacity headroom for either event on its own.
Within two hours the defect had been encountered across the fleet. "At its lowest," the report states, "more than 80 percent of servers were unresponsive." Automated recovery processes were not designed to restore service from that state, and a straightforward rollback was impossible because the defect was months old. Engineers instead built an update that returned the code path to the previously known good state in Windows and applied it to groups of servers.
Microsoft's connectivity anomaly detection monitor flagged rising connectivity issues at 1603 UTC on November 12; the first support case arrived at 1633 UTC. By 0030 UTC on November 13 the company had identified connection negotiation as the main culprit, and at 0720 UTC it applied the mitigation. Service was confirmed healthy at 1610 UTC, with a final customer update at 0415 UTC on November 14.
Teams, OneDrive, SharePoint and Office apps were able to fail out of the faulty network gateway infrastructure; Exchange Online could not, for scale reasons, which the report gives as the reason for the length of the Outlook disruption. A final post-incident report is due November 20.
Sources — how we know this
3 independent sources corroborated this
- PrimaryHacker News Front Page
- CorroboratingCNBC Top News
- CorroboratingThe Guardian — UK news