Microsoft said on Monday that a latent bug introduced in July caused last week’s global Microsoft 365 outage, knocking more than 80% of its networking gateway servers offline on 12 November. The disruption halted email across businesses, courts and hospitals worldwide, and blocked an automated chlorine alarm at a German theme park hotel.
Charles Lamanna, Microsoft’s executive vice president of Copilot, agents and platform, outlined the root cause on X, according to The Stack. Microsoft said the technical failure was unrelated to a cyberattack or artificial intelligence systems.
What triggered the July server bug
The fault sat inside a July update built specifically for Microsoft 365 networking gateway servers, which rolled out over several weeks without trouble, Independent Tech reported. Testing missed the defect because engineers used older hardware with fewer cores, while the bug only froze newer, multi-core servers under heavy load.
Fifteen hours before the outage began, Microsoft Teams failed over from an alternative gateway to the primary gateway that also handles Exchange Online, XDA reported. At the same time, nearly 14% of gateway server capacity was offline for maintenance after an unrelated issue the previous day.
Lamanna said Microsoft runs two independent, heterogeneous network gateways for high availability and described the Teams shift as “normal operations”. On their own, neither the traffic shift nor the maintenance outage would have broken the system.
When US traffic peaked on Thursday, 12 November, the extra load hit the reduced server pool and triggered the July bug. Automated systems routed requests to remaining healthy servers, overloading them in turn and causing a cascading failure across the fleet in under two hours.
Why recovery took around a day
Normal safeguards known as bulkheads failed to stop the cascade because of a separate issue, Lamanna said. Because the defective code had been running since July, Microsoft could not quickly revert it using standard rollbacks, flighting or feature flags.
Microsoft’s preliminary post-incident report in the Microsoft 365 Admin Center, relayed on Reddit and cited by TechRadar, noted that restored servers repeatedly crashed again as pent-up requests flooded back in. Engineers had to patch the code path, take servers offline in batches, install the fix and restart them.
Some Microsoft 365 apps moved off the broken gateway and returned earlier, though Outlook could not switch networks. Microsoft confirmed on Saturday, 14 November, that it had completed its fix and restored access to all remaining customers and regions.
The German hotel chlorine scare and global fallout
The email blackout stopped an automated warning at Hotel Ling Bao, part of the Phantasialand theme park in Brühl near Cologne, on Friday, 13 November, German public broadcaster WDR and Android Authority reported. A dosing system fault pumped excessive chlorine into the pool, and Phantasialand told WDR that the Outlook outage prevented the safety warning emails from reaching staff.
Around 40 emergency responders attended after guests began having trouble breathing. Fifteen people were triaged for airway irritation and four were taken to hospital, down from an initial report of five, with one staying for observation. Firefighters closed the pool, tested and replaced the water, and the pool was later cleared to reopen.
Across the global disruption, Downdetector logged more than 100,000 outage reports within an hour, and ABC News put the worldwide total above 7.4 million. Email-dependent features broke across Teams, Copilot, Defender, Purview, Planner and Intune, while Independent Tech reported that SharePoint, OneDrive and Office apps also failed.
Public services, courts and businesses reported widespread disruption during the outage, according to Android Authority:
- UK healthcare and courts: The GMB union said NHS emergency call staff could not pass on patient details normally, creating a backlog, while Manchester’s Minshull Street Crown Court sent some juries home early and did not bring some prisoners to court.
- Government and schools: Immigration New Zealand paused visa processing, Australian federal police, ASIO and schools were disrupted, CNN reported that courts in the Philippines were held up, Japanese officials switched to paper and text messages, and New York City Emergency Management told residents to call 311 or 911 because of delayed emails and online services.
- Transport and business: Britain’s National Rail, Germany’s Deutsche Bahn, VIA Rail Canada, IndiGo and Akasa Air reported email, ticketing or internal communications disruptions, Maersk warned customers of delayed updates, Hong Kong bank staff were disrupted and Workhorse Group postponed its third-quarter earnings release.
What happens next for Microsoft 365?
Lamanna said Microsoft was “deeply sorry” for the disruption to customers’ normal operations and stressed that neither the July defect nor its fix applies to broader Windows deployments.
Microsoft has released a preliminary explanation and said it will publish a final post-incident review later this week. The Stack noted that the full post-mortem is expected to clarify why 14% of networking server capacity was offline and why normal bulkheads failed to stop the cascading failure.