Advisory firm urges employers to check workflows and security controls before declaring operations fully restored
Last week's global Microsoft 365 outage, which disrupted workplaces for nearly a day, was triggered by a bug that had gone unnoticed since July, a senior Microsoft executive has said.
The disruption began on 12 November, cutting users off from their Exchange Online mailboxes before spreading to Teams, SharePoint, Copilot and other Microsoft 365 services. Microsoft confirmed services had fully recovered by 16:10 UTC the following day.
Charles Lamanna, Microsoft's executive vice president of Copilot, Agents and Platform, said three factors together caused the outage of the networking gateway that sits in front of many Microsoft 365 services.
"We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations," Lamanna said in a post on X on Monday.
According to Lamanna, the defect caused gateway servers to become overwhelmed and unresponsive under certain load conditions because of lock contention inside an internal process.
"This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers," he said, adding that it does not apply to broader Windows deployments.
The fix began deploying in July and ran worldwide for more than three months without issue. Lamanna said load testing missed the flaw because labs focused on weaker hardware with fewer cores, while the contention worsens on newer machines with more cores.
Factors that led to outage
Two further factors exposed the bug. About 15 hours before the outage, Teams failed over to the primary gateway that also supports Exchange Online, increasing its load. At the same time, nearly 14% of the gateway's server capacity was out of service because of an unrelated issue from the previous day.
The combination caused servers to stop responding during peak US traffic. As traffic was rerouted to healthy endpoints, those servers hit the same problem, triggering a cascading failure.
"In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive," Lamanna said. At its lowest point, more than 80% of servers were unresponsive.
Teams, OneDrive, SharePoint and Office apps were able to fail out of the faulty gateway and were restored earlier, but Exchange Online could not.
"Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again," Lamanna said.
He added that because the bug was months old, normal rollback tools could not quickly revert it. Engineers instead developed an update restoring the previous code path and applied it to groups of servers at a time.
A final post-incident review with more details will be provided later this week, Lamanna said.
What should employers do next?
Research and advisory firm Infotech has cautioned organisations against treating Microsoft's service-health updates as confirmation that their operations are back to normal.
In a research note issued on 13 November, before Microsoft disclosed the root cause, the firm advised IT leaders to verify that business-critical workflows, integrations and security controls had recovered before declaring normal operations restored.
"CIOs need a clear basis for telling the business it can return to normal operations," said Kate Wood, AVP of Research Development at Infotech. "That means knowing which critical processes have been tested, where security coverage still needs checking, and which teams must continue using workarounds."
The firm said contingency plans held up where a manual alternative existed but stalled where email was the only channel.
It also urged employers to warn staff about phishing, noting that threat actors have commonly impersonated Microsoft support staff during previous large-scale outages.
Employees should be alerted to credential-harvesting attempts and deepfake voice and video calls during the disruption and recovery, according to the note.
Infotech added that service credits are not automatic. Organisations seeking them should preserve evidence of their own disruption and have legal teams review agreements for eligibility and submission deadlines.
For the longer term, the firm recommended testing continuity plans against prolonged outages affecting several cloud services, and identifying where reliance on a single provider creates shared points of failure.