Microsoft 365 outage: timeline, failure mode and incident response breakdown

Exchange Online servers started failing health checks, the rest buckled under the load, and restored capacity kept collapsing for hours. A reconstruction from Microsoft’s own updates, with the mechanics and the config that prevents it.

This post is for engineers who run services other people depend on. It reconstructs the outage from Microsoft’s own status updates, works through the failure mode those updates describe, and looks at the incident response itself: what Microsoft did at each stage, why it worked or didn’t, and what you can configure in your own stack this week. We’ll stick to what Microsoft has said publicly and mark clearly where we’re reasoning beyond it. When Microsoft publishes its post-incident review, we’ll cover the confirmed root cause in a follow-up.

The timeline, from Microsoft’s updates

Microsoft posted 23 updates to EX1550543 in the first twelve hours, roughly one every 31 minutes, each with a “next update by” time. That cadence is the best public record of an incident this size you’ll get before a post-incident report, and the order in which Microsoft’s understanding changed is as instructive as any single update.

Three things stand out. First, Microsoft corrected its own diagnosis in public within an hour, from “elevated load” to “servers unavailable”. Second, the first containment step that worked (stopping removals) arrived about 25 minutes after the health-check finding, which is fast for a fleet that size. Third, the hardest stretch of the incident came after containment, when capacity that had been restored kept failing under real traffic. That last pattern is the one most worth understanding, because it can happen to a service of any size.

Why one service took eighteen others with it

Exchange Online holds mailboxes, calendars and contacts, and a large share of Microsoft 365 reads from it. Teams asks Exchange for your calendar when you schedule a meeting. Copilot asks it for mail when you query your inbox. Microsoft Graph proxies mail and calendar requests to it for thousands of third-party apps. Defender for Office 365 inspects mail and links that flow through it. When Exchange stopped answering, every feature on one of those paths failed with it, and Microsoft listed them one by one under MO1550821.

The useful part is the negative space. Microsoft said Teams calls, meetings, group chats and channel messages were largely unaffected, and files stored directly in SharePoint stayed reachable. None of those need Exchange on the request path. Teams as a product was partly up and partly down, split precisely along a dependency boundary.

That’s a design pattern you can apply deliberately. For every feature in your product, write down which dependencies sit on its request path and what it should do when each one times out: render from cache, hide the panel, queue the write, or fail the whole request. Most teams discover their real dependency graph during an outage. Writing it down beforehand is cheaper.

The failure mode: when the health check becomes the outage

A load balancer health check is a small, frequent request (open a connection, maybe fetch /healthz) with a timeout and a failure threshold. Fail it a few times in a row and the server leaves rotation. That’s exactly right when one server has a bad disk. It goes wrong when servers fail the check because they’re busy, or when the check itself is broken.

Microsoft’s 19:43 update covers both cases: servers were failing automated checks, and the failures may have traced back to a recent change in the component that processes those checks. Either way, the mechanics that follow are the same.

Why the loop accelerates

Utilization doesn’t rise linearly as you remove servers, and latency rises much faster than utilization. A rough model makes the shape visible. Take a pool of 100 servers at 60% CPU, a 100 ms service time, and a health probe with a 2-second timeout. Remove k servers and the survivors’ utilization becomes 0.6 × 100 / (100 − k). In a simple queueing model, mean response time scales with 1 / (1 − ρ), and the 99th percentile sits about 4.6 times above the mean.

Two things follow. Probes sample the tail, not the mean, so they start failing well before average latency looks alarming. And each server the probes remove pushes the survivors further up a curve that steepens as it climbs. Losing the first 20 servers costs you little headroom. Losing the next 5 tips the pool into the zone where probes fail on healthy-but-busy machines, which removes more of them. Add client retries, which raise demand exactly when capacity falls, and the pool can empty itself in minutes.

Google’s SRE book identifies overload as the most common cause of cascading failures, and lists health checks that fail under load as one of the ways a slowdown turns into a full outage. Microsoft’s 19:57 action, preventing additional servers from being removed, is the textbook brake.

Liveness, readiness, and the check that kills

Part of the fix is semantic. A check that answers “is this process alive?” should only ever restart a process that’s actually wedged. A check that answers “should this instance get traffic right now?” should only ever route traffic away, never restart. Conflating the two turns a load spike into a restart storm: busy pods fail liveness, get killed, come back cold, and take traffic from a queue that’s now even longer.

# Liveness restarts. Readiness only routes. Keep them separate.
livenessProbe:
  # event loop alive? no dependency calls
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 5
  # 60 s of failures before a restart: slow is not dead
  failureThreshold: 6
readinessProbe:
  # can it take traffic now? checks local queue depth
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3
startupProbe:
  httpGet: { path: /livez, port: 8080 }
  failureThreshold: 30
  # up to 150 s to warm caches before liveness applies
  periodSeconds: 5

Cap how much of the fleet a check may remove

The second part is a hard limit. If more than a third of a pool fails its checks at the same moment, the likelier explanation is a shared cause (the check, the network, a dependency) rather than a third of your servers independently going bad. Load balancers can encode that assumption. Envoy’s panic threshold routes to every host, healthy or not, once the healthy share drops below a set percentage. Outlier detection has a separate cap on how many hosts it may eject.

clusters:
- name: mail_frontend
  lb_policy: ROUND_ROBIN
  common_lb_config:
    # below 50% healthy, ignore health and use all hosts
    healthy_panic_threshold: { value: 50.0 }
  health_checks:
  - timeout: 2s
    interval: 5s
    unhealthy_threshold: 3
    healthy_threshold: 2
    http_health_check: { path: /readyz }
  outlier_detection:
    consecutive_5xx: 5
    interval: 10s
    base_ejection_time: 30s
    # never eject more than 30% of hosts
    max_ejection_percent: 30
  round_robin_lb_config:
    slow_start_config:
      # ramp traffic to a returning host over 2 min
      slow_start_window: 120s
      aggression: { default_value: 1.0, runtime_key: mail_frontend.ss_aggression }

Why restored servers kept falling over

The update at 23:59 UTC is the one experienced on-call engineers will have winced at. Microsoft said servers returned to service could become unresponsive again once they received traffic. At 01:06 it narrowed the symptom: a process on the affected servers responsible for establishing secure connections was being overwhelmed.

Microsoft hasn’t explained why the connection-setup process overloaded, and its later updates point at an upstream authentication and routing component, so treat what follows as general mechanics rather than a diagnosis. Still, the pattern is well known.

When a service has been failing for hours, demand doesn’t wait politely. Outlook desktop clients retry their connections. Mobile devices on ActiveSync reconnect on a timer. Sync tools and Graph integrations retry with their own policies, some of them aggressive. All of that accumulates. A server that comes back is the first available capacity the backlog has seen in hours, and it gets hit by connection attempts from far more clients than it served before the incident.

Each of those connections starts with a TLS handshake. A full handshake involves a key exchange and a signature with the server’s private key, which is expensive in CPU terms. A resumed session, using a ticket or cached session from an earlier connection, skips most of that work. After a restart, a server may have lost its local session cache, and clients whose tickets it can’t decrypt fall back to full handshakes. So a restored server faces its worst-case CPU cost per connection, at its highest connection rate, at a moment when caches above and below it are cold too.

If the handshake work alone saturates the CPU, the server can’t answer health probes either, and the load balancer pulls it again. Nothing about the original trigger needs to still be present for this to repeat. The overload is now self-sustaining, which is the behavior distributed-systems researchers call a metastable failure.

Measure your own handshake cost

You can see the gap between full and resumed handshakes on your own endpoints with stock OpenSSL. Run it against a staging host, not production.

# Full handshakes: a new session every connection
openssl s_time -connect staging.example.com:443 -new -time 10

# Resumed sessions: reuse the first session for every connection
openssl s_time -connect staging.example.com:443 -reuse -time 10

# Compare "connections per user sec". The ratio is your headroom cost
# when a restart or key rotation forces every client back to full handshakes.

Keep resumption working across restarts

If every host in a pool shares the same session-ticket keys, a client can resume against any of them, including one that just restarted. Distribute the keys from a secret store, rotate them on a schedule, and keep the previous key loaded for decryption so rotation doesn’t force a fleet-wide wave of full handshakes.

ssl_protocols        TLSv1.2 TLSv1.3;
# per-host cache, lost on restart
ssl_session_cache    shared:SSL:50m;
ssl_session_timeout  4h;
ssl_session_tickets  on;

# Same key files on every host. First key encrypts new tickets;
# the second still decrypts tickets issued before the last rotation.
ssl_session_ticket_key /etc/nginx/tls/ticket_current.key;
ssl_session_ticket_key /etc/nginx/tls/ticket_previous.key;

Shed load before the expensive step

Rejecting work is the move that actually breaks a metastable loop. A fast rejection costs almost nothing; a request that times out costs its full share of work and still fails. Where you reject matters too. If TLS setup is the expensive step, a limit that only applies after the handshake (most HTTP-layer rate limiting) protects nothing. The limit has to sit at the connection layer, before the handshake starts.

frontend fe_mail
  bind :443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
  maxconn 20000
  # new sessions/sec accepted, enforced before TLS
  rate-limit sessions 1500

  # Per-client connection rate, evaluated at accept time, pre-handshake
  stick-table type ip size 1m expire 60s store conn_rate(10s)
  tcp-request connection track-sc0 src
  tcp-request connection reject if { sc0_conn_rate gt 60 }

  default_backend be_mail

backend be_mail
  balance leastconn
  server m1 10.0.1.11:8443 check inter 5s fall 3 rise 2 slowstart 120s
  server m2 10.0.1.12:8443 check inter 5s fall 3 rise 2 slowstart 120s

Give every client a retry budget

Retries are the fuel. Three retries per call at each of three layers turns one user request into as many as 64 backend attempts during an outage. Exponential backoff with jitter spreads retries out but doesn’t reduce their number. A retry budget does: once retries exceed a fixed share of recent calls, the client stops retrying and fails fast.

import random, time, collections

class TransientError(Exception): pass

class RetryBudget:
    """Allow retries while they stay under `ratio`
    of calls in the last `window` seconds."""
    def __init__(self, ratio=0.1, window=10.0, min_retries=3):
        self.ratio, self.window, self.min = ratio, window, min_retries
        self.calls = collections.deque()
        self.retries = collections.deque()

    def _trim(self, q, now):
        while q and now - q[0] > self.window: q.popleft()

    def record_call(self):
        self.calls.append(time.monotonic())

    def try_retry(self):
        now = time.monotonic()
        self._trim(self.calls, now)
        self._trim(self.retries, now)
        if len(self.retries) < max(self.min, self.ratio * len(self.calls)):
            self.retries.append(now); return True
        return False

def call(fn, budget, attempts=3, base=0.2, cap=10.0):
    for n in range(attempts):
        budget.record_call()
        try:
            return fn()
        except TransientError:
            if n == attempts - 1 or not budget.try_retry():
                raise
            # full jitter
            time.sleep(random.uniform(0, min(cap, base * 2 ** n)))

What Microsoft did, and why each step makes sense

Read as a sequence, Microsoft’s public actions follow the standard playbook for a self-sustaining overload, in roughly the right order. Each one is worth naming because each one is easy to skip under pressure.

1. Freeze changes. All deployments and configuration changes were paused around three hours in. During an incident, an unrelated deploy can mask signal or add a second failure, and nobody can tell which change did what.
2. Stop the removals. Preventing more servers leaving rotation halts the capacity loss before anything else. It accepts some errors from unhealthy hosts in exchange for keeping the pool alive.
3. Change the definition of healthy. Validating an alternative health check on part of the fleet before applying it broadly is the careful version of “the probe is lying, replace it”.
4. Automate restarts, carefully. A mechanism that detects and restarts unresponsive servers keeps capacity in service while other work continues, as long as it doesn’t become a restart loop of its own.
5. Shed by traffic class. Limiting which types of traffic reach the affected servers both protects them and tests whether one kind of request is driving the load.
6. Roll back the correlated change, in parallel. Reverting the network configuration change that coincided with onset, while still treating the symptom, avoids waiting for proof that may take hours.
7. Roll out the fix by slice. The 07:29 patch went to part of the infrastructure first, then expanded. After several mitigations that hadn’t held, that caution was warranted.

The comms lesson: your status channel can’t share the failure

Microsoft’s update discipline held for the whole incident: a post roughly every 30 to 60 minutes, each naming the time of the next one, including when the honest update was that the last action hadn’t worked. Silence is what fills a support queue, and a predictable cadence lets admins stop refreshing and plan around it.

The harder lesson is about where updates live. Admins normally read service health in the Microsoft 365 admin center, but Microsoft listed some admin center pages among the affected features. It pointed admins to status.cloud.microsoft instead, a channel that doesn’t depend on the failing service.

The same applies one level down. Many companies tell their own staff about outages through the tools that just failed: an all-staff email that never arrives, a Teams post that can’t load the channel. Some got it right anyway. Michigan’s Department of Technology, Management and Budget posted a system outage alert warning that state email was down and responses would be delayed, which set expectations for thousands of people in one message.

Run the exercise for your own stack. List every channel you’d use during an incident, including paging, the incident chat, the status page and the internal announcement, and next to each one write the systems it depends on. Anything that shares a dependency with your most critical service needs a fallback that doesn’t.

Status: Degraded
Impact: Some users can't open email or calendars. Chat and calls are working.
What we know: A dependency is overloaded. We've stopped further servers
  being removed from service and are restoring capacity in stages.
Changed since last update: [one line, or "No change"]
What you can do: Use [chat tool] for urgent messages. Don't reinstall clients.
Next update by: [HH:MM UTC]

Running an incident like this in Xurrent IMR

Nothing in an incident management platform would have stopped Exchange servers failing their health checks. The fix belonged to Microsoft’s engineers, and from the outside they found it under unusually hard conditions. What a platform changes is everything around the fix: how quickly the right person knows, whether forty alerts read as one problem or forty, whether the team stays coordinated across a shift change at hour eight, whether users hear something on schedule, and whether the review afterwards starts from a record or from memory.

Here’s how those stages map to Xurrent IMR, using the same failure mode on a smaller service of your own.

To make that concrete, here is what the incident record would look like for a fictional team hitting the same failure mode on their own mail front end. The service, the times and the pull request are invented for illustration; the sequence mirrors the one Microsoft went through.

One caveat worth taking seriously. If your incident chat runs on the same platform that’s failing, as it would have for teams whose response lived in Microsoft Teams during this outage, keep a second channel ready. Xurrent IMR supports Slack, Microsoft Teams and Google Chat for that reason, but the fallback has to be agreed before the incident, not during it.

What we still don’t know

• What changed in the component that processes health checks, and did it cause the first servers to fail or only fail to recover them?
• Why did the process that establishes secure connections become overwhelmed: load, a defect, or a configuration that changed its cost per connection?
• Was the network configuration change that coincided with onset the trigger, a contributor, or coincidence?
• How does the upstream authentication and routing component patched at 07:29 relate to the health-check and TLS findings?
• How much of the load on restored servers came from client retries and reconnects versus internal routing?

Sources

1. Microsoft 365 service status, Microsoft. Incidents EX1550543 and MO1550821.
2. Microsoft 365 outage, 12 November 2026, systemdesign.academy. UTC timeline of Microsoft’s service health text, update count.
3. Microsoft 365 outage update, 07:29 UTC and 21:01 UTC, SmartScope. Mirrors of Microsoft’s status feed.
4. Microsoft 365 outage, current status, LazyAdmin. Deployment pause, overnight timeline, the separate Teams incident and SLA credit terms.
5. Microsoft 365 outage hit Outlook users across the country, Mercer County Outlook. Downdetector figures and the Michigan DTMB alert.
6. Addressing Cascading Failures, Google SRE book, chapter 22.
7. Timeouts, retries, and backoff with jitter, Amazon Builders’ Library.
8. Metastable Failures in Distributed Systems, Bronson et al., HotOS 2021.

Analysis based on Microsoft’s public status updates. Microsoft has not confirmed a root cause; we’ll correct anything its post-incident review contradicts.