Play
Patterns

One command removed the index: AWS S3

A mistyped input removed the servers behind S3's index and placement subsystems, and us-east-1 lost S3 for about four hours.

On 28 February 2017, an S3 team member ran a command to remove a few billing servers, and a mistyped input removed a larger set. The removed servers ran the index and placement subsystems for us-east-1, and S3 could not serve GET, LIST, PUT, or DELETE requests for about four hours.

The system#

The index subsystem holds the metadata and location of every object in the region. The placement subsystem allocates new storage and needs the index to work. The team was debugging a slow billing process and used a playbook to remove a few billing servers. The wrong input removed capacity the index and placement subsystems needed, and both required a full restart. The index had not been fully restarted in the region for years. Other AWS services failed, including EC2 launches and Lambda. The status dashboard also depended on S3.

The board#

Drag to pan. Hold Ctrl or Cmd and scroll to zoom.

At 3.00k Users and a running Disk full incident the Simulation solves this Design to Apdex 0.57, p99 351 ms, 35.0% errors and $35/s Burn. The board runs the game’s Disk full incident on the index Database, which stands for the index losing its servers. The caption is the game’s result, not AWS’s outage.
WhereSettingValueWhy
App ServerInstance sizelargeLarge, because the app tier serves the object requests.
App ServerCopies3Three copies stand for the S3 front end.
Object StorageStorage classstandardThe object store holds the bytes.
Object StorageLifecycle (days)30Older objects move to a cheaper class after a month.
DatabaseInstance sizelargeLarge, because the index holds the metadata for every object.
DatabaseBackupsnightlyA nightly snapshot is the only copy outside the index.

Research this Design needs: Load balancing, Object storage, Object storage, Multiple regions, Multiple zones.

The Object Storage holds the bytes, and the Database is the index that locates them. The Load Balancer and App Server are the front end. The board runs the game's Disk full incident on the index Database, which stands for the index losing the capacity it needs.

What broke#

A single destructive command removed capacity that two critical subsystems needed. The index had no fast restart path at that size, and placement could not recover until the index did. The blast radius crossed service boundaries, and the status page sat inside it, so customers could not see the status. The event lasted from 09:37 to 13:54 Pacific.

What changed#

AWS reprioritized its work to split the index subsystem into smaller cells, so a failure stays inside one cell and recovery starts smaller. The game's Multi-AZ and Multi-region research are the closest guardrails, and Bulkhead is the same idea for a service.

Where the game stops#

The game's Disk full incident takes the index's writes. S3 lost the servers that held the index, and the blast radius crossed into other services. The board has no cells, no playbook, and no status-dashboard dependency. The caption is the game's incident, not the 2017 outage.

Sources#