One control plane, 73 hours: Roblox
A Consul streaming change under load took Roblox down for 73 hours, because the control plane was also the path for deployments and monitoring.
On 28 October 2021, Roblox went down for 73 hours. A new streaming feature on Consul, the service discovery and coordination layer, caused contention under load, and a BoltDB bug made it worse. The control plane also carried deployments and monitoring, so the team could not see or fix the outage.
The system#
Roblox ran hundreds of internal services, and one Consul cluster supported several workloads. The team enabled a new Consul 1.10 streaming feature on a traffic-routing backend the day before the outage. Under high load, the new protocol contended, and BoltDB's free-page list made writes pathologically slow. Consul's leaders fell behind, and the control plane could not keep the services consistent.
The board#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| Load Balancer | Policy | least-conn | The load balancer spreads traffic across the app tier. |
| App Server | Instance size | large | Large, because the app tier carries the game traffic. |
| App Server | Copies | 1 | One copy stands for the fleet, because the control plane is shared by every copy. |
| Database | Instance size | large | Large, because the game state lives here. |
Research this Design needs: Load balancing, Container orchestration, Service mesh, Load balancing, Container orchestration, Service mesh, GitOps.
The Load Balancer and App Server are the game tier, the Database is the state, and the Kubernetes Cluster and Service Mesh are the control plane. The board runs the game's Bad config push incident on the App Server, which stands for a control-plane change that reached the fleet.
What broke#
A control plane is a shared dependency. A change to it under load can take down every service that relies on it, and the monitoring and deployment paths sit in the same blast radius. Roblox lost visibility and the ability to deploy a fix while the control plane was unhealthy. One Consul cluster for several workloads widened the impact. The recovery took 73 hours because the team had to rebuild the Consul cluster and the cache, not roll back one deploy.
What changed#
Roblox disabled the streaming feature, rebuilt the Consul cluster, deleted obsolete data, and worked with HashiCorp on a BoltDB successor. GitOps Staged is the closest guardrail: a change lands on one region first. Bulkhead is the other: keep the failure inside one part.
Where the game stops#
The game has no Consul, so the Cluster and Mesh stand for the control plane. The config incident is a config push, not a streaming feature. The board cannot show 73 hours, the BoltDB bug, or the monitoring dependency. The caption is the game's incident, not Roblox's outage.
Sources#
Roblox Return to Service, Roblox: the timeline, Consul streaming, the BoltDB bug, and the changes made.
An Update on Our Outage, Roblox: the initial statement and the 73-hour recovery.
Consul architecture, HashiCorp: what Consul does, and why a control plane is a shared dependency.