ReferenceIncidents
Era 4 · Series BBig tenant hot shard
The biggest tenant's key landed on one shard. Its reads and writes cost 3× the CPU, and everything else on that shard waits behind it.
What happens#
The biggest tenant's key landed on one shard. Its reads and writes cost 3× the CPU, and everything else on that shard waits behind it.
How to prevent it#
Shard the tenant by key, and keep its reads in a Cache in front of the Database.
The postmortem opens the first of these that you do not own yet: