Move tables between clusters: GitHub
GitHub split one MySQL cluster into schema domains, then moved tables to separate clusters with Vitess and a write cutover.
GitHub.com ran on one MySQL cluster called mysql1 for about a decade. By 2019 the cluster answered close to a million queries a second, and every feature shared the same tables. The team split the code first: tables went into named schema domains, and a linter failed a build when a query crossed a domain boundary. Only then did the tables move.
The system#
The move used two independent methods. Vitess put a proxy in front of MySQL and copied whole tables between clusters in the background. A custom write cutover rode on ordinary MySQL replication and ProxySQL, and the read-only window lasted tens of milliseconds. By 2021 the same tables answered 1.2 million queries a second across several clusters, and the average load on each host had halved.
The board#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| Load Balancer | Policy | least-conn | The load balancer sends each request to the cluster that owns its data. |
| App Server | Instance size | large | Large, because the core application keeps the busiest tables. |
| App Server | Copies | 3 | Three copies stand for the app tier over the core cluster. |
| App Server | Instance size | medium | Medium, because the feature domain is smaller than the core. |
| App Server | Copies | 2 | The feature domain scales on its own. |
| Connection Pooler | Pooler mode | transaction | A connection is lent for one transaction, so many app connections share a few. ProxySQL does this at GitHub. |
| Connection Pooler | Pooler mode | transaction | Each cluster gets its own pool, as each domain gets its own cluster. |
| Database | Instance size | large | Large, because the core tables still take most queries. |
| Database | Engine | mysql | MySQL, the engine GitHub runs. |
| Database | Instance size | medium | Medium, sized for the feature domain alone. |
| Database | Engine | mysql | MySQL, the engine GitHub runs. |
| Read Replica | Instance size | large | Match the primary, so the copy takes an equal share of reads. |
| Link Load Balancer → App Server | Kinds carried | write,login | Writes and logins go to the cluster that owns the core tables. |
| Link Load Balancer → App Server | Kinds carried | page,read | Pages and reads go to the feature domain, which the split made independent. |
Research this Design needs: Load balancing, Connection pooling, Read replicas, Connection pooling, Read replicas, Sharding.
The two App Servers are the two schema domains, core and feature. Each has its own Connection Pooler, which stands for ProxySQL, and its own Database. The Read Replica is one of the read copies that took load off the primary. The Load Balancer's kinds stand for the domain split, not a real read/write split.
Why it works#
The split is virtual before it is physical. Once the code cannot join across a domain, the tables can move one domain at a time, and a mistake stays inside one cluster. Split the database by feature and expand and contract are the two steps that make the move safe.
Where the game stops#
The game's Connection Pooler is PgBouncer, not Vitess or ProxySQL, and it cannot route a query by table. The board shows two domains and two databases, not the linters, the VTGate processes, or the 130 tables that moved in one cutover. The caption is Quorum's result, not GitHub's query rate.
Sources#
Partitioning GitHub's relational databases to handle scale, The GitHub Blog: the schema domains, the linters, Vitess, and the write cutover.
Vitess documentation: the proxy, the VTGate and VTTablet processes, and the sharding model.
ProxySQL: the connection pooler GitHub used for the cutover.