Play
Patterns

Canary the edge rules: Cloudflare

A single WAF rule took every Cloudflare edge CPU to 100%, and the fix was a canary before the global rollout.

On 2 July 2019, Cloudflare deployed a new WAF managed rule at 13:42. The rule contained a regular expression that backtracked on long inputs, and every CPU core that served HTTP and HTTPS went to 100%. The network served 502 errors for 27 minutes, and the fix was to disable the rule.

The system#

Cloudflare's edge is anycast: every point of presence runs the same proxy, WAF, and cache, and a rule deployed globally reaches all of them at once. That is the strength and the failure mode. The postmortem says the rule was a minor change to XSS detection, deployed through an automatic process, and the first page came three minutes later. The team had a kill switch, which is why the outage ended in 27 minutes rather than hours.

The board#

Drag to pan. Hold Ctrl or Cmd and scroll to zoom.

At 4.00k Users and a running Regex backtracking incident the Simulation solves this Design to Apdex 0.99, p99 86 ms, 0.5% errors and $9.18/s Burn. The board runs the 2019 edge shape with the game’s regex incident on the WAF, not the 27-minute outage. The caption is Quorum's result, not the company's numbers.
WhereSettingValueWhy
WAFParanoia level2A middle level blocks most bots and few real visitors. The 2019 rule that failed was a managed rule at this layer.
CDNTTL (seconds)60A minute keeps a page current while the edge absorbs repeat reads.
CDNPurge on deployOnA deploy clears the edge, so no visitor gets the old release.
Edge FunctionStacknodeThe edge runs JavaScript or TypeScript, and Rust or Go built to WebAssembly. Workers run in the same family of runtimes.
App ServerInstance sizemediumMedium, because the edge takes the cheap requests first.
DatabaseInstance sizemediumMedium, behind both the edge and the app tier.
Link CDN → Edge FunctionKinds carriedpage,readPages and reads run at the edge.
Link CDN → App ServerKinds carriedwrite,login,botWrites and logins need the app tier.

Research this Design needs: DNS round robin, Web application firewall, Content delivery, Edge compute, Web application firewall, Content delivery, Load balancing, Node.js.

The DNS is the anycast entry, the WAF is the managed rule layer, the CDN is the edge cache, and the Edge Function is a Worker. The App Server and Database sit behind them. The board runs the game's Regex backtracking incident on the WAF, so the caption shows what one bad rule does to the request path.

Why it works#

The shape puts one rule in front of every request, so a rule that costs CPU costs it everywhere. A canary sends the rule to one point of presence first, and a kill switch ends a bad rollout without a deploy. Blue-green and canary and WAF and bot management are the two controls.

Where the game stops#

The game's Regex incident costs a slice of the traffic. The real rule took the whole network down for 27 minutes. The board has one WAF, not a global anycast fleet, and it cannot show a kill switch. The caption is the game's incident, not Cloudflare's outage.

Sources#