決済システムを設計する
システム設計の面接向けに決済システムを設計します。冪等キーを使い、二重課金を避け、サーガとアウトボックスで手順を調整し、突合を行います。
A payment system moves money exactly once even though networks fail and clients retry. Give every payment attempt an idempotency key, record state in a ledger inside one database transaction, call the payment provider such as Stripe, and coordinate the steps around it with a saga and a transactional outbox. Reconcile your records against the provider's every day, because something will still disagree.
Requirements#
Functional:
Create a payment for an order, capture it, and refund it.
Show payment status to the customer and to support.
Support one or more providers.
Non-functional:
Never charge twice and never lose a charge. Correctness beats latency.
Every state change is auditable.
Survive a provider outage, a timeout and a crash between any two steps.
Estimate the load#
Assume 10 million orders a month.
Rate: 10,000,000 / (30 x 86,400) is about 4 per second on average. A flash sale at 20x is about 80 per second.
Storage: each payment writes about 5 ledger rows at 300 B, so 10,000,000 x 5 x 300 B is 15 GB a month.
Provider latency: a card charge takes 1 to 3 seconds, so at 80 per second you hold about 160 to 240 requests open at once.
This is small. A single relational database handles it. The difficulty is correctness, not scale.
The design#
The request path#
The client calls your API with an idempotency key it generated for this purchase. The App Server stores the key and a PENDING payment row in one transaction, then calls the Payment Provider with the same key. See idempotency keys.
Double charge#
The classic failure: the charge times out at 10 seconds but succeeded, the client retries, and the customer pays twice. With a key, the provider and your own table both return the first result. The check and the write must be atomic, so use a unique index on the key.
Orchestrating the steps#
Reserve stock, charge, then ship touches three services, so no single transaction covers it. Use a saga: each step is a local transaction, and a failure runs compensations such as a refund. Publish each step with a transactional outbox so the database change and the event never disagree, and let a Queue carry them. Two-phase commit is the alternative, and its costs usually rule it out.
Ledger and reconciliation#
Store money as an append-only double-entry ledger in a Database, never as a mutable balance. Each night compare your ledger with the provider's settlement report and flag differences for a person.
Where it breaks#
The provider is down. Checkouts fail as upstream errors. Queue the attempt and retry later, or fail over to a second provider. See Payment provider outage.
Retries cause duplicates. Retrying without a key is the double charge. Fix the key first, then add retry with backoff. See Double charge.
A flash sale saturates checkout. Demand jumps twentyfold and the database and provider saturate. Shed low-value traffic and keep checkouts, as in load shedding, and see Flash sale.
A crash between charge and record. The provider charged but your row says PENDING. The outbox and a sweeper that asks the provider about stale PENDING rows close this gap, and reconciliation catches what remains.
Build it in Quorum#
Place a Payment Provider after the App Server on an e-commerce Company and a Database for checkouts. Research Checkout and Idempotency keys. The Double charge incident only happens when the charge Link retries and there is no idempotency key, which is the interview's failure in miniature.
The first board is the retry half of that story. An App Server calls an undersized Database over a Link with a short timeout and backoff retries, so failed calls wait and spread out. Look at Apdex and the p99, and at how few extra calls the Database receives compared with naive retries. The charge Link to the Payment Provider takes the same policy. Backoff limits the harm of a retry, and only the idempotency key makes a retry safe, so both are needed. See retry with backoff.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | A large App Server never runs out first, so the Database link is where the pressure shows. |
| Database | Instance size | micro | A micro Database is too small for these Users, which stands in for a slow dependency. |
| Link App Server → Database | Timeout | 1 s | A Database call that takes a second is already stuck, so the caller gives up and frees its slot. |
| Link App Server → Database | Retries | backoff | Backoff waits between attempts and keeps the extra calls inside a small budget, so a struggling Database gets room to recover. |
Research this Design needs: Timeouts and retries, Backoff with jitter.
The second board is the queue side of the design, which carries the steps that happen after the charge. The App Server hands work to a Queue and Workers drain it, with a dead-letter queue and three attempts so a failing task cannot block the line. Look at Queue depth and the Database load, because Workers read and write beside the users. It stands for the events your outbox would publish, such as a receipt or a stock update, though the game's board is an export job and not a saga. See slow work to a queue and workers.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | small | It only enqueues exports now, so it stays small. |
| Database | Instance size | medium | Workers read from it while users do, so give it room. |
| Queue | Dead-letter queue | On | A task that keeps failing is parked after its attempts, so it cannot block the line forever. |
| Queue | Max attempts | 3 | Three tries cover a blip. Beyond that the task is broken and a retry only burns Worker time. |
| Queue | Max length | 10000 | A full Queue refuses new tasks instead of hiding a backlog that grows without limit. |
| Worker | Instance size | small | An export reads many rows, so a Worker needs room beyond the smallest box. |
| Worker | Copies | 3 | Several copies drain the backlog faster than one box could, and Queue depth tells you when to add more. |
| Worker | Prefetch | 10 | Each process takes ten tasks ahead of time, which spares broker round trips. A crash loses only those ten. |
Research this Design needs: Message queues, Background jobs, Background jobs, Message queues.
Research Transactional outbox for the event side, and the Stale Checkout Sandbox shows why checkouts need fresh reads.
The game covers the charge, retries, duplicates and the provider outage with a second provider. It has no saga, no ledger and no reconciliation, so those you learn from the pattern pages.
Common questions#
How do you prevent double charges in a payment system?#
Send an idempotency key with every charge, and store it with the result under a unique index. A retry with a known key returns the stored result and charges nothing.
What is an idempotency key?#
A unique value the client generates once per operation and sends on every attempt. The server uses it to recognise a repeat and replay the first answer instead of acting again.
What is the saga pattern in payments?#
A saga splits a business flow into local transactions, one per service, and undoes finished steps with compensating actions when a later one fails. A refund after a failed shipment is a compensation.
What is payment reconciliation?#
It is a scheduled comparison of your ledger against the provider's or bank's records. It finds charges that exist on one side only, such as a crash between charge and record, so a person can fix them.