Change data capture
Read the database's change log and publish each change as an event, so search, warehouse, and cache stay in step without the app writing twice.
Change data capture turns the database's own log into a stream of events that other systems follow.
Use it when#
A second copy of the data must follow the database: a search index, a warehouse, a cache, or another service.
The app currently writes to two stores, and they drift apart after a crash.
You cannot change every writer to emit events.
How it works#
Every database writes changes to a log before it applies them. A connector such as Debezium reads that log and publishes each insert, update, and delete to a stream. Consumers read the stream and apply the changes to their copies. The app writes once, to the database, so there is nothing to disagree.
Key the stream by the row's primary key. Order holds only inside a partition, so an unkeyed stream can apply an old write after the newer one.
What it costs#
Copies lag the database by however far behind the consumer is, and a consumer further behind than the stream's retention loses changes. Changes are row-level, so consumers see the schema, and a migration can break them. You run a connector and a stream. A consumer that writes back to the source can start a loop.
In Quorum#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | medium | Medium, so the App Server is not the limit. |
| Database | Instance size | medium | Medium. The app writes here once, and the connector reads its log. |
| Event Stream | Partition key | On | Keyed by the row, so changes to one row apply in the order they committed. |
| Event Stream | Partitions | 3 | Three partitions allow up to three consumers. This Design links one Worker, so that Worker sets the read rate. |
| Worker | Instance size | xlarge | Xlarge. Every write becomes a task, so it needs more room than the App Server's share of work. |
| Realtime Hub | Stack | node | An event loop holds each open connection for a few KB, where the default Stack holds a worker for each. |
| Realtime Hub | Clustering | On | Without it one box takes every connection. |
Research this Design needs: Full-text search, Change data capture, Event streaming, Background jobs, Checkout, Hosted models, WebSockets, Object storage, WebSockets, Message queues, Full-text search, Event streaming, Change data capture.
Link the Database into a CDC Connector, the connector into an Event Stream, and the stream into a Worker. The connector has no options of its own. It copies the writes the Database processes and never slows them. The stream's key and partitions set order and read speed.
The Search Engine in the picture sits beside the Database, and the other extra Nodes only serve the remaining Series B request kinds. Research Change data capture, which needs Event streaming and Data warehousing. The same effect without a stream is the transactional outbox. The glossary defines CDC.