Stream changes with CDC
Read the Database's write log and publish each change to a stream, so search and other copies follow the data without a second write.
Publish every change from the write log to an Event Stream, and let consumers build their copies from it.
Move up when#
The app writes to the Database and the search index in one request, and the two drift apart.
A warehouse, a cache, or another service needs to follow the data.
Stale search results appear after a crash between the two writes.
The Design#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | small | It writes once, to the Database, and never to the index. |
| Database | Instance size | small | The source of truth. The connector reads its write log, so it adds no load to a write. |
| Event Stream | Keyed by entity | On | One row keeps its changes in one partition, so the index never applies an old change after a newer one. |
| Worker | Instance size | small | It reads every change, so it needs room for the write rate. |
Research this Design needs: Full-text search, Change data capture, Event streaming, Background jobs, Full-text search, Background jobs, Message queues, Event streaming, Data warehousing, Change data capture.
The App Server writes once, to the Database. The CDC Connector reads the Database's log and publishes each change to the Event Stream. A Worker reads the stream, which in a real system is where the search index gets updated. The Search Engine here only answers the App Server's queries, because a Worker in the game cannot link to it. The stream is Keyed by entity, so a row's changes stay in one partition and arrive in order.
Why it works#
The app never writes the same fact twice, so a crash cannot leave the copies disagreeing for good. The connector reads the log, so it adds no work to a write. The stream keeps its messages, so a new index can replay them, and a second consumer can read them too.
Watch consumer lag, not the connector. A slow index shows up as growing lag, and the fix is in the consumer or the stream.
What it costs#
You add a connector, a stream, and a Worker, and each has its own Burn and failure modes. The copy is eventually consistent, so a search can miss the newest row for a moment. A consumer further behind than the retention window loses the oldest changes. An unkeyed stream with several partitions can apply an old change after a newer one.
In Quorum#
The CDC Connector needs Change data capture, which follows Event streaming and Data warehousing. Place it beside the Event Stream. The pattern is Change data capture, and the older option is the Transactional outbox. The glossary has CDC. The next rung moves reports off the primary: Reports off the primary, in a warehouse.