QuorumWiki
Jugar
Patrones

Transmite los cambios con CDC

Lee el log de escritura de la Base de datos y publica cada cambio en un flujo, para que la búsqueda y otras copias sigan a los datos sin una segunda escritura.

Publish every change from the write log to an Event Stream, and let consumers build their copies from it.

Move up when#

  • The app writes to the Database and the search index in one request, and the two drift apart.

  • A warehouse, a cache, or another service needs to follow the data.

  • Stale search results appear after a crash between the two writes.

The Design#

Drag to pan. Hold Ctrl or Cmd and scroll to zoom.

At 3.00k Users the Simulation solves this Design to Apdex 1.00, p99 262 ms, 0.0% errors and $18/s Burn.
WhereSettingValueWhy
App ServerInstance sizesmallIt writes once, to the Database, and never to the index.
DatabaseInstance sizesmallThe source of truth. The connector reads its write log, so it adds no load to a write.
Event StreamKeyed by entityOnOne row keeps its changes in one partition, so the index never applies an old change after a newer one.

Research this Design needs: Full-text search, Change data capture, Event streaming, Full-text search, Background jobs, Message queues, Event streaming, Data warehousing, Change data capture.

The App Server writes once, to the Database. The CDC Connector reads the Database's log and publishes each change to the Event Stream. The stream feeds the Search Engine, which applies each change to its index while it keeps answering the App Server's queries. The stream is Keyed by entity, so a row's changes stay in one partition and arrive in order.

Why it works#

The app never writes the same fact twice, so a crash cannot leave the copies disagreeing for good. The connector reads the log, so it adds no work to a write. The stream keeps its messages, so a new index can replay them, and a second consumer can read them too.

Watch consumer lag, not the connector. A slow index shows up as growing lag, and the fix is in the consumer or the stream.

What it costs#

You add a connector and a stream, and each has its own Burn and failure modes. The copy is eventually consistent, so a search can miss the newest row for a moment. A consumer further behind than the retention window loses the oldest changes. An unkeyed stream with several partitions can apply an old change after a newer one.

In Quorum#

The CDC Connector needs Change data capture, which follows Event streaming and Data warehousing. Place it beside the Event Stream. The pattern is Change data capture, and the older option is the Transactional outbox. The glossary has CDC. The next step moves reports off the primary: Reports off the primary, in a warehouse.