What is the transactional outbox pattern and what problem does it solve?
It solves the dual-write problem: updating a database and publishing an event can't be atomic. The outbox writes the event into a table in the same database transaction, and a separate process publishes it to the broker.
Explain it simply
A service saves an order in its database and then publishes "OrderPlaced" to Kafka. If it crashes between the two steps, the order exists but nobody hears about it. If it publishes first and the database write fails, others act on an order that does not exist. A database and a broker cannot share one transaction.
Outbox pattern:
- In one database transaction, insert the order row and an event row into an outbox table. Both commit or neither does.
- A relay process reads unsent outbox rows and publishes them to the broker, marking them sent. Alternatively, change data capture reads the database log and publishes it.
- If the relay crashes after publishing but before marking, it republishes, so consumers must be idempotent (at-least-once).
Result: no lost events and no phantom events, with the broker eventually consistent with the database.
Example
BEGIN;
INSERT INTO orders (id, user_id, total) VALUES (981, 12, 499);
INSERT INTO outbox (event_id, type, payload)
VALUES ('e-5501', 'OrderPlaced', '{"order_id":981}');
COMMIT;
-- relay later: publish e-5501 to the broker, then mark it sentWhat to say in the interview
"The outbox pattern makes the state change and the event a single local transaction, then publishes the event asynchronously via a relay or CDC. It guarantees no lost events, at the cost of at-least-once duplicates, so consumers need to be idempotent."