How do you make a message consumer idempotent?
Record the id of each processed message in the same transaction as its effect, and skip ids already seen. Or design the operation to be naturally repeatable, such as setting a value rather than incrementing it.
Explain it simply
Brokers deliver at least once, so a consumer will sometimes see the same message twice. If the message says "add 500 to wallet", processing it twice gives the wrong balance.
Two approaches:
- Deduplicate by id. Each message carries a unique id. Before applying it, check a processed_messages table. In one transaction, apply the change and insert the id. A unique constraint on the id makes a duplicate fail safely.
- Make the operation naturally idempotent. "Set order 981 status to SHIPPED" can be repeated harmlessly. "Set wallet balance to 1500" is repeatable but risky with out-of-order messages; adding a version number lets you ignore older ones.
Important: the check and the effect must be atomic. If you apply the change and then crash before recording the id, a redelivery applies it again.
Dedup records can be cleaned up after the broker's maximum redelivery window.
Example
BEGIN;
INSERT INTO processed (msg_id) VALUES ('m-8812'); -- fails if already there
UPDATE wallet SET balance = balance + 500 WHERE user_id = 12;
COMMIT;
-- Duplicate delivery: INSERT violates the unique key, whole transaction rolls back.What to say in the interview
"Because delivery is at least once, I make handlers idempotent: store the message id with the effect in one transaction, relying on a unique constraint, or use repeat-safe operations like set-status. Atomicity between check and effect is the key point."