What is a message queue and why would you use one?

Think of a busy restaurant. The waiter takes your order, writes it on a slip and pins it on the kitchen rail. Then the waiter goes to the next table. The cooks pick slips from the rail one by one at their own pace.

The rail is the message queue. The waiter (producer) does not wait for the food to be cooked. The cooks (consumers) are not disturbed by how fast orders arrive. If lunch rush brings fifty orders in five minutes, slips pile up on the rail and get cooked steadily, instead of the kitchen collapsing. If one cook leaves, the slips are still there for another cook.

A placement portal where a student submits an application:

Without a queue (synchronous):
  POST /apply
    save application          50 ms
    generate PDF summary    2000 ms
    send email to student    800 ms
    notify recruiter         500 ms
  -> user waits about 3.3 s; if email service is down, the request fails

With a queue:
  POST /apply
    save application          50 ms
    publish "ApplicationSubmitted" to queue
  -> user gets "Applied!" in about 60 ms

  Workers read the message:
    pdf-worker    -> generates PDF
    email-worker  -> sends email (retries if the email service is down)
    notify-worker -> alerts recruiter

• Producer sends a message to the queue.
• Broker stores it durably until it is processed.
• Consumer receives the message, does the work and acknowledges it. Only then is it removed.
• If the consumer crashes before acknowledging, the message becomes visible again and another consumer retries it.
• After too many failed attempts, the message moves to a dead-letter queue (DLQ) so a human can inspect it instead of it blocking the line.

Two common styles

• Work queue (point-to-point): each message is handled by one consumer out of many. Good for jobs like sending emails. Adding consumers increases throughput.
• Publish-subscribe: each message goes to every subscribed service. One "OrderPlaced" event can trigger inventory, email and analytics separately.

Queue vs log. RabbitMQ and SQS are classic queues: messages are removed once acknowledged. Kafka is a distributed log: messages are kept for a retention period, consumers track their own position (offset), and several consumer groups can read the same stream independently or replay it.

Delivery guarantees. Most systems give at-least-once delivery, so a message can arrive twice. Consumers must therefore be idempotent: processing the same message twice should not charge a card twice.

• During a Big Billion Days sale, order placement writes to a queue, and downstream services (invoice, warehouse, SMS) process at their own pace.
• Payment apps send transaction events to Kafka so fraud detection, rewards and analytics consume them independently.
• Video platforms queue uploads for transcoding, which takes minutes.
• OTP and notification systems use queues so a slow SMS provider does not slow down login.

• Assuming each message is delivered exactly once and writing non-idempotent consumers.
• Expecting strict global order from a queue that does not guarantee it. Ordering is usually per partition or per message group.
• No dead-letter queue, so one bad message is retried forever.
• Not watching queue length. A growing backlog means consumers are too slow, and users will notice delays.
• Using a queue for something the user must see immediately, like checking a password.

• How do you make a consumer idempotent?
• What is the difference between Kafka and RabbitMQ?
• How do you keep messages in order?
• What is backpressure, and how does a queue help with it?
• How would you know if consumers are falling behind?