Designing a Distributed Message Queue Architecture

Two clients were each told the same chair was theirs, and nothing in the system had failed.

It started with a cancellation. A client dropped her 2:10 colour appointment and the event went onto a queue. One consumer was slow to acknowledge it, so the broker assumed that consumer had died and handed the same message to a second. Each consumer picked someone from the waitlist, and both clients said yes. The queue had done exactly what it promised: deliver the message at least once.

That is what most failures in a distributed message queue architecture look like: a duplicate, two events handled out of order, or a backlog nobody was watching. The scheduling side, filling a cancelled slot without double-booking, is hard enough alone. The queue underneath adds more ways to get it wrong.

This blog covers the decisions that prevent those failures, with numbers you can check against your own workload, then replays the 2:10 incident with a fixed design. Any system where software acts on a live calendar, an AI receptionist included, ends up with queues between the event and the action.

What a distributed message queue actually is

A distributed message queue architecture is a cluster of brokers that stores messages durably, splits each topic into partitions so many consumers can work in parallel, and copies every partition to several nodes so one machine failing loses nothing. Producers pick a partition with a key, consumers track progress with offsets, and the ack and retry rules decide whether messages can be lost or duplicated.

For example, using customer_id as the key can keep all events for one customer in the same partition:

Order Created → Payment Received → Order Shipped

At the same time, events for other customers can be processed in parallel.

Each partition is also copied to multiple brokers through replication. If one broker fails, another replica can take over. How safely a message is stored depends on the acknowledgement policy. Waiting for more replicas improves durability but can add latency.

Consumers track their position using offsets. If a consumer crashes, another consumer can resume from the last committed offset. The timing of that commit affects delivery guarantees.

• At-most-once: messages may be lost, but are less likely to be processed twice.

• At-least-once: messages are retried, so duplicates are possible.

• Exactly-once: extra coordination is used to prevent duplicate effects.

Retries add another layer. Failed messages may be delivered again or moved to a dead-letter queue after repeated failures.

The main idea is simple: partitions provide parallelism, replicas provide fault tolerance, keys control ordering, and offsets track progress. The ack policy and retry rules determine what happens when something fails.

Start with four numbers, not a broker name

Choosing Kafka, RabbitMQ, or SQS first is backwards. Answer these instead:

1. Throughput in MB/s, not messages per second. Bytes decide disk, network, and partition count.

2. Retention. Do consumers need to replay last Tuesday after a bug fix, or does a message die once it is handled?

3. Ordering scope. Ordered relative to what? Everything, one customer, one calendar?

4. Loss tolerance. Which messages cost money or trust if one disappears?

Here is how the first two become a cluster size. At 50 MB/s with seven days of retention, the cluster holds about 30 TB (4.3 TB a day), and about 91 TB once every partition is replicated three times.

Partitions come from a rule of thumb that Confluent has long published: you need at least max(t/p, t/c) partitions, where t is target throughput, p is what one partition absorbs from producers, and c is what one consumer can process. Producers typically reach tens of MB/s on a single partition. Consumer speed depends entirely on your handler.

Take p = 10 MB/s and c = 5 MB/s as an illustration. At 50 MB/s that gives 10 partitions. Then size for where traffic will be in a year or two, which is Confluent's advice too, not for today. If 50 MB/s doubles, the answer is 20. Measure your own p and c before trusting either figure.

Now run the exercise on a workload shaped like a booking platform. Say 200 locations produce 600 events a day each. That is 120,000 events a day, about 1.4 per second on average. A Saturday-morning peak at 20 times the average is still under 30 per second.

That is far inside what a managed queue handles. An SQS FIFO queue in its default mode accepts 300 API calls per second per action, ten times that peak, and 3,000 messages per second when batched. A Kafka cluster here would be an operations bill with nothing on the other side of the ledger. Run the arithmetic before the architecture meeting, not after.

The building blocks, in one picture

Producer --key--> Topic "booking-events"
                    |- Partition 0  (leader: broker 1, replicas: 2, 3)
                    |- Partition 1  (leader: broker 2, replicas: 1, 3)
                    '- Partition 2  (leader: broker 3, replicas: 1, 2)
                                |
          Consumer group "slot-fill":   one consumer per partition
          Consumer group "reporting":   own offsets, sees every message

Two rules do most of the work.

Inside one consumer group, each partition belongs to exactly one consumer. That keeps order within the partition and spreads load across the group. It also caps parallelism: 12 consumers reading a 4-partition topic means 4 working and 8 idle.

Different groups are independent. Each sees every message and keeps its own position. That is publish/subscribe fan-out, and it is why one inbound customer message can feed a reply, a calendar write, a staff notification, and a log without those jobs knowing about each other. The same shape shows up in one inbox for voice, SMS, and web, and in omnichannel AI agent platforms such as YourGPT, where a message on any channel has to reach replies, logging, and handoff.

Why the log is append-only

The storage model matches the workload. Messages are added at the end, read in order, and removed later through retention. Existing records are rarely changed, so a relational table would pay for updates, indexes, and random lookups that the broker barely uses.

An append-only log keeps the write path simple. Each partition is split into segment files, and only the newest segment accepts writes. Older segments become immutable, which makes replication, recovery, and retention easier. When data expires, the broker can delete an entire segment instead of removing messages one by one.

Sequential reads also work well with the operating system's page cache. Producers write near the tail, while consumers usually read nearby offsets in order, so frequently accessed data can stay in memory without a separate cache for every message.

Offsets fit naturally into this model too. A consumer tracks its position in the log and can resume from that offset after a restart without changing the stored messages.

Batching adds another trade-off. Larger batches reduce network and disk overhead and improve throughput, but they take longer to fill. Smaller batches reduce latency but increase overhead. Since no setting gives both maximum throughput and minimum latency, batching is exposed as a tuning knob.

Pick the partition key like a schema decision

Ordering exists only inside a partition. Across partitions there is no global order, and building one would defeat the point of partitioning. SQS FIFO works the same way: order holds within a message group ID, and different groups can be processed out of order relative to each other.

So the key is the ordering promise. Everything that must stay in sequence needs the same key.

For booking events, key by the resource being contended, not the person. Book 2:00, cancel 2:00, fill 2:00 for Chair 4 must be processed in that order. Keying by customer ID would spread those three events across partitions and let the fill run before the cancel.

Location is the natural first cut, since each location keeps its own calendar, hours, and staff. It is also where mistakes hide. A free room at one site is not an open slot at another, which is exactly the two-location scheduling trap that double-books a clinician. A key that mixes locations puts unrelated calendars in one ordered lane.

Too coarse a key creates the opposite problem: a hot partition. One busy location with 40% of traffic pins one consumer while the rest sit idle. If that happens, move down to the chair or provider.

Changing the partition count changes the mapping

Kafka maps a key to a partition by hashing it, so messages with the same key keep landing together. Add partitions and that mapping shifts. Messages already written stay where they are, new messages for the same key may land elsewhere, and ordering for that key breaks across the change. Confluent's guidance is to over-partition a little up front for exactly this reason.

If you cannot, drain the topic before resizing. The same arithmetic punishes anyone sharding across several independent queues with plain modulo: going from 4 shards to 5 remaps about 80% of keys, since only 1 in 5 hash values keeps the same remainder under both.

A hash ring avoids that. Place shards on a circle, hash each key onto the same circle, and send it to the next shard clockwise.

RabbitMQ ships a consistent-hash exchange plugin for spreading keys across several queues this way.

Replication: decide what "acknowledged" means

Disks fail and machines die, so each partition gets replicas, typically three, on different nodes. One replica leads. Writes go to the leader, and followers pull from it the way a consumer would. Only committed records are visible to consumers, which stops anyone reading a record that a failover would erase.

"Committed" means the in-sync replicas have it: the followers currently keeping up with the leader. Waiting for every replica lets one slow disk stall the partition. Waiting for none loses acknowledged data on failover. The in-sync set sits between the two failures.

The producer picks its position on that spectrum:

Someone also has to decide which replica leads when a broker dies, and two nodes must never both believe they lead. Kafka hands that to a controller quorum. As of version 4.0, ZooKeeper support is gone entirely and only the built-in KRaft mode remains, so clusters still on ZooKeeper must migrate before they can upgrade.

RabbitMQ gets there another way. Quorum queues use the Raft consensus algorithm, and a publisher confirm is issued only after a message reaches a majority of members. Three nodes tolerate one failure, five tolerate two, and the docs advise against more than seven because throughput falls as members are added.

Two warnings from that documentation matter. Messages published without confirms get no safety guarantee at all, however many replicas exist. And classic mirrored queues were removed in RabbitMQ 4.0, so any tutorial recommending them is out of date.

Consumers: pull, commit after the work, accept the rebalance

Consumers pull. A pushing broker has to model every client's capacity, and a fast producer can bury a slow consumer. With pull, a backlogged consumer falls behind instead of falling over, and each fetch returns a natural batch. Long polling fixes the one downside, spinning on an empty topic, by letting the fetch wait on the server for new data.

Progress is one number per group and partition: the committed offset, which marks where reading resumes. A replacement consumer reads that number and picks up from there.

Membership changes trigger a rebalance: a consumer joins, leaves, or misses heartbeats, and a crash looks identical to a slow pause. The group stops, partitions are reassigned, and reading resumes from the committed offsets. Nothing is consumed meanwhile, so a heartbeat timeout that fires on a garbage-collection stall is an availability problem.

SQS has its own version. In a FIFO queue, once a consumer receives messages from a group, nothing more from that group is delivered until they are deleted or the visibility timeout expires. One stuck message blocks its group and no others. That is useful isolation, and it means the visibility timeout must be longer than the slowest handler. It can be set as high as 12 hours.

Delivery semantics are a commit-order decision

The broker does not hand you a guarantee. You get one from three choices: the producer's ack level, its retry behaviour, and when the consumer commits relative to doing the work.

In Kafka, producer idempotence is on by default. It requires acks=all and no more than five in-flight requests per connection, and within that limit ordering holds. Older tutorials set in-flight requests to 1 for strict order, which costs throughput for no gain once idempotence is on.

The third row is narrower than its name. A text message to a customer is an external side effect, and no broker transaction reaches it. For that, the practical design is at-least-once delivery plus a consumer that treats a repeat as a no-op.

Every action an agent takes on a calendar, checking availability, booking, and filling from the waitlist, is a write that can be replayed. Give each event a stable ID, and record it in the same transaction as the state change:

sql

INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;
-- 0 rows inserted: a duplicate. Skip the booking and the text.

If the text provider accepts an idempotency key, pass the same event ID there.

Retries, dead letters, and the point where a person takes over

Retries need three things: a delay, a cap, and a destination for what exhausts the cap.

Delay first. Immediate retries turn one failing dependency into a storm. RabbitMQ 4.3 adds delayed retry for quorum queues with linear back-off: the delay is the minimum delay times the delivery count, capped at a maximum. With a 1-second minimum and a 30-second cap, retries wait 1, 2, 3 seconds and so on until they hit 30.

The cap matters because of poison messages, the ones that fail every time. Since RabbitMQ 4.0, quorum queues default to a delivery limit of 20. After that a message is dropped, or dead-lettered if a dead-letter exchange is configured. Configure the exchange. Dropping silently is the worst of the options.

Then the destination. A dead-letter queue is only useful if somebody owns it. Alert when its depth goes above zero, and keep the original payload, the error, and the attempt count together. It is the machine version of handing off to a person: the system admits it cannot decide and preserves the full context for whoever can.

Bound the backlog as well. Set a maximum queue length, and treat it as approximate: RabbitMQ's reject-publish overflow can overshoot by the handful of messages already in flight.

Kafka, RabbitMQ, or SQS: choose by workload

Past SQS's five-minute window, a retry with the same ID is a new message. RabbitMQ's own documentation points very long backlogs and large fan-outs toward streams instead.

The decision rule is short:

• Need replay, several independent consumers, or volume in MB/s? Kafka.

• Need routing rules, priorities, and per-message handling at modest volume? RabbitMQ quorum queues.

• On AWS, small team, per-key order at modest volume? SQS FIFO.

Plenty of production systems run two of these. That is not a failure of planning.

Watch lag in seconds, not only uptime

A queue can be fully up and still be failing. Track these:

• Age of the oldest unprocessed message. Depth tells you size. Age tells you harm.

• Consumer lag per partition, to spot the hot key.

• Redelivery rate. A rising number means a handler is timing out or crashing.

• Dead-letter depth.

• Publish-confirm latency at p99. Replication trouble shows up here first.

Age is the one that maps to real cost. A no-show recovery agent works against a deadline, so a slot filled ten minutes late is already lost.

Then test under pressure. A quiet transcript proves little, so look for double-book risk on a busy Saturday rather than a quiet Tuesday. Kill a consumer mid-batch. Redeliver every message twice. Delay one partition by a minute. Then watch an agent run from trigger to result and confirm the outcome is still correct.

The 2:10 incident, replayed with a better design

Here is the opening scenario again, with each control in place. This is a sketch of the pattern, not any vendor's implementation.

1. The event is keyed and confirmed. The cancellation for Chair 4 is published with a key for that chair and waits for a broker confirm. It cannot be lost, and it cannot overtake the booking event that came before it.

2. The timeout outlasts the handler. The text provider is slow and the handler takes 40 seconds. The visibility or ack timeout is set to five minutes, so the job is not handed to a second consumer while the first is still working.

3. A replay does nothing. Suppose the consumer sends the text and then crashes before acknowledging. The message comes back, the event ID is already in processed_events, and the second attempt skips.

4. Two customers can still race. Two waitlisted clients get offers for the same slot, and both say yes. These are different events with different IDs, so the duplicate check cannot help. The calendar write has to settle it:

sql

UPDATE slots
SET booked_by = $1
WHERE slot_id = $2 AND booked_by IS NULL;
-- 1 row updated: the slot is yours.
-- 0 rows: someone else got it. Offer the next person on the waitlist.

5. Repeated failure reaches a person. If a job fails three times, it lands in the dead-letter queue and the front desk receives the full thread.

The takeaway sits in step four. Idempotency stops the same message acting twice. Only a conditional write stops two different messages claiming the same slot.

Five questions to ask any vendor whose agent writes to your calendar

What to test on an AI receptionist demo covers the booking side. These five cover the plumbing:

Vague answers to these tend to show up later as double bookings.

Design for the duplicate

Every queue eventually delivers a message twice, in the wrong order, or late. Good design means deciding in advance what happens when it does.

Before anything ships, write three sentences for each topic: which key orders it, which ack level protects it, and what happens to a message that keeps failing. If a sentence takes a meeting to agree on, that is the part still missing.

Bring one real booking, one ugly edge case, and the five questions above to a 30-minute demo and see where the duplicates would land.

Final verdict

A RabbitMQ quorum queue or an SQS FIFO queue is the right starting point for most appointment-scale systems. Reach for Kafka when replay, many readers of one stream, or heavy sustained throughput becomes a real requirement.

The broker matters less than the decisions on top of it: a key that follows the thing whose sequence matters, an ack level that matches what a lost message costs, and a consumer that survives seeing the same message twice. Add a conditional write wherever two different messages can claim the same resource, and watch the age of the oldest waiting message.

A team that gets those right will run well on any of the three brokers. A team that skips them will double-book on all of them.