By the end of the lesson, you can explain what a partition key guarantees and what it does not, how to handle late and out-of-order messages, and when it is safe to replay dead-letter messages.
Why is it needed?
Asynchronous messages can be delayed, repeated, and because of parallel consumption processed in a different order. The producer's emission order, the broker's storage order and the order of business effects are three separate concepts.
Ordering and partitions
According to Azure's competing consumers pattern, the order in which consumers receive messages is not guaranteed and does not necessarily reflect the order in which they were created; the recommendation is therefore idempotent processing, which removes the dependence on order. According to the Kafka introduction, however, events with the same key land in the same partition, and a consumer reads the events of a partition exactly in the order they were written. Some services (for example the message sessions of Azure Service Bus) can provide strict FIFO (first in, first out) order.
It follows that the order of the events of one entity (for example an order) can be made traceable with a partition key, but between separate partitions a common total order generally cannot be assumed. The choice of key also affects load: if every message gets the same key, one partition becomes a bottleneck; if it gets a random key, the order belonging to the same entity can be lost.
Version and late event
An event of an entity can carry an increasing version number. If the read model has already applied version 12, it cannot treat a later-arriving 11 as fresh state. With a delta event (the description of a change) a missing predecessor is significant: the intermediate operation cannot be skipped the way it can with a full snapshot. A timestamp does not give a reliable global order, because clocks can differ and messages travel by separate routes. If order is needed, the identifier, the version and the contract of the source should be the primary support.
Retrying by cause
A transient database access error can go away later; an invalid schema or a missing required field does not improve by repeating the same content. The consumer should separate transient, permanent and business rejection. Delayed retry and backoff (6.5) protect the failed dependency, but there should be an attempt or time limit. One failing message should not block the processing of all the others without limit, unless business order justifies it.
Dead-letter queue
The dead-letter queue (DLQ) serves to set aside failed or unhandleable messages. According to the Azure description, a poison message (malformed format, unavailable resource) should not return to the queue again and again; its details should be put elsewhere for analysis; Service Bus, for example, automatically moves a message to the dead-letter queue after the delivery attempts exceed a limit.
It is not a solution on its own: it needs the cause, the original identifier, attempt information and a replay procedure; the depth of the DLQ should be monitored so that the fault can be investigated and the message resubmitted where appropriate. Replay can repeat earlier effects, so idempotency is needed. The state "ended up in the DLQ" of a business-critical message is a user-visible or operational consequence, not a successful completion.
Characteristics
Aspect
Ordering and error handling
Direction
From the producer towards the consumer
Initiator
The producer / the broker redelivers
Latency
The backoff of retries increases it
Load
Retries can multiply it
Fault tolerance
DLQ, retry limit, idempotency
When it fits
Ordering within an entity; isolating failures
When it does not
A need for global ordering (expensive); endless retries
Common misconceptions
Statement
Clarification
"A partition key gives global order."
Only within a partition, for the same key.
"A timestamp is enough for ordering."
Clocks may differ; a version or identifier is needed.
"Every error should be retried."
For a permanent or business error retrying is pointless.
"A message in the DLQ is handled."
It is only set aside; its effect has not been carried out.
In short
Ordering can be designed per entity, with a partition key; global order is rarely needed.
The version helps recognise late and missing events.
Retry by cause, with a limit and backoff.
The DLQ serves isolation; replay requires idempotency.
Review questions
What order does a partition key guarantee, and what does it not?
Why is it dangerous to handle every error with unlimited retry?
Under what condition is replaying dead-letter messages safe?
Ordering, Retries and Dead-Letter Handling
Objectives
By the end of the lesson, you can explain what a partition key guarantees and what it does not, how to handle late and out-of-order messages, and when it is safe to replay dead-letter messages.
Why is it needed?
Asynchronous messages can be delayed, repeated, and because of parallel consumption processed in a different order. The producer's emission order, the broker's storage order and the order of business effects are three separate concepts.
Ordering and partitions
According to Azure's competing consumers pattern, the order in which consumers receive messages is not guaranteed and does not necessarily reflect the order in which they were created; the recommendation is therefore idempotent processing, which removes the dependence on order. According to the Kafka introduction, however, events with the same key land in the same partition, and a consumer reads the events of a partition exactly in the order they were written. Some services (for example the message sessions of Azure Service Bus) can provide strict FIFO (first in, first out) order.
It follows that the order of the events of one entity (for example an order) can be made traceable with a partition key, but between separate partitions a common total order generally cannot be assumed. The choice of key also affects load: if every message gets the same key, one partition becomes a bottleneck; if it gets a random key, the order belonging to the same entity can be lost.
Version and late event
An event of an entity can carry an increasing version number. If the read model has already applied version 12, it cannot treat a later-arriving 11 as fresh state. With a delta event (the description of a change) a missing predecessor is significant: the intermediate operation cannot be skipped the way it can with a full snapshot. A timestamp does not give a reliable global order, because clocks can differ and messages travel by separate routes. If order is needed, the identifier, the version and the contract of the source should be the primary support.
Retrying by cause
A transient database access error can go away later; an invalid schema or a missing required field does not improve by repeating the same content. The consumer should separate transient, permanent and business rejection. Delayed retry and backoff (6.5) protect the failed dependency, but there should be an attempt or time limit. One failing message should not block the processing of all the others without limit, unless business order justifies it.
Dead-letter queue
The dead-letter queue (DLQ) serves to set aside failed or unhandleable messages. According to the Azure description, a poison message (malformed format, unavailable resource) should not return to the queue again and again; its details should be put elsewhere for analysis; Service Bus, for example, automatically moves a message to the dead-letter queue after the delivery attempts exceed a limit.
It is not a solution on its own: it needs the cause, the original identifier, attempt information and a replay procedure; the depth of the DLQ should be monitored so that the fault can be investigated and the message resubmitted where appropriate. Replay can repeat earlier effects, so idempotency is needed. The state "ended up in the DLQ" of a business-critical message is a user-visible or operational consequence, not a successful completion.
Characteristics
Common misconceptions
In short
Review questions
Learning resources