Message Queue, Publish–Subscribe and Event Log

Objectives

By the end of the lesson, you can distinguish the work queue, the publish–subscribe model and the durable event log by delivery and retention, and characterise them along the seven aspects.

Why is it needed?

In asynchronous service cooperation the sender emits a message and the consumer processes it later. The broker between them organises transfer, storage and delivery. There are three basic models, and they differ in delivery and retention, not in product name.

Message queue (work queue)

According to Azure's "competing consumers" pattern, the application puts requests into a queue as messages and several consumer instances take them out; each message is received by one consumer. The queue thereby also works as a buffer between variable load and processing (queue-based load levelling), and messages are delivered at least once.

Publish–subscribe

According to Azure's distinction, with competing consumers one consumer receives a message, whereas in the publish–subscribe model every subscribed consumer receives every message. So one fact (for example "an invoice was issued") can be received by several independent consumers, each producing the result belonging to its own responsibility. A subscription has its own processing state; the instances of the same subscription typically share the work, while another subscription independently receives the same fact.

flowchart LR
  P["Producer"] --> T["Topic"]
  T --> S1["Subscription 1"]
  T --> S2["Subscription 2"]
  S1 --> W1["Consumer A (several instances)"]
  S2 --> W2["Consumer B"]

Durable event log

According to the Kafka introduction, an event records that "something happened"; producers and consumers are completely separate. A topic serves many producers and many subscribers, is divided into partitions, and events with the same key land in the same partition; a consumer reads the events of a partition exactly in the order they were written. An event is not deleted after reading: retention can be set per topic, and old events are deleted when the time has passed. The log can thus be re-read by the consumer, which follows its own position; several consumer groups can progress independently on the same data stream.

Do not interpret the difference between queue and log by product name: delivery model, retention, position handling and replay are what matter. One infrastructure can support several patterns.

Comparison

Aspect	Work queue	Pub/sub	Event log
Direction	One-way: producer → consumer	One-way: producer → many	One-way: producer → many, re-readable
Initiator	The producer sends; the consumer takes	The producer	The producer writes; the consumer reads at its own pace
Connection persistence	No direct connection; the broker is in between	Same	Same
Latency	Depends on queue length and consumers	The slowest consumer matters for processing	By position; reading can be late
Load	The queue buffers; its length can grow	Processing per subscription	Retention needs storage
Fault tolerance	An unacknowledged message is redelivered	Own state per subscription	Can be resumed from the consumer's position
When it fits	Distributing work, load levelling	Several independent reactions to one fact	Replay, a new consumer catching up later, rebuilding a view
When it does not	If every consumer needs the message	If an immediate answer is needed by the caller	If retention is superfluous or costly

The event contract

An event should have a stable type, identifier, schema version and business meaning. A timestamp does not guarantee a global order. The fact may contain enough data for the consumer, or only an identifier that needs a query (4.4). An event is not a public copy of the internal database.

Common misconceptions

Statement	Clarification
"A queue and pub/sub are the same."	One delivers to one consumer, the other to every subscriber.
"In a message queue every message is processed exactly once."	At-least-once delivery is typical (8.2).
"The log is deleted after reading."	It stays for the retention time (Kafka).
"A message queue has infinite capacity."	If arrival persistently exceeds processing, the queue grows (8.4).

In short

• In a work queue one consumer receives a message; in pub/sub every subscriber does.
• A durable log keeps events; the consumer follows its own position.
• The model is chosen by the need for retention, delivery and replay.
• The event contract is a shared dependency.

Review questions

1. What is the difference between competing consumers and pub/sub?
2. How does the durable log differ from the work queue in retention?
3. When would you choose a log over a queue?

Learning resources

• Competing Consumers pattern (Azure Architecture Center)
• Queue-Based Load Leveling pattern (Azure Architecture Center)
• Apache Kafka: Introduction — event, topic, partition, retention.