By the end of the lesson, you can explain the dual-write problem, the role of the outbox and the inbox, and say what they close and what they leave open.
Why is it needed?
A common situation is that a service saves data and sends a message. If the two happen in separate systems, the process can fail between them. A try–catch block does not turn a database and a message broker into a shared transaction.
The dual-write problem
According to the description of microservices.io, a service must update the database and publish a message atomically, and a two-phase commit is generally not available. If the save comes first and the send second, a failure after the save may leave the event missing. If the send comes first, the consumer may learn about something that never came into being.
Transactional outbox
The solution: the message is saved in the same local database transaction as the business data, into an outbox table (or collection). A separate relay then reads the outbox and publishes it on the broker. The relay can be implemented by querying (polling publisher) or by following the database's transaction log (transaction log tailing).
sequenceDiagram
participant S as Service
participant D as Own database
participant R as Relay
participant B as Broker
S->>D: BEGIN
S->>D: Business data + outbox record
S->>D: COMMIT
R->>D: Records to send
R->>B: Publish message
R->>D: Record the sending
The relay may fail after publishing but before recording, so the same message can go out twice. The outbox therefore does not eliminate duplication; the consumer must be idempotent (6.5, 8.2). Among the failure possibilities microservices.io also mentions that the developer may forget to write the entry.
Several relays and ordering
Several relays must coordinate who picks up which record (locking, lease, per-partition assignment), otherwise they work in parallel needlessly; after a failure, the work must be picked up again. The outbox table alone gives no ordering guarantee: two events of one entity may be sent in reverse order by two relays (8.3).
Inbox at the receiver
The inbox stores the identifier of the received message. The consumer records, in one transaction with the business modification, that it has applied the message; a unique key protects against parallel duplication. The acknowledgement sent to the broker follows. This way a redelivered message can be recognised, and the effect is not repeated.
For an external API call, the inbox and a local transaction cannot atomically include the remote effect. In that case the idempotency key of the external API, a status query or reconciliation is needed (6.5).
Operational consequences
The number and age of unsent records must be monitored. A long-waiting event means a business delay. Cleaning the table and retention are to be weighed: deletion should not make troubleshooting or replay impossible.
Common misconceptions
Statement
Clarification
"The outbox sends exactly once."
It sends at least once; duplication is possible.
"The outbox gives global ordering."
No; ordering must be handled separately.
"The inbox solves the duplication of an external payment."
For an external effect the idempotency of the external system is also needed.
"Try–catch is enough between the two steps."
It does not make the two systems one transaction.
In short
The dual-write problem: saving data and sending a message are not atomic across two systems.
The outbox saves the message with the business data in one local transaction; the relay sends it later.
The outbox gives at-least-once delivery; the consumer must be idempotent.
The inbox records the processed identifier in one transaction with the business modification.
Review questions
Which failure flow does the outbox fix, and which not?
Where can duplication arise in the operation of the relay?
Why is an inbox alone not enough for an exactly-once bank charge?
Transactional Outbox and Inbox
Objectives
By the end of the lesson, you can explain the dual-write problem, the role of the outbox and the inbox, and say what they close and what they leave open.
Why is it needed?
A common situation is that a service saves data and sends a message. If the two happen in separate systems, the process can fail between them. A try–catch block does not turn a database and a message broker into a shared transaction.
The dual-write problem
According to the description of microservices.io, a service must update the database and publish a message atomically, and a two-phase commit is generally not available. If the save comes first and the send second, a failure after the save may leave the event missing. If the send comes first, the consumer may learn about something that never came into being.
Transactional outbox
The solution: the message is saved in the same local database transaction as the business data, into an outbox table (or collection). A separate relay then reads the outbox and publishes it on the broker. The relay can be implemented by querying (polling publisher) or by following the database's transaction log (transaction log tailing).
The relay may fail after publishing but before recording, so the same message can go out twice. The outbox therefore does not eliminate duplication; the consumer must be idempotent (6.5, 8.2). Among the failure possibilities microservices.io also mentions that the developer may forget to write the entry.
Several relays and ordering
Several relays must coordinate who picks up which record (locking, lease, per-partition assignment), otherwise they work in parallel needlessly; after a failure, the work must be picked up again. The outbox table alone gives no ordering guarantee: two events of one entity may be sent in reverse order by two relays (8.3).
Inbox at the receiver
The inbox stores the identifier of the received message. The consumer records, in one transaction with the business modification, that it has applied the message; a unique key protects against parallel duplication. The acknowledgement sent to the broker follows. This way a redelivered message can be recognised, and the effect is not repeated.
For an external API call, the inbox and a local transaction cannot atomically include the remote effect. In that case the idempotency key of the external API, a status query or reconciliation is needed (6.5).
Operational consequences
The number and age of unsent records must be monitored. A long-waiting event means a business delay. Cleaning the table and retention are to be weighed: deletion should not make troubleshooting or replay impossible.
Common misconceptions
In short
Review questions
Learning resources