Heartbeat, Reconnection and State Resynchronisation

Objectives

By the end of the lesson, you can explain how a broken connection is recognised, how a client reconnects without raising the load, and how application state is restored.

Why is it needed?

A persistent connection can break because of a network change, a proxy timeout, a server failure or a sleeping client. The client and the server must jointly design how they recognise the break and how they restore the state. Restoring the connection is not restoring the state.

Heartbeat and time limit

One party expects a regular sign of life. The deadline must be aligned with the proxy's idle timeout, the network latency and the client's limits: a limit that is too short causes needless disconnection, one that is too long detects failure late. The protocol's Ping/Pong frames (10.1) are suitable for this; if the API gives no access, an application-level heartbeat can be used. A page pushed to the background or a sleeping device can delay timers, so the server should not assume an exact rhythm. A pong shows the life of the connection and of the handler, not the success of an earlier business command.

Reconnection with backoff

Immediate, endless reconnection from many clients can overload a server that has just restarted. According to the AWS description, exponential backoff (the wait grows with each attempt) and jitter (random deviation) spread the attempts; the wait and the number of attempts should be capped. A non-improving authentication error should not be treated as a network retry. The server may close the connection with code 1001 ("going away") when shutting down; mass disconnection causes a reconnection wave, so gradualness and jitter are needed (10.4).

stateDiagram-v2
  [*] --> Connecting
  Connecting --> Syncing: connection established
  Connecting --> Waiting: transient failure
  Syncing --> Active: state and subscriptions in order
  Active --> Waiting: connection lost
  Waiting --> Connecting: backoff expired
  Connecting --> Stopped: final rejection
  Active --> Stopped: user leaves
  Stopped --> [*]

The Active state does not begin with the connection being established, but after the necessary authentication and synchronisation. The client thereby does not show stale state as fresh data.

Replay by event identifier

The client may remember the identifier of the last applied event and ask to continue from that point on reconnection (like the Last-Event-ID mechanism of SSE, 9.3). The server must keep a truly available, authorisation-filtered event log for this; a lastId field alone does not create durable replay. If the retention time has expired or the position is unknown, the server gives a resync signal: the client requests a fresh snapshot and takes over the events from the version boundary of that snapshot.

The race between snapshot and stream

If the client first requests a snapshot and then subscribes, a change may be missed in between. If it first opens the stream and then requests a snapshot, the events and the snapshot must be merged. A common version or cursor decides which events are already contained in the snapshot. The client can recognise a repeated event by identifier or version; with a delta message a missing predecessor requires re-reading, with a full-state message a newer version can overwrite the older (8.3).

Characteristics

Aspect	Reconnection and resync
Direction	Two-way: the client asks to continue, the server replays
Initiator	The client the reconnection; the server the resync signal
Connection persistence	After a break, a new connection
Latency	No data during the break; replay makes up for it
Load	A reconnection wave at shutdown; the cost of replay
Fault tolerance	Replay or snapshot; idempotent event handling
When it fits	Short interruptions, retained log
When it does not	If there is no retention and the state cannot be queried separately

Common misconceptions

Statement	Clarification
"After reconnecting everything continues."	Without replay, missed events are lost.
"It is enough to reconnect immediately."	Without backoff and jitter it overloads.
"A pong proves a successful business operation."	Only the life of the connection and the handler.
"A lastId field is enough to resume."	The server must keep an actual log.

In short

• The heartbeat time limit must be aligned with the proxy's idle timeout.
• Reconnection: exponential backoff + jitter, with an upper bound.
• State is restored by replay or snapshot, with a common version boundary.
• The Active state begins after synchronisation.

Review questions

1. Why is it not enough to open a new socket after losing the connection?
2. How would you close the gap between the snapshot and the stream?
3. What state should the interface show during synchronisation?

Learning resources

• Timeouts, retries and backoff with jitter (AWS Builders' Library) — backoff and jitter.
• RFC 6455, 5.5 and 7.4 — control frames, closing codes.
• HTML Standard: Server-sent events — the id/Last-Event-ID mechanism.