Service Discovery, Load Balancing and Service Mesh
Objectives
By the end of the lesson, you can explain how a caller finds an instance of a service, how load is distributed, and what a service mesh gives and what it does not.
Why is it needed?
The logical name of a service can be stable while its instances start, stop and move to other addresses. The caller should not build on hand-typed addresses assumed valid for ever.
Service discovery
Service discovery is finding the current address of an instance. microservices.io describes two variants:
Client-side: the client queries the service registry (the directory of instances), chooses an instance itself and calls it directly. Advantage: fewer moving parts and network hops; drawback: the client is tied to the registry, and the discovery logic must be implemented in every language/framework.
Server-side: the client calls a router (load balancer) at a known address, which queries the registry and forwards. Advantage: a simpler client; drawback: one more component that needs availability and capacity, plus one more hop.
The Kubernetes Service object is a typical platform solution: it gives a stable virtual IP address and a name to a group of pods, and the endpoints are updated automatically as pods start and stop, so the client does not have to track the individual pods.
flowchart TB
subgraph CS["Client-side"]
A["Caller"] -->|"instance list"| R["Registry"]
A -->|"direct call"| B["Chosen instance"]
end
subgraph SS["Server-side"]
C["Caller"] --> D["Stable address: router"]
D --> E["Instance 1"]
D --> F["Instance 2"]
end
Discovery data is never perfectly fresh: the registry, the DNS cache and the health check can lag, and an instance can fail after the query. Discovery therefore does not replace the per-call time limit and failure handling (6.5). A connection kept open for a long time can stay with the old instance even if name resolution has already been updated.
Load balancing
Load balancing is the distribution among instances. According to the NGINX documentation the default method is round-robin, and there are least-connections (least_conn) and IP-hash modes; the weight parameter gives weighting. IP-hash directs further requests from the same client to the same server (sticky session); this can simplify local state, but it can distort the distribution and does not solve failure.
The "least connections" measure matters: an open WebSocket and a short HTTP request are not the same resource demand (10.4).
Health, readiness and failure
According to the Kubernetes documentation the readiness probe signals whether the container can accept traffic; if it fails, the pod is removed from the Service endpoints. The liveness probe decides when the container must be restarted (for example in a deadlock). A wrongly configured liveness probe can lead to cascading failures; checking external dependencies belongs in the readiness probe, not the liveness one. A successful health check does not guarantee the success of the next request: the path can fail, and the state can change.
When taking an instance out of service, it should first stop receiving new requests, and then finish those in progress within a limited time. For long connections a separate teardown and reconnection procedure is needed.
Service mesh
A service mesh is the traffic infrastructure between services: according to the Istio description it is a layer that gives applications security (mTLS, workload identity, access rules), traffic management (for example percentage traffic splitting, load balancing) and observability without code changes. According to the Envoy description the proxy runs as a separate process next to the application, so it is independent of the application's language; a network of such proxies forms a mesh.
A mesh can be divided into two parts: the data plane is the proxies forwarding the traffic, and the control plane gives them configuration and rules. (This distinction does not appear on the Istio page cited above; it is general professional usage.)
flowchart TB
CP["Control plane: rules, configuration"] -.-> P1
CP -.-> P2
subgraph SA["Environment of service A"]
S1["Business code"] --> P1["Traffic proxy"]
end
subgraph SB["Environment of service B"]
P2["Traffic proxy"] --> S2["Business code"]
end
P1 -->|"service-to-service traffic"| P2
The mesh does not know the business meaning of an operation: it cannot know whether a payment request repeated after a timeout is safe. The independent retries of several layers multiply (6.5): if the client, the gateway and the mesh each try three times, one operation can produce up to 3³ = 27 network attempts (an illustrative maximum; the exact behaviour depends on the configuration). It must be decided in one place who may repeat, under what condition and within what time.
When is it justified?
The number of common traffic rules, the size of the services and teams, and the operational capability together may justify a mesh. In a small system the gateway and the platform's service names may be enough. Every intermediate layer brings new configuration and a new diagnostic surface; the cause of a failure can be the service, the proxy, the routing or the certificate.
Common misconceptions
Statement
Clarification
"Publishing a new DNS address moves the traffic at once."
Caches and long connections delay it.
"A sticky session solves failure."
No; it only pins the target of the connection.
"A successful health check guarantees the success of the request."
It is an input to routing, not an execution guarantee.
"A service mesh is mandatory for microservices."
In a small system the platform services may be enough.
In short
Discovery finds the current address of an instance; it has client-side and server-side variants.
The algorithm and measure of load balancing (connections, weight, hash) follow the load.
Readiness relates to accepting traffic, liveness to restarting.
A service mesh is traffic infrastructure, not business architecture; the layers of retries must be coordinated.
Review questions
When is client-side and when server-side discovery better?
What is the difference between readiness and liveness?
Service Discovery, Load Balancing and Service Mesh
Objectives
By the end of the lesson, you can explain how a caller finds an instance of a service, how load is distributed, and what a service mesh gives and what it does not.
Why is it needed?
The logical name of a service can be stable while its instances start, stop and move to other addresses. The caller should not build on hand-typed addresses assumed valid for ever.
Service discovery
Service discovery is finding the current address of an instance. microservices.io describes two variants:
The Kubernetes Service object is a typical platform solution: it gives a stable virtual IP address and a name to a group of pods, and the endpoints are updated automatically as pods start and stop, so the client does not have to track the individual pods.
flowchart TB subgraph CS["Client-side"] A["Caller"] -->|"instance list"| R["Registry"] A -->|"direct call"| B["Chosen instance"] end subgraph SS["Server-side"] C["Caller"] --> D["Stable address: router"] D --> E["Instance 1"] D --> F["Instance 2"] endDiscovery data is never perfectly fresh: the registry, the DNS cache and the health check can lag, and an instance can fail after the query. Discovery therefore does not replace the per-call time limit and failure handling (6.5). A connection kept open for a long time can stay with the old instance even if name resolution has already been updated.
Load balancing
Load balancing is the distribution among instances. According to the NGINX documentation the default method is round-robin, and there are least-connections (least_conn) and IP-hash modes; the
weightparameter gives weighting. IP-hash directs further requests from the same client to the same server (sticky session); this can simplify local state, but it can distort the distribution and does not solve failure.The "least connections" measure matters: an open WebSocket and a short HTTP request are not the same resource demand (10.4).
Health, readiness and failure
According to the Kubernetes documentation the readiness probe signals whether the container can accept traffic; if it fails, the pod is removed from the Service endpoints. The liveness probe decides when the container must be restarted (for example in a deadlock). A wrongly configured liveness probe can lead to cascading failures; checking external dependencies belongs in the readiness probe, not the liveness one. A successful health check does not guarantee the success of the next request: the path can fail, and the state can change.
When taking an instance out of service, it should first stop receiving new requests, and then finish those in progress within a limited time. For long connections a separate teardown and reconnection procedure is needed.
Service mesh
A service mesh is the traffic infrastructure between services: according to the Istio description it is a layer that gives applications security (mTLS, workload identity, access rules), traffic management (for example percentage traffic splitting, load balancing) and observability without code changes. According to the Envoy description the proxy runs as a separate process next to the application, so it is independent of the application's language; a network of such proxies forms a mesh.
A mesh can be divided into two parts: the data plane is the proxies forwarding the traffic, and the control plane gives them configuration and rules. (This distinction does not appear on the Istio page cited above; it is general professional usage.)
flowchart TB CP["Control plane: rules, configuration"] -.-> P1 CP -.-> P2 subgraph SA["Environment of service A"] S1["Business code"] --> P1["Traffic proxy"] end subgraph SB["Environment of service B"] P2["Traffic proxy"] --> S2["Business code"] end P1 -->|"service-to-service traffic"| P2The mesh does not know the business meaning of an operation: it cannot know whether a payment request repeated after a timeout is safe. The independent retries of several layers multiply (6.5): if the client, the gateway and the mesh each try three times, one operation can produce up to 3³ = 27 network attempts (an illustrative maximum; the exact behaviour depends on the configuration). It must be decided in one place who may repeat, under what condition and within what time.
When is it justified?
The number of common traffic rules, the size of the services and teams, and the operational capability together may justify a mesh. In a small system the gateway and the platform's service names may be enough. Every intermediate layer brings new configuration and a new diagnostic surface; the cause of a failure can be the service, the proxy, the routing or the certificate.
Common misconceptions
In short
Review questions
Learning resources