kind of right, kind of wrong

* for client-side load balancing, it's entirely possible to move active healthchecking into a dedicated service and have its results be vended along with discovery. In fact, more managed server-side load balancers are also moving healthchecking out of band so they can scale the forwarding plane independently of probes.

* for server-side load balancing, it's entirely possible to shard forwarders to avoid SPOFs, typically by creating isolated increments and then using shuffle sharding by caller/callee to minimize overlap between workloads. I think Alibaba's canalmesh whitepaper covers such an approach.

As for scale, I think for almost everybody it's completely overblown to go with a p2p model. I think a reasonable estimate for a centralized proxy fleet is about 1% of infrastructure costs. If you want to save that, you need to have a team that can build/maintain your centralized proxy's capabilities in all the languages/frameworks your company uses, and you likely need to be build the proxy anyways for the long-tail. Whereas you can fund a much smaller team to focus on e2e ownership of your forwarding plane.

Add on top that you need a safe deployment strategy for updating the critical logic in all of these combinations, and continuous deployment to ensure your fixes roll out to the fleet in a timely fashion. This is itself a hard scaling problem.

I've never quite understood why there couldn't be a standardised "reverse" HTTP connection, from server to load balancer, over which connections are balanced. Standardised so that some kind of health signalling could be present for easy/safe draining of connections.

Back in the day, I thought about this problem domain a lot! I even wrote and open-sourced a service discovery framework called SmartStack, an early precursor to later approaches like Envoy, described here: https://medium.com/airbnb-engineering/smartstack-service-dis...

This was a client side framework, in the OPs parlance. What's missing in OP is the insight that the server-side load balancer can also fail -- what will load balance the load balancers? We performed registration based on health checks from a sidecar, and then we also did client side checks which we called connectivity checks. Multiple client instances can disagree about the state of the world because network partitions actually can result in different states of the world for different clients.

Finally, you do also still need circuit breakers. Health checks are generally pretty broad, and when a single endpoint in a service begins having high latency, you don't want to bring down the entire client service with all capacity stuck making requests to that one endpoint. This specific example is probably more relevant to the old days of thread and process pools than to modern evented/async frameworks, but the broader point still applies

I wrote this after seeing cases where instances were technically “up” but clearly not serving traffic correctly.

The article explores how client-side and server-side load balancing differ in failure detection speed, consistency, and operational complexity.

I’d love input from people who’ve operated service meshes, Envoy/HAProxy setups, or large distributed fleets — particularly around edge cases and scaling tradeoffs.

It seems like passive is the best option here but can someone explain why one real request must fail? So the load balancer is monitoring for failed requests. If it receives one can it not forward the initial request again?

I have to say I am not a fan of doing this on the client side.

API gateways (which is what server side load-balancer can be abstracted as) serve as important control points for service traffic, for example for auth, monitoring and observability, application firewall, rate limiting etc.

In my general experience code running on the client side is less reliable due to permutations of browsers, flaky networks, challenges with observability.

That said, client side already has one type of load balancing - DNS - but that doesn’t address the availability challenge.

“We won’t fix the simple, visible server-side problem, so we’ll distribute a harder version of it into every client.”