In the real world however, the bursts can be correlated, due to factors like timeouts/retries, thundering herd, correlated bursts.
so the real economics of load-balanced system is a simple reliability story: being able to reasonably serve the peak traffic, which leads to over-provisioning of those systems.
using cloud allows some form of scale up/down of resources, but doesn't completely solve the problem. I think the migration away from synchronyous systems towards async systems and letting clients gradually absorb the delays is a better approach (rather than forcing infrastructure to be dynamically scaled up/down and be billed per request-second by your cloud provider)
In the idealized case the comfortable place to operate a system is with around 2/3 utilization, like around there latency (customer experience, employee experience) is reasonable, slack is reasonable, etc.
A manager who is being managed by a manager who is being managed by a manager who is being managed (...) is going to see 0.99 utilization and want that last 0.01 and be oblivious to the fact that the math says the system is already past the breaking point, customers are furious, employees are worn out. Any slack at at all seems like an affront.
See also the gamedev technique of having sacrificial assets or code, so when you need to free up space late in the schedule to ship, you have something you can actually shed.
Bit hard to explain though.
Hawkes processes are what other fields use to model this
Very true, as application-layer load-balancing often explicitly pre-bakes the traffic schedule to several hundred distributed IPs for data locality. Essentially bypassing the functional need for DNS and local round-robin traffic balancers.
One trades concurrent bandwidth for slightly higher latency, and dynamically adapted capacity as traffic load changes. =3
The global edge networks that I’m aware of all use L4 LBs and L7 LBs. Cloudflare picks anycast over DNS LB, but DNS LB is still widely used.
I don’t see these things changing.
Time Division Multiplexing is usually already used on cellular and Wifi wireless protocols. It only requires slight modification to turn it into an effective network traffic balancer to avoid the naive "everyone update on Tuesday 6am UTC", or "It is Christmas morning and game registration is open".
Notably, it also allows tracking specific accounts by encoding disjoint ingress host lists (siloed concurrent user groups with client certs and firewall whitelist rules.) And users do not have global network knowledge as hosts are cycled into temporary stewardship under load. Thus, only the coordinators for one-time new-user registration operates on classical DNS/round-robin host services.
With DNS, by expected function everyone knows the global published ingress points within minutes. Under a DoS the traffic just hammers down, and small firms usually just pay for the Cloudflare like services.
For systems I've known, TDM reduced peak resource capacity costs down by around 37x. Generally speaking, a 100 user group having fun will not share their server details/invites with folks that exhibit lag-switching or other network shenanigans.
But you are correct, in that it doesn't help if PIBKAC. =3
If you control all clients and servers and are on a closed network, you can do all sorts of fun things... though load shedding is helpful for when your good clients turn bad due to a code bug. A self DoS is the worst kind of outage.
> Of course, this assumes independent events. World Cup, super bowls, etc break these assumptions.
Yes, this is very true. The model here works for Poisson arrivals and exponential service time (the M/M), which are poor approximations of real-world traffic patterns (which tend to be non-stationary and non-ergodic, and include substantial seasonality). However, the frequency of that seasonality is typically rather low (e.g. daily cycles), and so these stronger assumptions are quite defensible for short time periods.
A better approach is to do simulation with real traffic patterns, or even with more sophisticated parametric models, and get better answers (e.g. https://stability-sim.systems/). The good news is that kind of simulation is cheaper to do than ever before.
As in between the service and the load balancer? There's already an infinite queue in the load balancer. You can try that out on https://stability-sim.systems/ to see the effect, but the short version is that (in this model) it makes things worse.
If you're saying that the queue in the load balancer should be limited in size to reduce tail latency, then I agree.
If I were to guess there weren't any "backend servers" at all. It was just array of random increasing numbers (that stand for request arrival times) and arrays of numbers with minimum distance (that stand for time each consumer took a request)
there's no connections to "least-ify" the strategy about. There's no difference between consumers, no matter the amount of requests having been processed
- M/M/1 (vertical scaling, one queue and one fast server): fastest response time
- M/M/c (thread-pool, one shared queue, c slow servers): c-times slower for low loads, asymptotically similar to M/M/1 for high loads
- c-times M/M/1 (loadbalancer, c slow servers each which its own queue): always c-times slower than M/M/1.
Only the response times are different. Throughtput is the same in the ideal case.
Still, queuing theory is so cool.
But I suppose the question is underspecified. How does the load balancer know which systems are busy? What happens to a request if the load balancer routes a request to a busy server?
I think that the issue is in part due to the variables. Plotting the mean request time is less intuitive than plotting throughput.
If you plot throughput vs number of servers, it'll be a straight line. And asking people that, I think most would agree on a straight line. But who knows!
For example, most cloud load balancers I've worked with are stateless, non-queuing, and allocate work to back-ends strictly randomly.
Traditional non-cloud load balancers can implement this kind of perfect queuing, but these settings are generally off by default even when available.
- NetScaler: surgeProtection + maxClient=1
- F5 BIG-IP LTM: request queuing + pool/member connectionLimit=1
- HAProxy: server maxconn 1 + timeout queue
- NGINX Plus: server max_conns=1 + queue
Envoy, Apache, and Traefik have partial or limited support.
Conversely, most multi-threaded web server frameworks already do this by default! For example, ASP.NET has essentially an internal "load balancer" with a perfect queue if you pretend each core is a "node" and the whole server is the "scale out system".
And the systems that load balancers front almost never behave this way...
Dont even get me started on client performance here as well, latency, speed, caching -- that can all be impacted by payload size.
The article is interesting, but it is an ideal that almost never turns up in the real world.
a) There must be network packet buffers on each receiver, which is a per-server "queue".
b) The optimal performance can only be achieved by pipelining requests sent to the servers, otherwise there is "dead time" caused by the network latency.
Perfect queuing to workers that act as a "slot" can only be implemented in-process on a single machine, and even there having small queues or buffers per core can improve throughput!
We are actively revisiting our traffic simulation approach, and a surprisingly non-obvious part has been which charts to focus on. Our case is a gpu-server-backed interactive analytics app, like a photoshop for data, so we do focus both on latency and errors, and especially around handling bursty sessions as discussed in the article.