Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurrent use, while vLLM/SGLang easily crash on non-common setups (things like, if you do pipeline parallelism for RTX5090+RTX4090, they will randomly crash with RAM caching enabled or select wrong kernels because they usually assume that every rank is the same device type; they also don't support Q5-Q6).

For me what's most interesting is to optimize inference for lack of good datacenter hardware and how to optimize for it best. I've been running an AI server in the office, and so far I've find these techniques most important for concurrent use on cheap hardware: pipeline parallelism (to accomodate for PCie), RAM caching (to quickly restore contexts into VRAM), speculative decoding (including domain-specific ngrams, they already can speed up code generation considerably without the overhead of a draft model), good kernels highly optimized for a specific device, support for Q5-Q6 (almost as good as Q8), FP8 contexts (more context to fit), paged attention (for better VRAM utilization), prefix caching, continuous batching (this is the default everywhere).

So far the main bottlenecks have been llama.cpp's poor VRAM utilization for contexts (you either have fixed-size slots, or use unified KV cache where each request attends to attention from all other requests and then unnecessary portions of attention are masked out), and lack of decode/prefill segregation: when a request starts prefilling a long context, all decoding threads slow down to like 5 tok/sec. On the other hand, vLLM/SGLang feel superbuggy if you don't run them on some officially approved node like 8xH200

Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses.

As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.

Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.

Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].

Everything old is new again (:

[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...

[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf

[2] https://www.microsoft.com/en-us/research/publication/distrib...

Within LLMs, and even inference naturally, there are many other potential parameters that one might optimize: Unsloth typically shows a Pareto frontier for size of a quantized model versus KL divergence. Others trade total concurrent tok/s against single-stream tok/s. KV cache size, context length and context coherency are other trade-offs that are closely related to inference. Total intelligence is usually a defining characteristic of a "frontier model", with cost (per token or task) as a salient trade-off. Cost is one parameter that is implicitly fixed by the "throughput versus latency" analysis: using a GB300 versus Radeon R9700 moves the curve enormously and probably changes the shape of it. Lots of threads here argue over local vs cloud inference regarding cost efficiency, often with privacy and control as competing objectives.

the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there

I feel like P/D dissaggregation will be the next big one for providers, as prefill tends to be compute bound while decode mem bound which I guess each will have a different type of node

   Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate.

I think this presentation says more about what knobs you can turn and in what direction the outcome will move (it may be worse than a competitor) than it says about frontiers.

The efficient frontier of LLM inference is a line, not a frontier.

this is a frontier:

https://upload.wikimedia.org/wikipedia/commons/e/e1/Markowit...

no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.

If you want to steal an idea for this, steal indifference curves, they'd make more sense.

A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size.

I would define a "frontier model" as offering the highest degree of intelligence at any cost, or without regard to cost. The frontier today is clearly Fable/Mythos, with the "efficient frontier" at Opus/Sol.