Inference Infrastructure, From the Bottom Up
Seven connected pieces on what actually happens underneath an AI inference workload, and why the infrastructure decisions either help it or quietly throttle it. Written for people who know virtualisation and not transformers.
Seven connected pieces on what actually happens underneath an AI inference workload, and why the infrastructure decisions either help it or quietly throttle it.
I started writing these because most AI material aimed at infrastructure people stops at "allocate a GPU and watch utilisation", and that model falls over almost immediately. Inference does not behave like one workload. It behaves like two, with opposite requirements, fighting over the same silicon.
Part zero is where the data came from. Parts one to three explain how it works. Parts four to six are the decisions that follow. Each piece stands on its own if you only care about one of them.
Where it started
The measurement that prompted the rest.
Where the data came from. Broadcom's performance team published test results showing a single tenant using 36 to 38% of GPU streaming multiprocessors, and four tenants on the same hardware reaching 71%. The figures underneath most of what follows.
How it works
Start here if you are new to this. Each one builds on the last.
The mechanics. Prefill is compute-bound, decode is memory-bandwidth-bound, and the KV cache rather than compute is what sets your real capacity. Covers PagedAttention, continuous batching, chunked prefill, and how to tell which of three bound states your deployment is actually in.
Passthrough, time-sliced vGPU and MIG do quite different things to the thing that limits you. Model weights are a fixed cost at every slice size, so fractional profiles do not divide capacity the way a spreadsheet suggests.
Quantisation is usually framed as a quality tradeoff, but on a decode-heavy workload it is a bandwidth decision first. Covers weights, activations and the KV cache separately, because they behave nothing alike.
What you do about it
Cost, tooling, and the awkward case of retrieval.
Building a cost model that survives a finance review. What belongs in the numerator, why the denominator is the hard part, and the honest comparison against API pricing including when the API is the better answer.
vLLM, TensorRT-LLM, SGLang and TGI, plus what NIM and Triton actually are. They implement the same ideas, so the decision turns on operational fit rather than throughput. No benchmark numbers, deliberately.
Five stages in the request path and only the last two involve the model. Index memory, embedding dimensions as a long-term commitment, why prefix caching helps less than you hope, and the permissions problem nobody designs for.
Related, though not part of the series
Only 26% of AI-generated patches worked sits next door rather than inside. It is about software supply chain security rather than inference infrastructure, but the pattern underneath is the same one: AI adoption running ahead of the operational practice around it, and the gap showing up wherever your weakest process already was. Worth reading if automated remediation is on anyone's roadmap near you.
Who this is for
Infrastructure architects and platform teams being asked to run AI workloads, who want to understand the layer immediately above the hypervisor rather than treating the serving stack as something that happens elsewhere.
It assumes you know virtualisation well and nothing about transformers. Almost no maths, and the one formula that does appear is there because the shape of it explains a lot.
None of it is VMware-specific
The mechanics come from how transformer inference works and how modern serving systems manage memory. They apply identically whether you are running vLLM on VCF, on OpenShift AI, on Ray, or on bare metal in a cupboard.
Where the platform does matter is everything underneath: how the GPU is presented, how tenants are isolated, what the fabric looks like, and whether your automation can drive any of it. That is the layer I know best, and it is the one that decides whether the serving stack above ever gets a chance to perform.
Still to come
Agentic workloads as an infrastructure problem, since a single request becomes several model calls with retrieval between them and every latency budget in part six has to stretch to cover it. Disaggregated prefill and decode, which would change how you lay out a GPU estate entirely. And evaluation, which is the gap sitting underneath most of the decisions above.
If there is something you would rather I covered, or something here you think I have got wrong, reply to any post. Corrections are genuinely welcome and they usually improve the piece.