Your GPUs Are Two Thirds Idle

Broadcom's performance team published the first of four posts on Monday, and unusually for this kind of thing, they put the test rig and the raw numbers in it. Worth reading properly rather than skimming, because one figure in there should change how you buy GPUs.

Here it is. A single tenant running a lightweight inference workload across four GPUs used between 36 and 38 percent of the streaming multiprocessors. Put four tenants on the same hardware and that goes to 71 percent, with throughput rising from roughly 4,500 to 18,000 tokens a second.

Nearly four times the work out of the same box. No extra hardware.

Why the underutilisation happens

Most enterprise inference is not heavy. It is a chatbot behind an internal portal, a document summariser, a classifier sitting in a workflow. Request rates are low and batch sizes are small. The workload was never going to saturate a GPU, but it got a whole one anyway, because that is how we have always allocated hardware: somebody asked for it, so they own it.

I have seen this in enough estates now to recognise the shape of it. The GPU budget gets approved, the cards go in, and eighteen months later nobody can tell you what utilisation looks like because nobody is measuring it. Then a second team asks for GPUs and the answer is a purchase order rather than a look at what is already there.

With DRAM and accelerator pricing where it currently is, that is an expensive habit.

How you actually carve up a GPU

Three allocation methods are in play, and the post makes a point that is easy to miss: any of them works whether your tenants share a VM or sit in separate ones. The isolation decision and the allocation decision are independent.

There is also the option of not sharing at all. For workloads that genuinely consume a server, VCF can provision dedicated physical hosts that behave like bare metal while keeping the platform features around them. The MLPerf Inference 5.1 results Broadcom published last December were run that way, across H200 and B200 GPUs and Intel Xeon with AMX.

The two isolation models, and the catch

Separate VMs per tenant. GPUs allocated by passthrough, vGPU or MIG. This is the one for anything with a real isolation requirement.

Shared VM, tenants separated by containers. Denser, simpler to manage, lower isolation.

The performance result is the interesting part: there was almost no measurable difference between the two. So if you were assuming the stricter isolation model costs you throughput, it does not, at least not at this workload profile.

Now the catch, and it is buried in a bracket in the original post.

That is a genuine architectural fork and I do not think it is being talked about enough. The denser model is the one you cannot drive through self-service. If your platform strategy is a catalogue that tenants provision from, you are on VM-based isolation, and the container-in-shared-VM option is off the table until that changes.

Which is fine, because the performance data says you are not giving anything up by taking that route. But it does mean the choice is about operating model rather than performance, and those two conversations usually happen with different people in the room.

Before you buy another GPU

A short list. Tick them off as you go, nothing is stored.

The question nobody has answered yet

GPU memory persists between workloads. When tenant A's inference job finishes and tenant B's starts on the same card, what happened to what was in memory.

The MetalSoft material from Explore touched on this for bare-metal reprovisioning, mentioning GPU persistence reset as part of secure tenant deprovisioning. But for the live sharing case, running four tenants concurrently on the same physical GPUs, I have not seen a clear statement of what the isolation guarantee is and who verifies it. MIG gives you hardware partitioning, which helps the argument, but plain vGPU time-slicing is a different proposition.

For most enterprises that will not matter. For a bank running inference over customer data, or anyone in a regulated industry sharing accelerators across business units, it is a question worth asking your account team directly rather than assuming. I would want it in writing before I put two lines of business on the same card.

Caveats worth keeping in mind

• This is Broadcom's own performance team testing their own platform. Methodology is disclosed, which is more than most vendors do, but it is not independent.
• The workload is deliberately lightweight. Llama 3.1 8B at 35 queries a second is a small model at a modest rate. Do not extrapolate to training, to a 70B model, or to anything with high concurrency.
• RTX Pro 6000 rather than H200 or B200. The consolidation argument should hold across cards, but the numbers will not.
• vSphere 9.1 and all future 9.x releases are NVIDIA-Certified Hypervisor, which is the backdrop to all of this. Near bare metal is the claim being defended.

What is coming next

This is part one of four. The rest of the series covers GPU utilisation and total cost of ownership, then CPU and memory utilisation, then time-slicing to optimise inference throughput.

The time-slicing one is what I will be reading most carefully. Everything in part one is measured with four cooperative workloads at a fixed rate. Time-slicing is where you find out what happens when two tenants want the card at the same moment, and that is the question any real multi-tenant platform has to answer.

Broadcom put a contact address on the post for questions about AI workload performance, which is a fair offer and not one I see often. If you have a specific sizing question, that is probably a better route than guessing from a blog.

Sources

• Optimizing AI Deployments with VMware Cloud Foundation, Part 1, Lan Vu, Hari Sivaraman and Uday Kurkure, 21 September 2026
• Optimize AI Deployments with VCF, the white paper behind the series
• Tenancy deployment models with VMware Cloud Foundation
• VCF achieves NVIDIA Hypervisor Certification
• MLPerf 5.1 results on VCF

Views expressed here are my own.