> ## Content Index > Fetch the complete content index at: https://www.mohammadsiddiqui.com/llms.txt > Use this file to discover other available public pages before exploring further. # Your GPUs Are Two Thirds Idle - URL: https://www.mohammadsiddiqui.com/your-gpus-are-two-thirds-idle/ - Published: 2026-09-23T04:45:27.000Z - Updated: 2026-09-23T04:45:27.000Z - Description: A single tenant on four GPUs used 37% of the streaming multiprocessors. Four tenants used 71%, with four times the throughput. Same box, no extra hardware. - Author: Mohammad Siddiqui - Tags: architecture, operations Broadcom's performance team published the first of four posts on Monday, and unusually for this kind of thing, they put the test rig and the raw numbers in it. Worth reading properly rather than skimming, because one figure in there should change how you buy GPUs. Here it is. A single tenant running a lightweight inference workload across four GPUs used between 36 and 38 percent of the streaming multiprocessors. Put four tenants on the same hardware and that goes to 71 percent, with throughput rising from roughly 4,500 to 18,000 tokens a second. Nearly four times the work out of the same box. No extra hardware. 37% SM UTILISATION, 1 TENANT 71% SM UTILISATION, 4 TENANTS 4x THROUGHPUT 0 EXTRA SERVERS **The test rig, since it matters:** a Dell PowerEdge XE7745 with two AMD EPYC 9555 64-core processors, 256 logical cores, 768 GB of memory and four NVIDIA RTX Pro 6000 96 GB GPUs. The workload was MLPerf Inference running Llama 3.1 8B at 35 queries per second per tenant, so 140 queries per second with four tenants running. Tested with both passthrough GPU and vGPU. ## Why the underutilisation happens Most enterprise inference is not heavy. It is a chatbot behind an internal portal, a document summariser, a classifier sitting in a workflow. Request rates are low and batch sizes are small. The workload was never going to saturate a GPU, but it got a whole one anyway, because that is how we have always allocated hardware: somebody asked for it, so they own it. I have seen this in enough estates now to recognise the shape of it. The GPU budget gets approved, the cards go in, and eighteen months later nobody can tell you what utilisation looks like because nobody is measuring it. Then a second team asks for GPUs and the answer is a purchase order rather than a look at what is already there. With DRAM and accelerator pricing where it currently is, that is an expensive habit. ## How you actually carve up a GPU Three allocation methods are in play, and the post makes a point that is easy to miss: any of them works whether your tenants share a VM or sit in separate ones. The isolation decision and the allocation decision are independent. Passthrough GPU The whole physical card goes to one VM. Simplest to reason about, best raw performance, and the least flexible. If that VM is running a light workload, the rest of the card is doing nothing and no one else can reach it. vGPU NVIDIA's virtual GPU. A VM can be given a full card or a fraction of one, which is what lets several tenant VMs share the same physical GPU. This is the option that makes VM-based isolation and consolidation compatible with each other. MIG vGPU Multi-Instance GPU. Hardware-level partitioning on supported cards, giving each instance its own compute and memory slice rather than time-sharing. Stronger separation than plain vGPU, and correspondingly less elastic once you have set the partitions. There is also the option of not sharing at all. For workloads that genuinely consume a server, VCF can provision dedicated physical hosts that behave like bare metal while keeping the platform features around them. The MLPerf Inference 5.1 results Broadcom published last December were run that way, across H200 and B200 GPUs and Intel Xeon with AMX. What was in those MLPerf 5.1 runs, if you are curious Speech to text with Whisper, text to video with Stable Diffusion XL, large language models including Llama 3.1 405B, Llama 2 70B and Llama 3 8B, graph neural networks with R-GAT, and computer vision with RetinaNet. Worth knowing the breadth, because the multi-tenant case in this new post is deliberately at the light end of that range. ## The two isolation models, and the catch **Separate VMs per tenant.** GPUs allocated by passthrough, vGPU or MIG. This is the one for anything with a real isolation requirement. **Shared VM, tenants separated by containers.** Denser, simpler to manage, lower isolation. The performance result is the interesting part: there was almost no measurable difference between the two. So if you were assuming the stricter isolation model costs you throughput, it does not, at least not at this workload profile. Now the catch, and it is buried in a bracket in the original post. Co-locating multiple tenants inside one VM is supported **only when you manage tenancy manually**. VCF Automation requires each tenant to sit in its own VM. That is a genuine architectural fork and I do not think it is being talked about enough. The denser model is the one you cannot drive through self-service. If your platform strategy is a catalogue that tenants provision from, you are on VM-based isolation, and the container-in-shared-VM option is off the table until that changes. Which is fine, because the performance data says you are not giving anything up by taking that route. But it does mean the choice is about operating model rather than performance, and those two conversations usually happen with different people in the room. ## Before you buy another GPU A short list. Tick them off as you go, nothing is stored. Measure streaming multiprocessor utilisation across the GPUs you already own, over a fortnight rather than a snapshot. Establish what a tenant is in your organisation. Department, project, or external customer. The answer changes the isolation requirement. Decide the isolation model deliberately, knowing that the denser one rules out VCF Automation. Work out how you meter and charge shared GPU capacity, before the platform team defaults back to dedicated allocation. Check your failure domain. Four tenants on one host means one host outage hits four teams. Ask what happens in GPU memory between tenants, and get the answer in writing if you are in a regulated industry. ## The question nobody has answered yet GPU memory persists between workloads. When tenant A's inference job finishes and tenant B's starts on the same card, what happened to what was in memory. The MetalSoft material from Explore touched on this for bare-metal reprovisioning, mentioning GPU persistence reset as part of secure tenant deprovisioning. But for the live sharing case, running four tenants concurrently on the same physical GPUs, I have not seen a clear statement of what the isolation guarantee is and who verifies it. MIG gives you hardware partitioning, which helps the argument, but plain vGPU time-slicing is a different proposition. For most enterprises that will not matter. For a bank running inference over customer data, or anyone in a regulated industry sharing accelerators across business units, it is a question worth asking your account team directly rather than assuming. I would want it in writing before I put two lines of business on the same card. ## Caveats worth keeping in mind - This is Broadcom's own performance team testing their own platform. Methodology is disclosed, which is more than most vendors do, but it is not independent. - The workload is deliberately lightweight. Llama 3.1 8B at 35 queries a second is a small model at a modest rate. Do not extrapolate to training, to a 70B model, or to anything with high concurrency. - RTX Pro 6000 rather than H200 or B200\. The consolidation argument should hold across cards, but the numbers will not. - vSphere 9.1 and all future 9.x releases are NVIDIA-Certified Hypervisor, which is the backdrop to all of this. Near bare metal is the claim being defended. ## What is coming next This is part one of four. The rest of the series covers GPU utilisation and total cost of ownership, then CPU and memory utilisation, then time-slicing to optimise inference throughput. The time-slicing one is what I will be reading most carefully. Everything in part one is measured with four cooperative workloads at a fixed rate. Time-slicing is where you find out what happens when two tenants want the card at the same moment, and that is the question any real multi-tenant platform has to answer. Broadcom put a contact address on the post for questions about AI workload performance, which is a fair offer and not one I see often. If you have a specific sizing question, that is probably a better route than guessing from a blog. ## Sources - [Optimizing AI Deployments with VMware Cloud Foundation, Part 1](https://blogs.vmware.com/cloud-foundation/2026/09/21/optimizing-ai-deployments-with-vmware-cloud-foundation-part-1/?ref=mohammadsiddiqui.com), Lan Vu, Hari Sivaraman and Uday Kurkure, 21 September 2026 - [Optimize AI Deployments with VCF](https://www.vmware.com/docs/optimize-ai-deployments-with-vcf?ref=mohammadsiddiqui.com), the white paper behind the series - [Tenancy deployment models with VMware Cloud Foundation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/design/design-library/vcf-automation-deployment-models-9-x/tenancy-deployment-models-with-vmware-cloud-foundation.html?ref=mohammadsiddiqui.com) - [VCF achieves NVIDIA Hypervisor Certification](https://blogs.vmware.com/cloud-foundation/2026/08/27/vcf-achieves-nvidia-hypervisor%5Fcertification/?ref=mohammadsiddiqui.com) - [MLPerf 5.1 results on VCF](https://blogs.vmware.com/cloud-foundation/2025/12/15/mlperf-5-1-confirms-vcf-future-of-ai-ml-performance/?ref=mohammadsiddiqui.com) *Views expressed here are my own.*