VCF 9.1 Design Topology Field Guide Part 3: Compute, Storage, and AI Topology Patterns

This is Part 16 of an architect’s read of VCF 9.1, and the third of four Design Topology Field Guide articles. Parts 14 and 15 covered foundational and network topology. This one covers compute, storage, and AI the workload-class architectural decisions that shape what VCF 9.1 can actually run.

Five patterns: vSphere Supervisor consuming NSX VPC for cloud-native Kubernetes networking; VKS at 500-cluster scale per Supervisor; vSAN ESA with Auto-RAID and Global Deduplication as the new storage baseline; NVMe Memory Tiering as the most underrated TCO feature in the release; and Enhanced DirectPath I/O for AI workloads that actually scale across hosts.

Pattern 1: vSphere Supervisor with NSX VPC cloud-native Kubernetes networking

vSphere Supervisor is the Kubernetes control plane embedded in vSphere. In VCF 9.1, the Supervisor consumes the VPC consumption model (covered in Part 15) the same way the rest of the platform does. This means VKS clusters get cloud-native networking by default.

The construct chain when VKS is deployed:

• vSphere Supervisor is deployed in a VCF Workload Domain, consuming ESX for compute, NSX or vSphere Distributed Switch for networking, and vSAN or other shared storage

• Each NSX Project has one Transit Gateway interconnecting the NSX VPCs in that Project

• Each VPC has an implicit VPC Gateway providing inter-subnet communication within the VPC

• The VPC Gateway connects upstream to the Transit Gateway

• Connectivity between workloads in the same VPC is open by default unless the East-West firewall (DFW) is configured

• VPC subnets come in types: Private (NAT required to reach external), Public (routable north of TGW), Isolated (no external routing), and Private Transit Gateway subnets (new in VCF 9 accessed via the Default TGW)

Three networking stacks are supported for vSphere Supervisor in VCF 9.x:

• VCF Networking (NSX VPCs) the recommended stack. Full feature set, cloud-native VPC consumption

• Classic NSX (Tier-0/Tier-1 with overlay) still supported. Familiar pattern for teams with deep NSX-T history

• vSphere Distributed Switch (VDS) supported but expected to be rare in VCF deployments. NSX on dVPGs feature must be enabled. vDefend + Antrea still provides unified security across this stack

HLD considerations:

• For the VPC Gateway, you can use a separate NSX Edge cluster from the one hosting the Tier-0/Transit Gateway tier isolation by Edge cluster matters for large-scale designs

• Supervisor consumes Storage Policy Based Management (SPBM) datastores provisioned through any supported storage path (vSAN ESA, NetApp ONTAP via OTV, Pure FlashArray, Dell PowerStore/PowerMax, HPE Alletra) can be consumed by Supervisor

• Multi-vNIC nodes (new in 9.1) allow Supervisor control plane and workload nodes to have multiple NICs for traffic segregation

• Namespace Capture (new in 9.1) provides rapid snapshotting of namespaces for backup, replication, or troubleshooting

Anti-patterns:

• Deploying Supervisor on the legacy classic NSX stack “because it’s familiar” when VPC consumption is the strategic direction you’ll re-platform eventually

• Mixing VKS networking stacks within a single Project keep one stack per Project for operational clarity

• Forgetting the NSX on dVPGs requirement when using VDS-based Supervisor

Diagram to use: the vSphere Supervisor + NSX VPC architecture diagram. Source:

• Broadcom TechDocs vSphere Supervisor Design Library

• MyNetworker.blog vSphere Supervisor Networking with NSX VPC VCF 9

Pattern 2: VKS at 500 Clusters per Supervisor the platform-engineering scale point

VCF 9.1 increases vSphere Kubernetes Service (VKS) scalability to 500 Kubernetes clusters per Supervisor approximately a 7x increase over previous releases. This is a structural change in what VKS can be used for.

What 500 clusters per Supervisor unlocks:

• Multi-tenant Kubernetes platforms supporting hundreds of development teams from a single management backbone

• Compliance-driven cluster isolation dedicate clusters per regulated workload class rather than sharing namespaces

• Application-isolation-by-default patterns cluster-per-app becomes economically viable

• Faster blue/green and canary release patterns at infrastructure level

• Reduced operational coupling between teams sharing the platform

Plus 70% faster provisioning the cluster-creation tax drops substantially. New clusters spin up faster, which reshapes how dev teams consume the platform.

Combined with multi-vNIC nodes and Namespace Capture (both new in 9.1), VCF 9.1 becomes a genuinely cloud-grade Kubernetes platform on private cloud infrastructure. Persistent volume scale also increases to 25,000 RWO PVs per Supervisor enough for production storage at meaningful scale.

Design topology patterns for VKS at scale:

Pattern A: Cluster-per-team. Each development team gets their own VKS cluster, with their own namespaces inside. RBAC at the cluster boundary, not just at the namespace boundary. Strong isolation, easy quota management. Suitable for organisations with strict team-level autonomy.

Pattern B: Cluster-per-application. Each significant application gets its own cluster. Used by organisations where applications have distinct lifecycle, scaling, and operational profiles. Particularly common for stateful applications (databases, message brokers) where node-level operations differ from stateless workloads.

Pattern C: Cluster-per-environment. Dev, staging, prod each get their own cluster (or set of clusters). Standard pattern for organisations with strong environment separation. Sometimes combined with cluster-per-app for prod-only.

Pattern D: Shared cluster with namespace separation. Multiple teams or applications share clusters, separated by namespaces. Lower operational overhead but weaker isolation. Used where teams are tightly coupled or scale doesn’t justify separate clusters.

Pattern E: Cluster-per-compliance-zone. Regulated workloads (PCI, HIPAA, FedRAMP) get dedicated clusters with cluster-level compliance controls. Often combined with Isolated VPC Connectivity Policy (Part 15).

HLD considerations:

• Sizing the Supervisor itself control plane sizing matters more at 500-cluster scale. “Small” control plane sizing may not fit

• Storage policy alignment each cluster needs its persistent volume storage policy mapped to the underlying storage capability

• Network design 500 clusters means significant IP consumption for control plane traffic, workload traffic, and load balancers. Plan IP block allocations carefully

• Observability scale VCF Operations natively monitors Supervisor and VKS via Telegraf agent. Verify scale handling at 500-cluster level

Diagram to use: the VKS scalability diagram or the multi-tenant platform architecture. Source:

• Broadcom Announcing VCF 9.1

• Broadcom Operations in VMware Cloud Foundation 9.0 (VKS monitoring)

Pattern 3: vSAN ESA + Auto-RAID + Global Deduplication the new storage baseline

vSAN Express Storage Architecture (ESA) in VCF 9.1 stops being the “modern alternative” and becomes the design baseline. The 9.1 release closes most of the remaining feature gaps with the legacy OSA path.

Headline architectural points:

Global Deduplication now GA in 9.1. Up to 8x dedup ratios. Works with Data-at-Rest encryption (this is the major differentiator dedup + encryption together was the long-standing gap). ZSTD compression replaces LZ4 for better space efficiency. Operates across the entire cluster rather than per-disk-group.

Auto-RAID. The platform picks the right RAID level for each VM based on storage policy and current cluster state. RAID-1 for small or IO-intensive workloads. RAID-5 or RAID-6 for capacity-intensive workloads. No manual tier decisions per VM the policy engine handles it.

Concurrent OSA + ESA mounting. Mixed clusters supported during migration windows. Workloads can be Storage-vMotioned from OSA to ESA datastores within the same cluster, simplifying the OSA-to-ESA migration path.

vSAN Storage Clusters. Cross-vCenter sharing of vSAN clusters. One vSAN cluster can serve multiple workload domain vCenters. Fan-in protection for workloads sourced from multiple datastores. Multi-source consolidation onto a single ESA target.

Data-in-Transit Encryption for Storage Clusters. Added in 9.1. Encrypted vSAN traffic between hosts. Combined with Data-at-Rest encryption for end-to-end vSAN encryption.

Native S3 Object Storage. Tech preview in VCF 9.1 native S3-compatible object storage on vSAN. Useful for cloud-bursting patterns, backup targets, AI training data, and any S3-compatible workload that benefits from local storage.

HLD considerations:

• Design new clusters on ESA by default. The cost-per-GB and operational simplicity favour ESA at scale

• Migrate OSA workloads to ESA via concurrent mounting and Storage vMotion plan migration windows per cluster

• Enable Global Dedup + DaR encryption from Day 0 for compliance and cost efficiency

• Hardware: ESA requires NVMe-only configurations. Check the VCF 9.1 HCL

• Auto-RAID policy: validate that policy-driven RAID selection aligns with workload performance requirements before enabling at fleet scale

• vSAN Storage Clusters useful for consolidation but plan latency between consuming vCenter and serving cluster

Anti-patterns:

• Deploying new clusters on OSA. ESA is the strategic direction; OSA-first designs build technical debt

• Sizing vSAN ESA on the same assumptions as OSA dedup ratios are higher; usable capacity calculations differ

• Forgetting that Global Dedup applies across the cluster, not per disk group capacity planning needs the new mental model

• Skipping the vSAN ESA HCL validation for AI workloads NVMe Memory Tiering and Enhanced DirectPath I/O often share hardware constraints

Diagram to use: the vSAN ESA architecture diagram showing Global Deduplication, Auto-RAID, and storage cluster topology. Source:

• Broadcom Announcing VCF 9.1 (storage section)

• Broadcom TechDocs Storage Models in VCF Architectural Options

• vStellar VCF-9 Part 1: Introduction & Architecture

Pattern 4: NVMe Memory Tiering the most underrated feature in 9.1

NVMe Memory Tiering in VCF 9.1 turns NVMe SSDs into a tier of memory the hypervisor can use to handle cold memory pages. The result: workloads can be sized with less DRAM, materially cutting hardware cost.

Broadcom’s claim: up to 40% TCO reduction. Given current DRAM pricing, this becomes one of the most economically impactful features in VCF 9.1.

The mechanism:

• The hypervisor identifies cold memory pages pages not recently accessed by the VM

• Cold pages are tiered down to NVMe SSDs

• When the VM accesses a tiered page, the hypervisor pulls it back into DRAM

• Active hot pages remain in DRAM at full performance

• The end result: VMs effectively have access to more memory than the host has DRAM, with cold-page access incurring NVMe-speed latency

Workload suitability:

• Large VMs with significant cold memory strong candidates

• Databases with deep buffer pools but predictable hot-data access strong candidates

• AI inference with cached models that aren’t all active simultaneously strong candidates

• VDI / virtual desktops with large memory allocations but mostly-idle workloads strong candidates

• Latency-sensitive workloads (real-time trading, telemetry processing) keep on full DRAM

• Memory-bandwidth-bound workloads keep on full DRAM

VCF Operations 9.1 surfaces Memory Tiering recommendations:

• Memory Tiering statistics in Manage > Optimize > Resource Efficiency

• Capacity insights that identify candidates for tiering

• Workload placement recommendations to optimise across the fleet

HLD considerations:

• Hardware: requires supported NVMe devices. Check VCF 9.1 HCL

• Sizing impact: a host with NVMe Memory Tiering can effectively run more memory-intensive workloads than a comparably-DRAM-sized host. Update workload-to-host ratios

• Workload classification: not every VM benefits. Use VCF Operations capacity insights to identify candidates rather than blanket-enabling

• Latency budget: confirm workload tolerates occasional NVMe-speed page-fault latency. For most VMs this is invisible; for some it matters

• Cost model update: capacity planning conversations change. Less DRAM purchased per host; more NVMe purchased. Total cost still drops

Diagram to use: the NVMe Memory Tiering architecture diagram showing hypervisor-driven tiering between DRAM and NVMe. Source:

• Broadcom Announcing VCF 9.1 (Memory Tiering section)

• Broadcom Scale, Simplify, and Secure VCF 9.1 (Memory Tiering capacity insights)

• Gibson Virtualization VCF 9.1 What’s New: VCF Operations (Memory Tiering statistics)

Pattern 5: Enhanced DirectPath I/O GPU-to-GPU RDMA for AI

Enhanced DirectPath I/O in VCF 9.1 is the architecture that makes production-grade distributed AI training viable on private cloud. The mechanism: GPU-to-GPU RDMA over RoCE, allowing GPUs in one VM to directly access GPUs in another VM’s memory across the network, bypassing the hypervisor data path.

Validated hardware:

• NVIDIA ConnectX-7 the predominant 400 Gbps NIC for AI fabrics

• NVIDIA BlueField-3 the DPU-class NIC, also supporting GPU RDMA

• AMD vIOMMU AMD’s equivalent of Intel VT-d, enabling DirectPath on AMD-based hosts

• AMD Instinct MI350 supported in 9.1 alongside NVIDIA H100/H200/B100

Why this matters: distributed deep learning workloads (multi-node training, model parallelism, large language model fine-tuning) need GPU-to-GPU bandwidth between hosts approaching bare-metal performance. Without RDMA, the network becomes the bottleneck. With it, VCF’s AI workloads scale across hosts as they would on bare metal.

Design topology for AI-ready VCF 9.1:

• ConnectX-7 or BlueField-3 NICs on AI hosts typically dual-NIC for redundancy and aggregate bandwidth

• RoCE-capable network fabric lossless ethernet with PFC (Priority-based Flow Control) and ECN (Explicit Congestion Notification). Typically dedicated AI fabric, separate from general data centre traffic

• DirectPath I/O profiles for AI workloads in vSphere GPUs and NICs passed through to AI VMs

• vGPU profiles for multi-tenant GPU sharing (where workloads tolerate sharing)

• NVIDIA Multi-Instance GPU (MIG) partitioning for fine-grain GPU allocation

• Co-location with NVIDIA AI Enterprise stack for the broader Private AI Foundation story

HLD considerations:

• Network fabric design is critical the RoCE fabric needs to be properly tuned for lossless behaviour. PFC + ECN configuration on every switch in the path

• PCIe topology matters GPU and NIC should be on the same NUMA node, ideally connected via the same PCIe switch, for optimal performance

• Cluster sizing for AI hosts differs from general-purpose hosts. AI hosts are typically GPU-rich (4–8 GPUs per host), heavy on PCIe lanes, and on dedicated AI fabric

• VKS or VM-based AI workloads both supported. VKS for cloud-native AI (Kubernetes-orchestrated), VMs for traditional model serving

• Storage: AI training data often lives on FlashBlade, FAS-NFS, or other high-throughput NFS. Plan storage path bandwidth alongside compute and network

Anti-patterns:

• Trying to do AI training on general-purpose VCF clusters GPU sharing on shared fabric breaks performance

• Skipping the RoCE network fabric tuning packets dropped under congestion destroy RDMA performance

• Using ConnectX-6 or older NICs the validated hardware is ConnectX-7 / BlueField-3 in 9.1

• Treating AI hosts as fungible compute they’re specialised hosts with specific HCL requirements

Diagram to use: the Enhanced DirectPath I/O architecture diagram showing GPU-to-GPU RDMA over RoCE. Source:

• Broadcom Announcing VCF 9.1 (Enhanced DirectPath section)

• Broadcom TechDocs Private AI Foundation Platform Models

• NVIDIA NVIDIA AI Enterprise + VMware reference architecture

Decision framework: composing a VCF 9.1 workload topology

A simplified sequence for compute, storage, and AI topology design decisions:

• Step 1: Identify workload classes traditional VMs, Kubernetes workloads, AI training, AI inference, mission-critical apps, dev/test, VDI

• Step 2: Decide on Supervisor deployment scope one Supervisor per Workload Domain (typical) or per cluster (larger scale)

• Step 3: Choose VKS cluster strategy cluster-per-team, cluster-per-app, cluster-per-environment, cluster-per-compliance-zone, or shared with namespace separation

• Step 4: Choose storage architecture per cluster vSAN ESA (default for new), vSAN OSA (existing migration target), or external storage (NetApp/Pure/Dell/HPE per Parts 10–13)

• Step 5: Enable Auto-RAID + Global Dedup + DaR encryption on new ESA clusters from Day 0

• Step 6: Identify NVMe Memory Tiering candidates via VCF Operations capacity insights enable selectively, not globally

• Step 7: Plan AI cluster topology separately dedicated hosts with ConnectX-7/BlueField-3, dedicated RoCE fabric, GPU-aware VKS / VM placement

• Step 8: Validate hardware against the VCF 9.1 HCL for every feature you’re using

Closing

VCF 9.1’s workload topology is the layer where the platform pays off. VKS at 500-cluster scale changes what Kubernetes can be used for on private cloud. vSAN ESA + Auto-RAID + Global Dedup + DaR encryption gives storage that’s structurally cheaper to operate. NVMe Memory Tiering quietly delivers the largest TCO improvement in the release. Enhanced DirectPath I/O makes distributed AI training viable on-prem. These aren’t features in isolation; they compose into an architecture.

In my design reviews, the workload topology questions I work through every time:

• Which workload classes does this customer run, and how many of each?

• Which need their own Workload Domain? Which can share?

• What’s the VKS cluster strategy by team, app, environment, or compliance zone?

• What’s the storage architecture vSAN ESA, OSA, external? Per workload class or globally?

• NVMe Memory Tiering enabled yes/no, and for which workload classes?

• AI workloads in scope dedicated hosts, dedicated fabric, dedicated VKS Project?

• Hardware HCL validated for the feature set you’re using?

• Storage policy alignment with workload performance requirements?

Get those answered and the workload topology HLD writes itself.

Part 17 of this Field Guide the final article in the series covers identity, security, operations, and recovery topology patterns: Identity Broker, Security Services Platform, the Build/Manage/Operate/Protect console, Fleet Update Service, Private AI Foundation, and Advanced Cyber Compliance with the on-prem Isolated Recovery Environment.

Sources

• Broadcom Announcing VCF 9.1: Modern Private Cloud Built for Efficiency and Resilience

• Broadcom Scale, Simplify, and Secure VCF 9.1

• Broadcom TechDocs vSphere Supervisor Design Library

• Broadcom Operations in VMware Cloud Foundation 9.0

• Broadcom TechDocs Architectural Options in VCF (Storage, Compute, AI Models)

• MyNetworker.blog vSphere Supervisor Networking with NSX VPC

• Gibson Virtualization VCF 9.1 What’s New: VCF Operations

• vStellar VCF-9 Part 1 Introduction & Architecture

• NVIDIA AI Enterprise + VMware reference architecture