VCF 9.1 Networking Deep Dive: VPCs, Transit Gateways, and the New Consumption Model

This is Part 4 of an architect’s read of VCF 9.1. Part 1 was the field guide across the twenty design changes. Part 2 was the fleet and instance HLD/LLD walkthrough. Part 3 was the lifecycle deep dive. This one is networking VPCs, Transit Gateways, the new consumption model, and what 9.1 actually changed.

If you’ve designed NSX networking in VCF 5.x or earlier, your mental model is Tier-0 → Tier-1 → segments, NSX Edges as mandatory infrastructure, segments managed via SDDC Manager or NSX UI, with multi-tenancy bolted on through NSX Projects when you needed it.

That mental model doesn’t apply anymore. VCF 9.x rebuilt the networking consumption model around Virtual Private Clouds as the primary unit, with Transit Gateways as the central routing element, and the option to skip NSX Edges entirely for many deployments. 9.1 then added multi-TGW, Connectivity Policies, IPSec VPN, the Virtual Network Appliance, span control, native Infoblox IPAM, and EVPN-VXLAN interoperability with the physical fabric.

For architects, this is the part of the platform that needs the most re-learning. The Tier-0 / Tier-1 / segment vocabulary still exists, but it’s no longer how you talk to tenants about networking. Talk about VPCs, projects, and Transit Gateways instead.

The networking problem VCF used to have

Pre-9.x VCF networking design:

• Tier-0 gateways at the edge, peered with the physical fabric via BGP

• Tier-1 gateways per workload domain or per tenant, providing north-south routing

• Segments as the unit of L2 broadcast domain created either at the provider level or inside an NSX Project for tenants

• NSX Edge VMs as mandatory infrastructure for any north-south traffic, deployed via SDDC Manager

• Multi-tenancy via NSX Projects, but with all of the above stacked behind it

This worked, but it leaked NSX concepts into every tenant conversation. Tenants asking for “a network” had to understand Tier-1, segments, and Edge clusters before they could reason about anything. Day-2 was complex modifying a segment, adding a route, changing a NAT rule typically required platform-level coordination.

VCF 9.x rebuilt the consumption model on top of NSX so tenants (and architects) deal in higher-order primitives. The underlying NSX is still there, but the abstraction layer above it is different.

The 9.x consumption model

The new model has three primary objects.

Virtual Private Cloud (VPC) an on-demand logical network construct. Each VPC has its own VPC Gateway (handles east-west routing within the VPC and north-south to the Transit Gateway), its own IP space, and its own subnets.

Transit Gateway (TGW) the central routing element inside a project. Connects multiple VPCs together for cross-VPC east-west traffic, and connects to the external network for north-south.

External Connection defines how the Transit Gateway reaches the physical/external network. Can be Centralized (via NSX Edges + Tier-0) or Distributed (directly to a VLAN, no Edge VMs).

A VPC has three types of subnets:

• Private subnets only accessible within the VPC. Not advertised externally

• Private Transit Gateway subnets accessible across all VPCs that share the same TGW. Used for shared services patterns

• Public subnets IP block allocated from the external pool and advertised via BGP. Reachable from the physical network

A VPC is configured through three profiles:

• VPC Connectivity Profile defines outside connectivity (which TGW, which IP Blocks, default outbound NAT)

• VPC Service Profile defines DHCP, DHCP Relay, subnet profiles

• VPC Security Profile Gateway Firewall configuration (stateless GFW comes with VCF entitlement; stateful GFW/DFW require vDefend license)

The consumption surface for VPCs is deliberately broad:

• vCenter tenants can create VPCs and subnets directly from the Networks tab

• NSX Manager admins and advanced users can configure VPCs and projects directly

• VCF Automation IaC and self-service catalog for VPCs

• VCF Operations monitoring and observability of VPC networking

This is the “cloud networking” consumption surface tenants interact with VPCs the way they would in AWS or Azure, not with raw NSX constructs.

Projects: the multi-tenancy boundary

Projects are NSX’s multi-tenancy primitive. Each Project owns its own Transit Gateway (in 9.0, one TGW per project; in 9.1, multiple TGWs per project more on this below).

VCF deployments come with a default Project that’s always present and cannot be deleted. VPCs in the default project can be created and managed from vCenter this is the common consumption surface for non-multi-tenant deployments.

Custom Projects are for true multi-tenancy. Each project can represent a tenant, a business unit, an environment, or a regulatory zone. Resources (external connections, IP blocks, Edge nodes) need to be explicitly shared into a custom project unlike the default project, which has implicit access to all platform resources.

For most enterprise deployments, the default project plus custom projects for distinct tenants is the right pattern.

Centralized vs Distributed Transit Gateway

This is the big design choice in 9.x networking. Both modes are supported, with different trade-offs.

Centralized Transit Gateway (CTGW) the more traditional path:

• NSX Edge VMs are deployed (minimum 2 per Edge cluster)

• Tier-0 gateway sits on the Edge cluster

• Transit Gateway Service Router runs on the same Edge VMs as the Tier-0 SR

• North-south traffic flows through the Edges

• Supports all NSX services: stateful firewall, full load balancer, IPSec VPN, advanced NAT

• Required for VCF Automation All Apps and vCenter Supervisor per published guidance

CTGW is the right choice when:

• You need stateful services for tenant traffic

• You need VCF Automation full functionality

• You’re running VKS / vCenter Supervisor

• You want active/active or active/standby HA modes for tenant networking

Distributed Transit Gateway (DTGW) the edgeless model:

• No NSX Edge VMs required for VPC external connectivity

• Transit Gateway runs directly on ESX hosts using existing TEP uplinks

• Tier-0 setup is skipped VPC connects directly to a VLAN on the physical network

• Hard requirement: every ESX host must attach to the same VLAN on the physical network

• Supports only distributed services: stateless 1:1 NAT (in 9.0), 1:N SNAT (new in 9.1), distributed DHCP

• North-south traffic goes directly from the ESX host to the physical network

DTGW is the right choice when:

• Edge VMs are operationally expensive (cost, capacity, or lifecycle overhead)

• Stateless services are sufficient for the tenant

• You want a closer-to-cloud-native distributed routing model

• The physical network can support the VLAN-attach requirement

A common pattern in 9.x designs is hybrid CTGW projects for tenants that need stateful services or VKS, DTGW projects for tenants where edgeless is sufficient. Both can coexist in the same VCF instance.

What 9.1 actually changed in networking

VCF 9.1 layered a substantial number of new networking capabilities on top of the 9.0 foundation. The ones I’d flag in any HLD review:

Multiple Transit Gateways per project. In 9.0, each project had exactly one TGW. In 9.1, projects can have multiple TGWs, each with multiple connections. This changes the multi-tenancy and topology design substantially you can now build segmented routing planes within a single project (think: production vs staging traffic in the same tenant kept on separate TGWs).

Connectivity Policies. New 9.1 feature for VPC-to-VPC communication patterns within the same Transit Gateway. Three modes:

• Community of VPCs VPCs grouped together that should communicate (e.g., connected applications)

• Isolated VPCs restricted patterns (e.g., DMZ-style isolation)

• Promiscuous VPCs service-host VPCs that can communicate with all other VPCs (e.g., shared services)

Connectivity Policies replace what used to be manual route table manipulation and firewall rule design at the platform level. Tenants and operators define communication intent at a higher level; NSX implements it underneath.

Span control for VPCs and Transit Gateways. New 9.1 capability the span of VPCs and TGWs can be defined from vCenter or NSX, letting them be restricted to specific clusters or span all vCenter instances of the networking domain. This is significant for HLD design: previously, VPC span was implicit; now it’s an explicit design input.

IPSec VPN on Centralized Transit Gateway. 9.1 adds IPSec VPN support for VPCs using centralized external connectivity. For tenant designs that need encrypted tunnels to remote sites or external services, this is now a first-class capability rather than a workaround through external Tier-0 services.

1:N SNAT for Distributed Transit Gateway. 9.0’s DTGW supported only 1:1 NAT. 9.1 adds 1:N SNAT (PAT), letting multiple internal VMs share a single external IP. This significantly improves external connectivity efficiency for distributed VPC deployments and reduces external IP consumption.

Virtual Network Appliance (VNA). New 9.1 construct an appliance designed to run network services within distributed VPC environments. Provides the foundation for advanced services (load balancing, future service types) in the edgeless model. The first service available on VNA is L4 load balancing.

Load Balancer on VNA. L4 load balancing for distributed VPCs runs on the VNA. Lets distributed VPCs support tenant load balancing without going back to centralized Edges.

Native Infoblox IPAM and DNS integration. Covered in Part 1, worth re-flagging here. NSX discovers Infoblox Network Views, DNS Views, and Network Containers directly. Subnets are created from Network Containers; VM IP/FQDN updates flow back into Infoblox. IP Blocks now support up to 10 CIDRs, 10 ranges, and explicit IP exclusions per block.

EVPN-VXLAN interoperability with the physical fabric. Also covered in Part 1. NSX in 9.1 speaks EVPN-VXLAN with the data centre fabric, with a dedicated integration for Arista Networks. Overlay and underlay now share a single protocol vocabulary instead of being stitched at L2/L3 boundaries.

IPv4/IPv6 dual stack support. VCF 9.1 improves the Day 0 experience with support for IPv4/IPv6 dual stack configurations. Combined with IPv6-only Guest Customization (Part 1), VCF networking now supports the full range of v4-only, dual-stack, and v6-only patterns.

LACP-based designs. 9.1 adds support for LACP-based designs in the installer, simplifying setups that previously required post-deployment manual configuration.

Edge Cluster simplified UI. New, user-friendly UI workflow in NSX Manager for defining Edge Node and Edge Cluster configurations. Faster, more intuitive deployment of Edge infrastructure.

Bare-metal Edge NIC support. Broadcom 574X/575X and Mellanox CX6 LX support for BM Edge datapath NIC relevant for high-throughput Edge deployments.

Subnets shared across Namespaces. For VKS workloads, subnets can now be shared across multiple Kubernetes namespaces useful for shared-services patterns in K8s tenancy.

Terraform Provider updates. The NSX Terraform Provider evolves with 9.1 to expose multi-TGW, advanced connectivity options, VLAN Extension, and improved IPAM capabilities.

Fleet networking and latency the cross-instance layer

Step back from per-tenant VPC design and look at the fleet. From Part 2, we know fleet-level services (VCF Operations, Automation, Identity Broker, Management Services Cluster) live in the management domain of the primary instance. When the fleet spans multiple instances especially across regions the network design has to account for fleet-level traffic.

The constraint is fleet latency. Broadcom publishes the VCF Fleet Latency Diagram (at ports.broadcom.com under Network Diagrams) and updated it for 9.1 with the new constructs. Before committing to a multi-instance, multi-region fleet design, validate round-trip latency against the published budgets.

Other fleet-network considerations:

• MTU consistency across the underlay between instance management planes

• Routing between management domains in different instances (typically BGP-peered to the same upstream)

• DNS fleet-level components and instance components must resolve each other consistently

• Time synchronization NTP across the fleet must be tight, especially with identity and certificate workflows

For multi-region fleet designs, the LLD must capture these as inputs, not assumptions.

HLD/LLD design patterns

Three common shapes worth naming.

Pattern 1: Single project, single VPC, single tenant. Simplest design. Default project, one VPC, vCenter-consumed. CTGW or DTGW depending on stateful service requirements. Common for small-to-medium enterprise deployments where multi-tenancy isn’t a real requirement.

Pattern 2: Default project plus custom projects per tenant. Multi-tenant design. Each tenant gets a custom project with their own TGW (or multiple in 9.1), their own VPCs, their own external connectivity. Self-service surface via vCenter, NSX, or VCF Automation. Common for service providers and large enterprises with strict tenant isolation.

Pattern 3: Hybrid CTGW/DTGW. Some projects use CTGW (for stateful services, VKS, VCF Automation full functionality); others use DTGW (for edgeless deployments where stateless services suffice). Both coexist in the same VCF instance. Common in environments balancing operational simplicity with service requirements.

For multi-instance fleets, layer the fleet-latency check on top of any of these patterns.

Common HLD/LLD anti-patterns

Designing for Tier-0/Tier-1/segments. Old habit. In 9.x, design for VPCs, TGWs, and projects. Tier-0 still exists but it’s behind the curtain. Tenants don’t care about it; your HLD shouldn’t lead with it.

Defaulting to CTGW without considering DTGW. Edge VMs are operationally expensive. If your tenant only needs stateless services, DTGW is genuinely lighter. At least evaluate it.

Skipping the VLAN-attach requirement for DTGW. Every ESX host needs to attach to the same VLAN on the physical network for DTGW external connectivity. If your fabric design doesn’t support this consistently, DTGW isn’t viable. Validate early.

Not planning for 9.1 multi-TGW. If you architected a 9.0 project with the implicit “one TGW per project” assumption, your 9.1 design can be more nuanced. Use multiple TGWs where it improves topology clarity.

Ignoring fleet latency. Multi-region fleet designs that skip the latency check are setting up operational pain. The Fleet Latency Diagram is published use it.

Not integrating Infoblox where it already exists. Customers standardised on Infoblox have been maintaining custom integration glue for years. The 9.1 native integration removes that. Don’t leave the integration as a Day-2 afterthought; specify it in the HLD.

Designing without span control. 9.1’s explicit VPC and TGW span control is a design lever. Use it to scope networking to the right clusters or vCenter instances rather than defaulting to fleet-wide span.

Decision framework: where does this network requirement live?

When a requirement arrives, work through the hierarchy:

• New subnet for an existing tenant add a subnet to an existing VPC

• New isolated network for an existing tenant add a new VPC to the tenant’s existing project

• New cross-VPC communication pattern add a Connectivity Policy or use Private Transit Gateway subnets

• New tenant add a custom Project with its own TGW(s), VPCs, and (if needed) external connectivity

• New routing plane within an existing tenant in 9.1, add a second TGW to the existing project

• New external connectivity model add a new External Connection (centralised or distributed)

• New region depending on scope, either a new VPC in the existing fleet, or a new instance with its own networking, or a new fleet entirely

This sequence keeps the design grounded in the consumption model rather than reverting to NSX-T primitives.

Closing

VCF 9.x changed how networking is consumed. 9.1 finishes the architectural picture with multi-TGW, Connectivity Policies, the Virtual Network Appliance, IPSec VPN, span control, native Infoblox IPAM, and EVPN-VXLAN interop with the physical fabric. The networking chapter in an HLD stops being a tour of NSX-T primitives and becomes a tour of VPCs and the consumption model that sits on top.

For my own design reviews, the networking questions I work through every time:

• Which projects, and how do they map to tenants or zones?

• Which VPCs in each project, and what’s their span?

• Centralised or distributed Transit Gateway and why?

• Connectivity Policies between VPCs?

• External connectivity model: BGP-peered Tier-0, distributed VLAN-attach, or hybrid?

• IPAM source of truth: NSX-native or Infoblox?

• Fleet latency budget for any multi-instance design?

Get those answered and the networking LLD writes itself.

That closes Part 4. Together with the Architect’s Field Guide (Part 1), the Fleet and Instance HLD/LLD (Part 2), and the Lifecycle Deep Dive (Part 3), this is the four-part read of VCF 9.1 I’d hand to a senior architect coming to the platform fresh.

Sources

• VMware Virtual Private Cloud in VMware Cloud Foundation 9.0 (Broadcom)

• VPC Distributed Network Connectivity No NSX Edge VMs (Broadcom)

• VPC Centralized Network Connectivity with Guided Edge Deployment (Broadcom)

• VCF 9.1 What’s New: NSX (Broadcom TechDocs)

• Virtual Private Cloud in NSX (Broadcom TechDocs)

• Simplify Workload Connectivity and Enhance Network Scale and Performance with VCF 9.1 (Broadcom)

• VCF 9 NSX VPC Part 1: Centralized Transit Gateway (Daniel Krieger / evoila)

• VCF 9 NSX VPC Part 2: Distributed Transit Gateway (Daniel Krieger / evoila)

• VCF 9 NSX VPC Part 1: Centralized Transit Gateway (SDN Warrior)

• VCF-9 Part 3: Networking Models (vStellar)

• VCF-9 Part 4: NSX Edge Cluster Deployment (vStellar)

• VCF 9.1 Updated Design Blueprints and Fleet Latency Diagrams (William Lam)