VCF 9.1 Design Topology Field Guide Part 2: Network Topology Patterns

This is Part 15 of an architect’s read of VCF 9.1, and the second of four Design Topology Field Guide articles. Part 14 covered foundational topology patterns. This one covers networking the area where VCF 9.x makes the most fundamental architectural shift over previous releases.

VCF 9.x ships AWS-style VPCs as the native consumption model for NSX. Projects replace ad-hoc tenant constructs. Default Transit Gateways replace direct Tier-0 attachment. Centralized and Distributed Transit Gateway options give architects a real footprint-vs-features trade-off. VCF 9.1 adds multiple External Connections per Transit Gateway. VPC Connectivity Policies introduce a governance layer for tenant networking.

For network architects coming from previous NSX experience, this is a substantial mental model change. For application architects, it’s the first time on-prem networking has looked genuinely cloud-native. This article walks through the four key patterns.

Pattern 1: The VPC Consumption Model

VCF 9.1 NSX consumption splits responsibilities between Provider and Tenant. The Provider role configures the underlying networking; the Tenant consumes it via VPCs without needing networking expertise.

The construct hierarchy:

• Project the tenant or department boundary in NSX. A Project owns one or more VPCs. The default Project exists for non-tenanted infrastructure use

• VPC a logical isolated environment within a Project. Can be a test/staging/prod separation, a per-team allocation, or any other logical grouping. New in VCF 9: the default Project can now also have one or more VPCs

• Default Transit Gateway the central routing hub for each Project. All VPCs in the Project connect to it. Replaces direct Tier-0/VRF attachment from NSX 4.x. At present, only one Transit Gateway per Project (though there are signals this may evolve)

• Subnets within each VPC. Types: Private (NAT required to reach external), Public (routable north of the TGW), Isolated (no external routing). Private Transit Gateway subnets are new in VCF 9 routed via the Default Transit Gateway, useful for shared services accessible across VPCs

Provider responsibilities (Enterprise Admin role):

• Defining External Connections the abstracted paths to the physical network

• Setting up Project structure which tenants get their own Project boundary

• Configuring connectivity profiles and IP block allocations

• Setting policy boundaries for what tenants can self-service

Tenant responsibilities:

• Creating VPCs within their Project

• Allocating subnets (Private, Public, Isolated)

• Configuring NAT and external IP allocation within their VPCs

• Self-servicing connectivity changes within their VPCs

Architectural significance:

• On-prem networking finally looks cloud-native mental model maps cleanly to AWS VPC patterns

• Provider/tenant separation enables platform-team self-service the network team isn’t in the loop for every tenant change

• VKS Supervisor and Kubernetes workloads consume the same VPC primitives unified networking across VMs and containers

• vSphere admin can configure VPCs from vCenter (the Network tab > Networks > Network Connectivity), no longer needing NSX Manager for basic VPC operations

Diagram to use: the Project / VPC / Default Transit Gateway hierarchy diagram. Source:

• SDN-Warrior VCF 9 NSX VPC Part 1: Centralized Transit Gateway

• evoila Blog VCF 9 NSX VPC Part 1

• vStellar VCF-9 Part 3: Networking Models

Pattern 2: Centralized vs Distributed Transit Gateway

The Transit Gateway design decision is the most consequential network topology choice in VCF 9.1. It determines whether you need NSX Edge VMs and what services are available on north-south traffic.

Centralized Transit Gateway. The TGW Service Router (SR) component is collocated on the same NSX Edge VMs that host the upstream Tier-0 Gateway. The TGW inherits its HA mode (Active/Standby or Active/Active) from the Tier-0. Each TGW SR is tightly coupled with its corresponding Tier-0 SR.

Centralized works when:

• You need stateful services on north-south traffic NAT (default outbound NAT requires Active/Standby or Active/Active Stateful T0), load balancing, firewall services

• You’re willing to deploy NSX Edge VMs as part of the design footprint

• You want full feature parity with traditional NSX deployments

Distributed Transit Gateway. The TGW connects directly to the physical network via VLAN. No NSX Edge VMs required. The External Connection is created as a Distributed Connection rather than a Centralized one.

Distributed works when:

• You don’t need stateful services on north-south only static routing required (NSX 9.x distributed currently only supports static routing)

• You want to eliminate NSX Edge VM footprint

• You can configure the physical network for the VLAN and external IP block routing

• Only manual static NAT is acceptable (no default outbound NAT)

Service supportability comparison (at a high level):

• Centralized: full stateful services, dynamic routing, load balancing, default outbound NAT

• Distributed: static routing only, manual static NAT only, no stateful services

HLD considerations:

• Mixed designs are common production Project gets Centralized for full services, dev/test Project gets Distributed for footprint efficiency

• The TGW SR is distributed across the same NSX Edge VMs that host the Tier-0 SR for Centralized sizing the Edge cluster matters

• VPC Gateway (the implicit gateway inside each VPC) can use a separate NSX edge cluster from the one hosting the TGW/Tier-0 tier isolation by Edge cluster matters for large designs

• Distributed Connection requires VLAN and gateway in the physical network, with external IP block routable from your core router

Anti-patterns:

• Choosing Distributed and then needing stateful services later the path-of-least-resistance is to choose Centralized when in doubt, then optimise to Distributed where confirmed unnecessary

• Mixing Centralized and Distributed within the same VPC architect at the External Connection level

• Forgetting that in VCF 9, Edges are no longer deployed via SDDC Manager as in VCF 5.x they’re deployed via Network Connectivity in vCenter or NSX Manager. This changes the workflow

Diagram to use: the side-by-side Centralized vs Distributed Transit Gateway comparison. Source:

• SDN-Warrior VCF 9 NSX VPC Part 1: Centralized Transit Gateway

• SDN-Warrior VCF 9 NSX VPC Part 2: Distributed Transit Gateway

• Broadcom VPC Distributed Network Connectivity (No NSX Edge VMs)

Pattern 3: Multiple External Connections per Transit Gateway (New in VCF 9.1)

VCF 9.1 removes one of the most restrictive constraints from the 9.0 networking model: a Transit Gateway is no longer limited to one default External Connection. Tenants can now attach a TGW to multiple External Connections, with route-based selection driving which traffic uses which path.

The mechanism is the new Remote Networks field on each External Connection:

• When no Remote Networks is specified, the External Connection provides a default route (the only pre-9.1 option still available, still the most common pattern)

• When specific Remote Networks are listed, those routes determine that traffic from those networks should use that External Connection

• Multiple External Connections can attach to one TGW with different Remote Networks specifications

Why this changes design topology:

• Multi-egress designs become native different traffic classes can exit through different physical egress paths

• Provider/tenant separation evolves the provider pre-creates multiple External Connections; tenants self-select via routing rather than requesting new connectivity each time

• Compliance topologies simplify regulated traffic can route through dedicated paths while general traffic uses the default

• Hub-and-spoke connectivity to multiple downstream networks becomes a clean design pattern

Real-world use cases that benefit:

• Customer with separate internet egress and corporate WAN egress traffic destined for corporate networks uses the WAN connection, everything else uses the internet egress default

• Customer with regulatory-isolated egress for specific workloads PCI workloads exit through a dedicated path

• Service provider with per-tenant or per-tier egress paths

• Customer migrating between WAN providers dual-attach both connections during the migration window

Other VCF 9.1 networking additions worth noting in the same design conversation:

• Private subnet extension extending private subnets across External Connections

• Provider-managed SNAT the provider can manage SNAT centrally rather than per-tenant

• Additional flexibility for route selection at the External Connection level

• Centralized Transit Gateway now supports IPSec VPN connections

• 1:N SNAT for Distributed Transit Gateway

HLD considerations:

• Most environments won’t need multi-External-Connection the simple default-route pattern remains the right answer for most VPCs

• When you do need it: be explicit about which Remote Networks go to which External Connection, document the routing intent

• Verify the physical network supports the route advertisements you need NSX 9.x distributed only supports static routing

• This is an advanced VPC configuration most tenants won’t interact with it directly. Provider may use it for specific designs

Diagram to use: the VCF 9.1 multiple External Connections diagram. Source:

• Broadcom Transit Gateway Connectivity Options in VCF 9.1

• Broadcom VPC Distributed Network Connectivity

Pattern 4: VPC Connectivity Policies Communities, Isolated, Promiscuous

VPC connectivity in VCF 9.1 isn’t binary. The platform introduces Connectivity Policies as a first-class governance layer, with three modes that define how VPCs in a Project can communicate with each other.

Community. VPCs that share a Community can communicate with each other. The on-prem equivalent of AWS Transit Gateway Communities. Useful for grouping VPCs that need mutual connectivity all production VPCs in one Community, all dev VPCs in another, with no cross-Community communication.

Isolated. The VPC cannot communicate with other VPCs in the Project. Useful for sensitive workloads that must not be reachable from peer VPCs PCI environments, regulated data stores, security tools. The VPC can still communicate externally if it has appropriate External Connection routing.

Promiscuous. The VPC can communicate with any other VPC in the Project. Useful for shared services VPCs AD, DNS, NTP, monitoring, security tooling that need to be reached by everything. The shared services VPC is Promiscuous; the consuming VPCs talk to it but maintain their own connectivity policies for east-west between themselves.

Combined with Span control the visibility and scope of VPCs across Projects you get a multi-tenant network governance model that maps cleanly to cloud-native consumption patterns.

Design patterns that use Connectivity Policies effectively:

• Tiered environment separation prod and non-prod in separate Communities, no cross-Community traffic, shared services VPC in Promiscuous mode for ops tooling access

• PCI-DSS scope isolation PCI workload VPC in Isolated mode, with explicit External Connection routing for required upstream connectivity only

• Multi-tenant service provider each customer’s VPCs in their own Project, with selective Communities for shared services

• Acquisition isolation newly acquired company’s workloads in Isolated VPCs while integration policy is being defined, transition to Communities once integration is complete

HLD considerations:

• Map workload security and compliance boundaries to Connectivity Policies during design it’s much harder to add segmentation later

• Document which VPCs use which Connectivity Policy and why this is the network governance documentation

• Combine Connectivity Policies with DFW for layered enforcement Connectivity Policies for VPC-level reachability, DFW for VM-level rules

• Don’t over-Promiscuous every Promiscuous VPC becomes a potential lateral movement vector. Limit Promiscuous to genuine shared services

Anti-patterns:

• Defaulting all VPCs to Promiscuous “until we figure out the policy” you’ll never tighten it. Default to Isolated and open as needed

• Using Connectivity Policies as a substitute for DFW they’re complementary, not interchangeable

• Skipping Span control across-Project visibility needs explicit decisions, not implicit defaults

Diagram to use: the VPC Connectivity Policies diagram showing Community, Isolated, and Promiscuous modes. Source:

• Broadcom Transit Gateway Connectivity Options in VCF 9.1

• Broadcom VPC Distributed Network Connectivity

• vStellar VCF-9 Part 3: Networking Models

Decision framework: composing a VCF 9.1 network topology

A simplified sequence for the network topology design decisions in this article:

• Step 1: Map tenants to NSX Projects. One Project per tenant or per major workload class. The default Project covers shared infrastructure

• Step 2: Within each Project, define VPCs. One VPC per logical environment (prod/staging/dev), per application tier, or per security boundary

• Step 3: Choose Centralized or Distributed for each Project’s External Connection(s). Centralized for stateful services; Distributed for footprint efficiency

• Step 4: Decide whether multiple External Connections per TGW are needed (most: no). If yes, plan Remote Networks per connection

• Step 5: Apply Connectivity Policies to each VPC Community / Isolated / Promiscuous. Default to Isolated and open as needed

• Step 6: Layer DFW on top for VM-level enforcement (covered in Part 5 of the main series)

• Step 7: Validate north-south path requirements NAT, load balancing, IPSec VPN, BGP against Centralized vs Distributed feature support

Closing

VCF 9.1’s networking model is the most cloud-native NSX has ever been. The VPC consumption model gives tenants AWS-style self-service. Centralized vs Distributed TGW gives architects a real footprint-vs-features trade-off. Multiple External Connections per TGW (new in 9.1) opens multi-egress designs. VPC Connectivity Policies give the governance layer that used to require complex DFW configurations.

In my design reviews, the network topology questions I work through every time:

• How are tenants modelled Projects per tenant, per workload class, or per business unit?

• How are VPCs scoped per environment, per application, per security tier?

• For each Project: Centralized or Distributed TGW? What stateful services do they need?

• Any need for multiple External Connections per TGW?

• Connectivity Policy default typically Isolated, with Communities and Promiscuous opened explicitly

• Shared services VPC where does AD, DNS, monitoring live? Promiscuous, by design

• PCI / regulated workloads Isolated VPCs with specific External Connections?

• Edge VM sizing for Centralized TGW host count, Edge cluster placement, DSwitch design

Get those answered and the network topology HLD writes itself.

Part 16 of this Field Guide covers compute, storage, and AI topology patterns vSphere Supervisor with NSX VPC, VKS scale, vSAN ESA + Auto-RAID, NVMe Memory Tiering, and Enhanced DirectPath I/O for AI.

Sources

• Broadcom Transit Gateway Connectivity Options in VCF 9.1

• Broadcom VPC Distributed Network Connectivity (No NSX Edge VMs)

• SDN-Warrior VCF 9 NSX VPC Part 1: Centralized Transit Gateway

• SDN-Warrior VCF 9 NSX VPC Part 2: Distributed Transit Gateway

• evoila Blog VCF 9 NSX VPC Part 1: Centralized Transit Gateway

• vStellar VCF-9 Part 3: Networking Models

• MyNetworker.blog vSphere Supervisor Networking with NSX VPC VCF 9

• vStellar VCF 9.1 Home Lab Series Part 6: Configure Network Connectivity