VCF 9.1 Upgrade Series Part 5: Post-Upgrade Operations, Validation & Rollback
This is Part 23 of the series — the fifth and final article of the Upgrade Series, and the closing piece of the broader 23-part read of VCF 9.1. Parts 19–22 covered pre-upgrade readiness, the VCF 5.x to 9.1 major version path, the 9.0 to 9.1 minor version path, and Brownfield Import. This article covers what comes after the upgrade completes.
In my experience, the difference between a successful upgrade and a problematic one is rarely the upgrade itself. Both upgrades complete the same technical steps. The difference is what happens after: how thoroughly is the upgrade validated, how deliberately are new capabilities enabled, how cleanly does the team transition to the new operational model, how prepared is the environment for the next upgrade.
This article walks through the post-upgrade work that turns a completed upgrade into a working environment: immediate validation, the 30–90 day capability enablement roadmap, rollback procedures and their limitations, configuration drift baseline capture, the Day-2 operating model transition, and the lessons-learned capture that improves the next engagement.
Immediate post-upgrade validation — the smoke test
The first hour after upgrade completion is for smoke tests. The minimum validation set proves that core platform functions work. Anything that fails here is a Day-1 incident, not a Day-2 issue.
Smoke test checklist:
• VCF Operations console accessible. Build / Manage / Operate / Protect pillars all loading. No console errors
• SDDC Manager UI accessible. Fleet inventory accurate. No outstanding alerts on critical components
• All vCenters accessible. Inventory render correct. Host connectivity restored. VM operations work (power, vMotion, snapshot)
• NSX Managers healthy. Edge nodes operational. Overlay segments connected. DFW rules in effect
• All ESX clusters healthy. Hosts on the expected ESX version. vSAN reporting healthy where applicable. Storage paths active
• Identity Broker functional. Federated SSO working. User login confirmed for representative roles
• Management Services Cluster healthy. K8s control plane responsive. Worker nodes ready. Service components running
• Critical workloads running. Power state matches pre-upgrade state. Application teams confirming basic application functionality
• Backup software responsive against the new environment. At least one workload backup job kicks off successfully
• Monitoring integrations sending data. Third-party tools (SIEM, observability) receiving expected telemetry
Any smoke test failure triggers immediate investigation. Most failures in this window trace to DNS issues, certificate problems, network configuration drift, or service startup timing. Identify, fix, re-validate. Do not declare the upgrade complete until the smoke tests pass.
First 48 hours — deep validation
Beyond smoke tests, the first 48 hours need deeper validation to catch issues that don’t surface in basic functional tests.
Deep validation activities:
Performance baseline capture. Throughput, latency, IOPS, vMotion timing, backup window timing — capture in the new environment for comparison with the pre-upgrade baseline. Performance regressions are most likely to surface in the first 48 hours under real production load.
Storage health validation. vSAN resync activity, datastore capacity reporting, snapshot integrity, replication status. Verify storage paths are stable under load.
Network deep validation. DFW logging at expected rates, NSX overlay traffic patterns, vmkernel traffic, NSX Edge throughput, BGP/static route propagation.
Identity and RBAC validation. Walk through representative user journeys for each major role: platform admin, application owner, operator, auditor. Confirm RBAC permissions match expectations.
Lifecycle operation testing. A non-impacting test of Fleet Update Service — e.g., a no-op cluster compliance check, a host configuration query — confirms the lifecycle plane is responsive.
Test restore validation. Within 7 days of upgrade completion, perform a test restore from backup to confirm the backup chain is intact against the new environment.
Application owner walkthrough. With representative application teams, walk through their applications’ critical operations. Application-level functional validation catches issues that infrastructure-level checks miss.
Rollback — when it’s possible and when it isn’t
Rollback isn’t always possible. Once certain upgrade steps complete, the environment cannot be cleanly reverted to the previous state. Understanding the rollback boundaries is essential because it tells you when continuation forward is the only safe path.
Rollback feasibility by upgrade phase:
Pre-upgrade: full rollback. Nothing has changed. Cancel the upgrade window. No technical risk to rolling back the decision.
ALSM upgrade only: rollback via snapshot. ALSM snapshot taken pre-upgrade can be restored. Other components untouched.
Aria-to-VCF Operations transition: limited rollback. Aria Operations cannot be restored after the transition is committed. If the transition fails mid-stream, the surviving state may be neither fully old nor fully new. Restore from backup if needed.
vCenter upgraded to v9: rollback via image restore. Image-based backup of vCenter can be restored. File-based backup may not restore cleanly to the pre-upgrade vCenter binary. Plan image-based backup before vCenter upgrade.
ESX upgraded: rollback per-host via reimage. ESX rollback is reimage-based at the host level. Practically infeasible for many hosts simultaneously. ESX upgrade is effectively a one-way operation at fleet scale.
Brownfield Convert / Brownfield Import committed: no rollback. Once SDDC Manager 9.x has recognised and integrated the environment, the Management Services Cluster deployed, Identity Broker activated, and Workload Domains established — there is no rollback path. Forward is the only direction.
This means: the decision point for rollback is BEFORE the conversion step, not after. If you discover issues during conversion that warrant rollback, do not let the conversion complete — pause, investigate, decide. Once it completes, you’re committed.
For the 9.0 to 9.1 minor version upgrade, the rollback story is different. The architectural state doesn’t change; the version does. Patch rollback is technically possible for some components (vCenter image revert, ESX cluster image revert) but not all. Plan rollback decisions per component, with explicit cutoff criteria.
Configuration drift baseline capture
Post-upgrade is the right time to capture a new configuration baseline for drift detection. The environment is in a known-good state — the closest you’ll get to a clean reference for the next 6–12 months.
Baseline capture activities:
• Cluster image definitions documented for each Workload Domain. Image content, firmware levels, ESX version, component versions
• vCenter configuration export — inventory, advanced settings, custom roles
• NSX configuration export — policies, profiles, segments, Edge configurations
• SDDC Manager configuration export — fleet state, lifecycle baselines
• VCF Operations dashboard snapshots — baseline alert rates, key metrics ranges, normal operating envelope
• Identity Broker configuration — IDP federation settings, group mappings, role assignments
• Backup configuration documentation — jobs, schedules, retention, targets
VCF Operations’ Configuration Drift Management feature uses this baseline to detect drift over time. Configuration changes that diverge from the baseline are surfaced — some intentional, some not. Without a current baseline, drift detection is operating against stale reference and missing real changes.
The 30–90 day new-capability enablement roadmap
The upgrade unlocked capabilities; the rollout of those capabilities is a project of its own. Enabling everything on Day 1 is usually wrong — the operational risk compounds, the team doesn’t have time to absorb the new features, and validation gets lost in the noise.
A typical 90-day enablement roadmap:
Days 1–30: Operational stability and baseline. Focus on validating that the upgraded environment is stable. Capture configuration baseline, performance baseline, alert baseline. Train the platform team on the new operational surface (Build / Manage / Operate / Protect pillars). Address any post-upgrade issues. Don’t enable new capabilities yet.
Days 31–60: Quick-win capabilities. Enable capabilities that deliver value with low operational risk: programmatic licence management, Real-Time Metrics, Active Findings as primary alerting model, VCF Operations dashboard refresh. These improve operations without changing workload behaviour.
Days 61–90: Storage and compute optimisation. NVMe Memory Tiering enablement on candidate workloads. vSAN ESA Global Deduplication on ESA clusters. Auto-RAID policy review. These deliver TCO improvements and require validation of workload performance impact.
Days 91+: Strategic capabilities. vDefend deployment per the DFW 1-2-3-4 maturity workflow. ACC + IRE deployment if licensed. Private AI Foundation rollout where AI workloads are in scope. These are programmes of their own — not single Day-2 actions.
Adjust the roadmap to customer priorities. A customer urgently needing cyber resilience improvements may compress the ACC + IRE timeline. A customer with AI workloads ready to deploy may prioritise Private AI Foundation enablement. The principle is deliberate sequencing, not particular dates.
Day-2 operating model transition
The upgraded environment requires a refreshed operating model. The team that ran vSphere + NSX-T + Aria for years (in the Brownfield Import case) or VCF 5.x (in the major version upgrade case) now operates VCF 9.1. The platform changes; the operating practices need to change with it.
Day-2 transition activities:
Operational documentation refresh. Runbooks, knowledge base articles, troubleshooting guides updated to reflect the new platform. Where the previous workflow goes through Aria Operations, the new workflow goes through VCF Operations. Where the previous workflow involved per-product consoles, the new workflow uses the four-pillar console.
Service desk categorisation update. Ticket categories, escalation paths, and routing rules may need adjustment. Issues that previously logged against Aria Operations now log against VCF Operations — the categorisation should reflect that.
Training plan execution. The platform team needs training on the new operational surface. Customers without recent training will be slower to adopt new capabilities and more prone to errors. Schedule formal training (HOL, Broadcom courses, partner-led) and supplement with internal knowledge transfer.
Change management process update. The new lifecycle capabilities (Fleet Update Service, Quick Patch, Live Patching) change what “change” means. Sub-minute vCenter downtime doesn’t need the same change management overhead as 4-hour windows. Adjust change management governance to the new realities.
Monitoring integration validation. SIEM forwarders, observability tool integrations, ticketing integrations — all may need configuration updates against the new platform APIs and event sources. Validate end-to-end.
Customer team operating rhythm. Establish a regular cadence for VCF Operations review — weekly health review, monthly capacity review, quarterly capability review. The new operational surface enables observability the team didn’t have before; turning that observability into operational practice takes deliberate work.
Lessons-learned capture
Every upgrade is also a learning opportunity. The customer’s next upgrade benefits from the lessons of this one. The partner team’s next engagement benefits from documenting what worked and what didn’t.
Lessons-learned topics worth capturing:
• Time estimates per phase — how long did each upgrade phase actually take, versus the plan?
• Unexpected issues — what failures occurred that weren’t in the risk register?
• Customer-specific gotchas — customer configurations, integrations, processes that required special handling
• Stakeholder communication — what worked, what didn’t, what would be done differently
• Validation gaps — what wasn’t validated that should have been, what was over-validated
• Hardware HCL surprises — components on the HCL that turned out to have specific firmware requirements or interaction issues
• Tool gaps — places where automation could have helped, where manual work was needed unexpectedly
• Customer team capability — where the team was prepared and where it wasn’t. Drives the next training plan
A retrospective meeting within 30 days of upgrade completion captures these while memory is fresh. Document the output. Share it with the customer for their internal records and with the partner team for future engagements.
The next upgrade conversation
VCF 9.1 won’t be the last upgrade. With the new lifecycle capabilities, future upgrades become much more routine than the one just completed. Plan that posture from the post-upgrade phase:
• Establish a regular patching cadence using Quick Patch and Live Patching. Monthly minimum, more frequent for security patches
• Configure Fleet Update Service orchestration policies that match the customer’s change management appetite
• Document the cluster image governance — who owns which image, who approves changes
• Schedule the next major version upgrade preparation early — customer teams that go years between major version upgrades face the same fresh learning curve each time
• Capability adoption is continuous — plan periodic capability reviews to evaluate what new VCF features merit adoption
The post-upgrade state isn’t a destination; it’s the platform from which the customer continues to evolve. The work in this article is the foundation for that continuous evolution.
Closing
Post-upgrade is where the upgrade engagement either pays off or doesn’t. The technical work is done; the operational adoption is just beginning. Validation, capability enablement, Day-2 transition, lessons-learned capture — these turn a completed upgrade into a working platform that delivers what the upgrade was supposed to deliver.
In my engagements, the post-upgrade questions I work through every time:
• Are the smoke tests all passing? Any failures still under investigation?
• Is the performance baseline captured for comparison with pre-upgrade?
• Is the configuration drift baseline captured in VCF Operations?
• Is the 30–90 day capability enablement roadmap agreed with the customer?
• Is the Day-2 operating model transition planned with documentation refresh, service desk update, and training execution?
• Are change management processes updated to reflect the new lifecycle capabilities?
• Has the lessons-learned retrospective been scheduled within 30 days?
• Is the next upgrade preparation cadence established?
Get those answered and the upgrade engagement delivers what it set out to deliver.
And that closes the Upgrade Series — Parts 19 through 23 — and with it the broader 23-part VCF 9.1 read. From foundational architecture (Parts 1–7), through migration and Day-2 operations (Parts 8–9), through storage-vendor-specific migrations (Parts 10–13), through the Design Topology Field Guide (Parts 14–17), through DR / Backup / Restore (Part 18), to the Upgrade Series (Parts 19–23), this is the practical architect’s read of VCF 9.1 in 2026 — from “what is it” through “how do you operate it” to “how do you get there.”
Sources
• Broadcom — Scale, Simplify, and Secure VCF 9.1
• Broadcom — Announcing VCF 9.1
• Broadcom — VCF 9.1 is Available: Hands-on Labs
• Broadcom — VCF 9.1 Licensing: Programmatic, Centralized, and Built to Scale
• Broadcom — Operations in VCF 9.0: The Modern Way
• Gibson Virtualization — VCF 9.1 What’s New: VCF Operations
