Most IT teams don’t lose to ransomware. They lose, slowly, to entropy. A node goes EOL. A firmware advisory gets deferred for “the next change window.” A vSphere cluster is two minor versions behind because the last upgrade was bad. None of those decisions look reckless on the day they’re made — and that’s exactly the problem.
We see this most often in regulated industries — healthcare, insurance, transportation, aerospace — where the infrastructure has been described, truthfully, as “mission-critical” for so long that the underlying platform has quietly become fragile.
The illusion of high availability
Virtualized HA was sold, fairly, on a powerful idea: any single component can fail, and your workloads ride through. That’s still true. The thing nobody puts on the marketing slide is that this guarantee depends on every other component being healthy and current. HA isn’t a property of the cluster — it’s a property of the maintenance regime around the cluster.
Some of the things we routinely find on first walk-through:
- Hosts running ESXi releases that went general-support EOL 18 months ago.
- iDRAC / BMC firmware from before three CVE advisories.
- Backup software that hasn’t had a successful application-consistent restore tested in over a year.
- A vCenter on the same cluster it manages — a topology that turns a single bad host into a recovery puzzle.
- Distributed switches with hand-edited port groups that aren’t in the IaC repository, because the IaC repository was abandoned.
Each is survivable. The combination is the thing that ends a Tuesday at 9 PM and turns into a 60-hour week.
Layered dependencies, layered debt
Virtualization stacks have at least six layers that need coordinated attention: hypervisor, hardware firmware, storage, networking, the management plane, and the protected workloads themselves. Most teams patch one or two of those well, two of them on a slow cadence, and the rest reactively when something breaks. Compounding interest does the rest.
When compliance compounds the cost
In regulated industries, the cost of an HA failure is no longer measured only in operational disruption. A failed upgrade in a HIPAA-aligned environment can become an audit finding. A delayed firmware patch in an insurance back-office can collide with a state DOI examination. A weekend-long outage in an aerospace MRO setting can trigger a customer clause that requires written notification to their regulator.
Translation: the work to keep a regulated cluster healthy isn’t operations work. It’s a control. And like any control, it should be documented, scheduled, evidenced, and reviewed.
Lifecycle discipline beats heroics
The teams we admire most aren’t the ones who can do a flawless all-nighter. They’re the ones who never need to. The discipline that replaces heroics is unspectacular and effective:
- A written lifecycle calendar. Every component has a known support window, a planned upgrade quarter, and a named owner — for every cluster, every fabric, every backup target.
- Pre-validated firmware/driver bundles. Vendor-blessed combinations get tested in a small lab cluster before any production rollout. The lab cluster is a single mid-tier host, not a fantasy.
- Configuration as evidence. The state of every cluster is captured in declarative form (Terraform, Ansible, vSphere tags-as-policy). When the auditor asks, you point to a commit, not a screenshot.
- Real RPO/RTO drills. Quarterly. Timed. Recorded. With the people who’d actually do it under pressure, not just the architect.
- An exit strategy for every appliance. If the appliance vendor drops support tomorrow, you have a plan that doesn’t require a forklift. Especially relevant after the recent Broadcom/VMware licensing changes.
A short, anonymized case
We were brought in by a regional health-insurance carrier whose vSphere environment hadn’t had a major upgrade in three years. Claims processing rode on it. The internal team — two excellent engineers — had been holding the line, but were 18 months behind on hypervisor patches and four firmware advisories deep on the storage fabric. The carrier had an audit on the calendar in 100 days.
The remediation wasn’t glamorous. We sequenced four weekend changes, each with a documented rollback. We added an off-cluster vCenter, a dedicated jump host, and an immutable backup target the production domain admin couldn’t reach. We wrote the lifecycle calendar in Confluence and put a name next to every line.
At the audit, the conversation about IT controls took 22 minutes. Two years earlier, it had taken seven hours.
A quiet 90-day plan, if you suspect drift
- Days 1–14 — full inventory. Every host, every firmware, every license, every owner. The deliverable is a single spreadsheet — boring, complete.
- Days 15–30 — RTO/RPO drill on one tier-1 workload. Time it. Document the gaps.
- Days 31–60 — first sequenced upgrade weekend, with rollback plan, written change record, and a post-mortem the next Monday.
- Days 61–90 — codify the lifecycle calendar, pin owners, and brief leadership in a one-page summary.
Nothing here requires a six-month consulting engagement. It requires the decision to treat “keeping the platform healthy” as a discipline, not a chore. If you’d like a fresh pair of eyes on yours, we run the inventory and the first drill at no cost — and you keep the spreadsheet whether you hire us or not.