Thirty-four years in IT - Building out Disaster Recovery (Part 6)

In the mid-2000's, our organization started to get serious about disaster recovery. By that time our core application was an e-learning application that was heavily used (a hundred thousand students on a typical day). That app became critical to our mission.

To bootstrap a DR capability we paid consultants for what was at best a craptastic DR plan. The plan was not implementable under any realistic scenario.

The consultants ignored our total lack of a DR site, insisted that we could buy servers overnight, and that because every server had its own tape drive, we could hire an army of techs from Geek Squad and recover all servers simultaneously from individual tape backups. Of course we had no failover site, no hardware, and we had tape-changers and a Legato infrastructure that streamed and interleaved multiple backups onto a single tape instead of individual tape drives in each server. I couldn't imagine buying dozens of servers and successfully recovering in any reasonable time frame. The consultants formally presented a 56 hour RTO to our Leadership, when my own gantt charts showed a 3-week RTO after we had a DR site leased, a data center network built, and hardware purchased and racked. So I pushed back hard - and stopped getting invited to the meetings.

They used nice fonts though. Give them credit for that.

After seeing where consultants were taking us, I pushed our organization toward full hardware and application redundancy and full data center failover capability for all data center hosted systems. My goal was to have two fully functional data centers, identically configured, identical hardware, full redundancy at the failover site, and near real-time data replication between them, all matched to realistic and achievable RPO and RTO. My rational was that an organization as small and under-resourced as ours would not be able to built, maintain, and routinely test a disaster recovery site that was not already built, running, and replicated;  and that the failover hardware would be usable as pre-production, staging or some other purpose.

The way we accomplished this was to tackle the longest lead-time constraints first, starting with space. We learned that our partners at the State of Minnesota has several thousand square feet of data center space sitting empty - as they had just consolidated down to smaller mainframes. I offered to lease that space, and then worked with their electricians to preposition the correct power under the floor, having them build out PDUs and pigtails for the servers and storage that we'd parachute in if we had a disaster. That took care of the longest lead time items - space and power. We then built out a data center network - stubbed out at first, but eventually fully configured and routed to the backbone.

We then invested heavily in failover hardware.

The 'full failover' strategy meant that if a back end database required 'N' CPU's in production, we had to purchase and maintain '2N' in each of primary and secondary data centers, and in most cases a fraction of N in one or more QA and development instances. The QA and development instances were configured behind a fully redundant network stack that was used by the network team to QA network, firewall and load balancer technologies.

As we cycled through normal hardware replacement and rotation, we first filled out the failover data center with one-generation old hardware, figuring that half a loaf was better than none. Later we started buying and configuring identical hardware in both data centers - ideally upgrading failover first, so we never were in a spot where failover was behind production.

Hardware vendors loved us.

I felt strongly that if we didn't use the failover environment regularly, it would fall behind production and become unusable for failover - primarily because of configuration rot. This meant that wherever possible we needed to automate the configuration of devices and systems. It simply is not possible to ensure that two systems are identical in any case where they are manually configured. I.E. - you must have Structured System Management - scripts, not clicks. For UNIX systems this was fairly straight forward. For Windows, the options were few and painful. On the Windows systems we had far more clicks than scripts.

We achieved usable DR capability for our primary e-learning application in 2006, full capability for that application in about 2009, and for our ERP some years later. The team that ran the e-learning environment conducted a run-from-failover exercise annually, so we were assured that we could meet our published RPO and RTO for that application and its supporting technology.

Selling Disaster Recovery is hard. Most teams did not buy into the 'Failover is a first class citizen' mantra that I'd been preaching. For example, even though we had identical failover hardware for the ERP, the ERP team did not maintain failover in a fully configured state - often not even acknowledging the existence of the failover servers - and hence was not capable of conducting a failover within a reasonable RTO.

We did however - after a 6-month reconfiguration and testing effort - fail the ERP over to a new data center, so we knew it was possible. That effort required a reverse-engineering of an app (that we had written ourselves) sufficiently so we understood exactly how it was configured. We were then able to re-configure both production and failover identically, and successfully fail over the application. The team that ran the app didn't think it could be done.  My team proved them wrong.