Thirty-four years in IT - Building out Disaster Recovery (Part 6)

In the mid-2000's, our organization started to get serious about disaster recovery. By that time our core application was an e-learning application that was heavily used (a hundred thousand students on a typical day). That app became critical to our mission.

To bootstrap a DR capability we paid consultants for what was at best a craptastic DR plan. The plan was not implementable under any realistic scenario.

The consultants ignored our total lack of a DR site, insisted that we could buy servers overnight, and that because every server had its own tape drive, we could hire an army of techs from Geek Squad and recover all servers simultaneously from individual tape backups. Of course we had no failover site, no hardware, and we had tape-changers and a Legato infrastructure that streamed and interleaved multiple backups onto a single tape instead of individual tape drives in each server. I couldn't imagine buying dozens of servers and successfully recovering in any reasonable time frame. The consultants formally presented a 56 hour RTO to our Leadership, when my own gantt charts showed a 3-week RTO after we had a DR site leased, a data center network built, and hardware purchased and racked. So I pushed back hard - and stopped getting invited to the meetings.

They used nice fonts though. Give them credit for that.

Thirty-four years in IT - System Administration, Backups, and Data Centers (Part 5)

As a side effect of building and running the backbone, I introduced UNIX systems into what was then a wholly VMS organization. We initially used Linux - roughly from 1994 - 1997, then over the next 20+ years, briefly migrated to Solaris x86, then to Solaris SPARC and back to Solaris x86/x64, and then back to Linux.

Our CIO at the time recognized that a pure VMS/RDB shop was not a valid long-term strategy and as a result had us host a UNIX/Oracle application on behalf of another organization as a part of building out a new capability that he recognized we'd need someday. As our VMS/RDB team didn't appreciate (or were genuinely hostile toward) non-VMS platforms, they declined to take on the building and management of UNIX/Oracle stack. So I and my team did.

Thirty-four years in IT - Security and firewalling (Part 4)

As a natural fit with running the network my team took on the task of securing the campuses and data centers, starting with firewalling the data centers from the rest of the network. We started fairly simply by just segmenting enterprise-wide servers from networks with users and students and restricting unfettered access to enterprise servers, database and systems. This gave us the ability to control access to the core servers and systems. As expected, this initial segmentation was resisted most by the system managers and DBA's who managed the individual servers and databases. They were convinced that the only way they could possibly do their job was if they had full access to everything all the time from everywhere - even if they had no idea how they were accessing the system. This was a pretty typical attitude at the time, and to me an indicator that they didn't actually know how their systems worked.

Thirty-four years in IT - The System Office, Novell Directories, and Building a State Backbone (Part 3)

Unfortunately nearly all the work we put into administrative and academic technology had to be abandoned. As a part of a larger initiative across the state, the various colleges and universities were being merged together into a single system that today is know as Minnesota State. In that process our college president retired, and the new college leadership de-emphasized the use of technology In business practices. Additionally, I recognized that at merger time most of the software that I had written would not usable. So I spent some time getting us off the software I wrote and on to other software that I knew would be used post merger.

In a lot of ways that was a set back for both the college and the students. It was many years before faculty and students would have the functionality we had in 1993.