Thirty-four years in IT - Building out Disaster Recovery (Part 6)

In the mid-2000's, our organization started to get serious about disaster recovery. By that time our core application was an e-learning application that was heavily used (a hundred thousand students on a typical day). That app became critical to our mission.

To bootstrap a DR capability we paid consultants for what was at best a craptastic DR plan. The plan was not implementable under any realistic scenario.

The consultants ignored our total lack of a DR site, insisted that we could buy servers overnight, and that because every server had its own tape drive, we could hire an army of techs from Geek Squad and recover all servers simultaneously from individual tape backups. Of course we had no failover site, no hardware, and we had tape-changers and a Legato infrastructure that streamed and interleaved multiple backups onto a single tape instead of individual tape drives in each server. I couldn't imagine buying dozens of servers and successfully recovering in any reasonable time frame. The consultants formally presented a 56 hour RTO to our Leadership, when my own gantt charts showed a 3-week RTO after we had a DR site leased, a data center network built, and hardware purchased and racked. So I pushed back hard - and stopped getting invited to the meetings.

They used nice fonts though. Give them credit for that.

Thirty-four years in IT - System Administration, Backups, and Data Centers (Part 5)

As a side effect of building and running the backbone, I introduced UNIX systems into what was then a wholly VMS organization. We initially used Linux - roughly from 1994 - 1997, then over the next 20+ years, briefly migrated to Solaris x86, then to Solaris SPARC and back to Solaris x86/x64, and then back to Linux.

Our CIO at the time recognized that a pure VMS/RDB shop was not a valid long-term strategy and as a result had us host a UNIX/Oracle application on behalf of another organization as a part of building out a new capability that he recognized we'd need someday. As our VMS/RDB team didn't appreciate (or were genuinely hostile toward) non-VMS platforms, they declined to take on the building and management of UNIX/Oracle stack. So I and my team did.

Thirty-four years in IT - Security and firewalling (Part 4)

As a natural fit with running the network my team took on the task of securing the campuses and data centers, starting with firewalling the data centers from the rest of the network. We started fairly simply by just segmenting enterprise-wide servers from networks with users and students and restricting unfettered access to enterprise servers, database and systems. This gave us the ability to control access to the core servers and systems. As expected, this initial segmentation was resisted most by the system managers and DBA's who managed the individual servers and databases. They were convinced that the only way they could possibly do their job was if they had full access to everything all the time from everywhere - even if they had no idea how they were accessing the system. This was a pretty typical attitude at the time, and to me an indicator that they didn't actually know how their systems worked.

Thirty-four years in IT - The System Office, Novell Directories, and Building a State Backbone (Part 3)

Unfortunately nearly all the work we put into administrative and academic technology had to be abandoned. As a part of a larger initiative across the state, the various colleges and universities were being merged together into a single system that today is know as Minnesota State. In that process our college president retired, and the new college leadership de-emphasized the use of technology In business practices. Additionally, I recognized that at merger time most of the software that I had written would not usable. So I spent some time getting us off the software I wrote and on to other software that I knew would be used post merger.

In a lot of ways that was a set back for both the college and the students. It was many years before faculty and students would have the functionality we had in 1993.

Thirty-four Years in IT - Networking and Software Development (Part 2)

At the college we were extremely fortunate to have a president who had a very forward looking view of technology. In the mid 1980s he was already using personal computers regularly and had written some of his own software. Sometime around 1988 or so he described what he thought would be appropriate use of technology in education. He wanted all student records and curriculum to be electronic, all student testing to be electronic, and all grading to be electronic. He envisioned that students could walk up to a computer, login and access the curriculum, access and complete tests and quizzes, look up their progress toward graduation and any fees they may owe, and generate a transcript.

And of course he wanted it all tied together on a network.

Thirty-four Years in IT - Instructor, Machinist, CNC and CAD/CAM (Part 1)

As I've now ended 34+ years of public service, I'm going to burn a few posts on where I've been and what I've tried to accomplish.

Like many people my age, my path toward a career in technology was non-linear. My first stop after a Baccalaureate in Physics was a move into teaching Machine Tool trades at a 2-year college. Make sense, right? Actually I had taken a few programming courses in college (FORTRAN, Pascal, PDP-8 Assembler, SNOBOL, FORTH), had worked my way through college as a machinist, and taught myself how to program CNC machines. So the trade school route wasn't too much of a stretch.