Skip to main content

Thirty-four years - The System Office, Novell Directories, and Building a State Backbone (Part 3)

Unfortunately nearly all the work we put into administrative and academic technology had to be abandoned. As a part of a larger initiative across the state, the various colleges and universities were being merged together into a single system that today is know as Minnesota State. In that process our college president retired, and the new college leadership de-emphasized the use of technology In business practices. Additionally, I recognized that at merger time most of the software that I had written would not usable. So I spent some time getting us off the software I wrote and on to other software that I knew would be used post merger.

In a lot of ways that was a set back for both the college and the students. It was many years before faculty and students would have the functionality we had in 1993.

In the summer of 1995, as the colleges were being merged into what is now Minnesota State, I decided that since the college was not emphasizing technology as much as before and since I was ready to leave the small town environment, I would move to the system office and see what opportunities would be available there.

I became a Novell System Administrator and built out an NDS tree that turned out to be far more similar to modern directories then what It was to the best practices of the time. Conventional wisdom was that you divide up your directory by business unit or business function and create the user, printer file share, and other and directory objects related to the business unit in the OU for the unit. I quickly realized that conventional wisdom and the Novell Directory design guides were wrong. The 'organization hierarchy'  style directory architecture didn't make a whole lot of sense, so I implemented a relatively flat directory where the only OU's were at a very high level and only used to categorize types of objects rather than business units, departments or divisions. The flat directory Is something much closer to what you'd see today and something like Azure Active Directory.

That directory had continuous up-time - I.E. you could log into the directory -  for over a decade. Of course the servers providing services we not always up, but the directory was. :)

Conventional wisdom isn't always right.

While troubleshooting a statewide IPX routing problem I realized that In the post merger chaos, the wide area network was essentially unmanaged. There were routers all across the state on either 56 kbit circuits or T1's running along without any active oversight. So I grab the Cisco manuals, taught myself IP routing, asked a co-worker for the passwords, and started managing the wide area network.

That meant physically locating all the routers around the state, documenting circuit locations and circuit ID's, cleaning up routing tables, cleaning up configurations, and getting the hardware on common operating system versions. We also created a common campus network edge design, wrote our own database driven network monitoring package, combined monitoring and CMDB databases, and built out a system of thresholds, triggers and alerts to help us keep tabs on the hundreds of devices and dozens of sites. Written in Perl and leveraging MRTG, of course.

To help us monitor the network we push purchased Netscout probes for every site. Because we had more than 50 probes, and GUI's don't scale well, I dug into the Netscout management software and figured out that the graphical user interface was just a skin over the top of some very powerful command line programs.  I wrote a package around those programs that automated the maintenance and monitoring of probes and automatically gathered 'Top N' IP, TCP host and port counters for their network traffic data from the probes. I exposed that data to campuses so that campuses had a pretty complete view into their network traffic readily available to them. My idea was that if they knew more about what was happening on their network, they'd be able to do a better job of running their network, and most importantly - they'd be able to resolve more of their own problems themselves. And call me less often.

As the wide-area network evolved and became more critical to our operation, we had to make significant investments in bandwidth and availability. Making those investments on our own without a partner turned out to be very difficult, so we partnered with the State of Minnesota to leverage our resources and their skills to build a common statewide backbone usable by all State agencies and our system - Minnesota State. For nearly twenty years, the State of Minnesota has been the backbone provider for Minnesota State Colleges and Universities. As the State had already partnered with the University of Minnesota, the three largest public entities in the state all  share a common state backbone and Internet connection. In the partnership, Minnesota State benefits from the resources and expertise at the University of Minnesota and State of Minnesota, allowing us to concentrate our skills on other aspects of running a state-wide enterprise.


Popular posts from this blog

Cargo Cult System Administration

Cargo Cult: …imitate the superficial exterior of a process or system without having any understanding of the underlying substance --Wikipedia During and after WWII, some native south pacific islanders erroneously associated the presence of war related technology with the delivery of highly desirable cargo. When the war ended and the cargo stopped showing up, they built crude facsimiles of runways, control towers, and airplanes in the belief that the presence of war technology caused the delivery of desirable cargo. From our point of view, it looks pretty amusing to see people build fake airplanes, runways and control towers  and wait for cargo to fall from the sky.
The question is, how amusing are we?We have cargo cult science[1], cargo cult management[2], cargo cult programming[3], how about cargo cult system management?Here’s some common system administration failures that might be ‘cargo cult’:
Failing to understand the difference between necessary and sufficient. A daily backup …

Ad-Hoc Versus Structured System Management

Structured system management is a concept that covers the fundamentals of building, securing, deploying, monitoring, logging, alerting, and documenting networks, servers and applications. Structured system management implies that you have those fundamentals in place, you execute them consistently, and you know all cases where you are inconsistent. The converse of structured system management is what I call ad hoc system management, where every system has it own plan, undocumented and inconsistent, and you don't know how inconsistent they are, because you've never looked.

In previous posts (here and here) I implied that structured system management was an integral part of improving system availability. Having inherited several platforms that had, at best, ad hoc system management, and having moved the platforms to something resembling structured system management, I've concluded that implementing basic structure around system management will be the best and fastest path to…

The Cloud – Provider Failure Modes

In The Cloud - Outsourcing Moved up the Stack[1] I compared the outsourcing that we do routinely (wide area networks) with the outsourcing of the higher layers of the application stack (processor, memory, storage). Conceptually they are similar:In both cases you’ve entrusted your bits to someone else, you’ve shared physical and logical resources with others, you’ve disassociated physical devices (circuits or servers) from logical devices (virtual circuits, virtual severs), and in exchange for what is hopefully better, faster, cheaper service, you give up visibility, manageability and control to a provider. There are differences though. In the case of networking, your cloud provider is only entrusted with your bits for the time it takes for those bits to cross the providers network, and the loss of a few bits is not catastrophic. For providers of higher layer services, the bits are entrusted to the provider for the life of the bits, and the loss of a few bits is a major problem. These …