Skip to main content

Thirty-four years - Security and firewalling (Part 4)

As a natural fit with running the network my team took on the task of securing the campuses and data centers, starting with firewalling the data centers from the rest of the network. We started fairly simply by just segmenting enterprise-wide servers from networks with users and students and restricting unfettered access to enterprise servers, database and systems. This gave us the ability to control access to the core servers and systems. As expected, this initial segmentation was resisted most by the system managers and DBA's who managed the individual servers and databases. They were convinced that the only way they could possibly do their job was if they had full access to everything all the time from everywhere - even if they had no idea how they were accessing the system. This was a pretty typical attitude at the time, and to me an indicator that they didn't actually know how their systems worked.

In about 2001 we installed firewalls between each campus and the Internet. This firewall project was one of the first times that I got exposed to vendor and consultant FUD. We had vendors telling us that only certain models of firewall could actually secure our networks. We had consultants tell us that we were not capable of firewalling our own network and that only they had the necessary skills. Some of the consultants that were most closely working with various public sector agencies were also the ones that were most clearly full of FUD.

One of the leading service providers in the area offered that a minimum of $5000 per site and ongoing operation cost of $2500  per site per month would be required for the project. We had 55 sites so we'd have had a project cost of over $1.5M per year. Instead we decided to do the project internally with minimal consulting help and low-cost Cisco PIX firewalls. We hired a contractor to help us configure the first couple of firewalls, then rolled out and managed the firewalls ourselves. I wrote a series of shell scripts that would automatically configure firewalls, switches, routers and terminal servers. We came up with a standard campus network edge design that was detailed enough to specify every connection, port, and even the color of every cable. We went around that state working with campus staff to make sure that they understood how firewalls worked, and how to work with us to manage the firewall rules.

Total cost of the project ended up far less than budgeted and far less than any vendor or consultant. Years later we still had not spent as much as the first-year cost had we gone with vendors and consultants.

When we firewalled the campuses we also had to go against the common wisdom that educational institutions were impossible to firewall - a stance that some educational institutions maintained for many years. We obviously were able to prove conventional wisdom wrong. As described below, not only were we able to firewall campuses in 2001, we were also able to implement strict network segmentation policy in our data centers with outbound default deny as early as the mid-2000's.
Over time, we  hired dedicated security staff, and with their help improved on the data center security model by segmenting servers within the data center based on the relative importance of the data. We built on that model for many years, eventually adding fine grained network segmentation, dedicated jump server networks, dedicated management networks, and dedicated console networks. The intent was to isolate data center networks from desktops as much as possible and to prevent propagation of security incidents through the data center networks and across unrelated applications. As the data center networks contained only known servers for known applications we were able to implement bidirectional 'default deny' network security policy. In other words, servers within the data center could not connect to addresses on the Internet unless it was specifically permitted by firewall policy.

The strict firewall policy mitigated many of the common attack vectors to which other organizations had succumbed. By restricting an applications ability to connect out to random Internet IP addresses, we also lessened our dependency on application security - something which applications were (and still are) notorious for failing.

We developed a strong operating principle that "If it can surf the Internet it can not be secured". In other words, when securing our applications and data we did not trust our own desktops. This principle was and still is validated by even the most trivial following of desktop and application security.

By following this principle we were able to move nearly all critical user data off of desktops and on to data center servers, where we felt reasonably confident in our ability to secure the data. We came up with a methods for allowing remote access to data center servers and applications from what were relatively insecure desktops. We were able to shut off all direct desktop access to all database listeners by installing all applications that required access to a database listener onto remotely accessible servers configured to run the desktop application and manage the data that would normally have been downloaded to desktop. The data never left the data centers, so it was relatively easy to secure vs. had it been downloaded to desktops.

This was fairly complex and expensive to run, as it required thorough understanding of exactly how every application and technology worked - in many cases something that not even vendors who wrote the application understood. We oftentimes ran into vendors who told us that their application or technology could not be firewalled, or that if we were to attempt to firewall it they would not support us, or they told us how to firewall the app or technology but they were wrong -  they simply didn't know how their app or technology worked.

This also required  a significant effort to work with users to convince them that the inconveniences of having to remotely access their data in the data centers reduced security risk enough that their obligation and  responsibility towards the owners of the data would be well met. In most cases the user aspect of the problem was harder to solve than the technical aspect.


  1. That was an interesting series! I sheepishly admit that I did not notice the four posts until today. But, with a lull in work activity this morning (aka, me sitting at the repair shop getting my vehicle serviced), I trawled through my Feedly backlog of several hundred posts out of dozens of feeds. This one was near the bottom of the feeds list, in the "uncategorized" block of feeds, which means it was a port over from Google Reader, and that I usually missed it when I did quick looks through my common feeds.

    I started reading this blog in 2009-ish when I was in college. I liked this blog for two reasons: it involved my area of interest: technology (note: I became a software developer) and it was close to home (another Minn-ah-so-tan here).

    As a software consultant myself, I can understand the problems you wrote regarding costs. Often times outside contracting & consulting is done to save on costs, but many times it would be cheaper to just use internal employees. It is even more interesting that the same problem you had regarding costs back in 2001 is equally applicable nearly 20 years later.

    Thank you for this series. I enjoyed reading it. Having just now noticed your 2015 post about resurrecting this blog or not, I just want to say thank you. I hope you can continue to post about your experiences, a memoir of sorts, but regardless, thank you for your thoughts over the years.

  2. Thanks Paul -

    Good to know that someone is still reading my stuff. :)

    I have a bunch more in my head - and hopefully will get them down soon.



Post a Comment

Popular posts from this blog

Cargo Cult System Administration

Cargo Cult: …imitate the superficial exterior of a process or system without having any understanding of the underlying substance --Wikipedia During and after WWII, some native south pacific islanders erroneously associated the presence of war related technology with the delivery of highly desirable cargo. When the war ended and the cargo stopped showing up, they built crude facsimiles of runways, control towers, and airplanes in the belief that the presence of war technology caused the delivery of desirable cargo. From our point of view, it looks pretty amusing to see people build fake airplanes, runways and control towers  and wait for cargo to fall from the sky.
The question is, how amusing are we?We have cargo cult science[1], cargo cult management[2], cargo cult programming[3], how about cargo cult system management?Here’s some common system administration failures that might be ‘cargo cult’:
Failing to understand the difference between necessary and sufficient. A daily backup …

Ad-Hoc Versus Structured System Management

Structured system management is a concept that covers the fundamentals of building, securing, deploying, monitoring, logging, alerting, and documenting networks, servers and applications. Structured system management implies that you have those fundamentals in place, you execute them consistently, and you know all cases where you are inconsistent. The converse of structured system management is what I call ad hoc system management, where every system has it own plan, undocumented and inconsistent, and you don't know how inconsistent they are, because you've never looked.

In previous posts (here and here) I implied that structured system management was an integral part of improving system availability. Having inherited several platforms that had, at best, ad hoc system management, and having moved the platforms to something resembling structured system management, I've concluded that implementing basic structure around system management will be the best and fastest path to…

The Cloud – Provider Failure Modes

In The Cloud - Outsourcing Moved up the Stack[1] I compared the outsourcing that we do routinely (wide area networks) with the outsourcing of the higher layers of the application stack (processor, memory, storage). Conceptually they are similar:In both cases you’ve entrusted your bits to someone else, you’ve shared physical and logical resources with others, you’ve disassociated physical devices (circuits or servers) from logical devices (virtual circuits, virtual severs), and in exchange for what is hopefully better, faster, cheaper service, you give up visibility, manageability and control to a provider. There are differences though. In the case of networking, your cloud provider is only entrusted with your bits for the time it takes for those bits to cross the providers network, and the loss of a few bits is not catastrophic. For providers of higher layer services, the bits are entrusted to the provider for the life of the bits, and the loss of a few bits is a major problem. These …