Thirty-four years in IT - Security and firewalling (Part 4)

As a natural fit with running the network my team took on the task of securing the campuses and data centers, starting with firewalling the data centers from the rest of the network. We started fairly simply by just segmenting enterprise-wide servers from networks with users and students and restricting unfettered access to enterprise servers, database and systems. This gave us the ability to control access to the core servers and systems. As expected, this initial segmentation was resisted most by the system managers and DBA's who managed the individual servers and databases. They were convinced that the only way they could possibly do their job was if they had full access to everything all the time from everywhere - even if they had no idea how they were accessing the system. This was a pretty typical attitude at the time, and to me an indicator that they didn't actually know how their systems worked.

In about 2001 we installed firewalls between each campus and the Internet. This firewall project was one of the first times that I got exposed to vendor and consultant FUD. We had vendors telling us that only certain models of firewall could actually secure our networks. We had consultants tell us that we were not capable of firewalling our own network and that only they had the necessary skills. Some of the consultants that were most closely working with various public sector agencies were also the ones that were most clearly full of FUD.

One of the leading service providers in the area offered that a minimum of $5000 per site and ongoing operation cost of $2500  per site per month would be required for the project. We had 55 sites so we'd have had a project cost of over $1.5M per year. Instead we decided to do the project internally with minimal consulting help and low-cost Cisco PIX firewalls. We hired a contractor to help us configure the first couple of firewalls, then rolled out and managed the firewalls ourselves. I wrote a series of shell scripts that would automatically configure firewalls, switches, routers and terminal servers. We came up with a standard campus network edge design that was detailed enough to specify every connection, port, and even the color of every cable. We went around that state working with campus staff to make sure that they understood how firewalls worked, and how to work with us to manage the firewall rules.

Total cost of the project ended up far less than budgeted and far less than any vendor or consultant. Years later we still had not spent as much as the first-year cost had we gone with vendors and consultants.

When we firewalled the campuses we also had to go against the common wisdom that educational institutions were impossible to firewall - a stance that some educational institutions maintained for many years. We obviously were able to prove conventional wisdom wrong. As described below, not only were we able to firewall campuses in 2001, we were also able to implement strict network segmentation policy in our data centers with outbound default deny as early as the mid-2000's.
Over time, we  hired dedicated security staff, and with their help improved on the data center security model by segmenting servers within the data center based on the relative importance of the data. We built on that model for many years, eventually adding fine grained network segmentation, dedicated jump server networks, dedicated management networks, and dedicated console networks. The intent was to isolate data center networks from desktops as much as possible and to prevent propagation of security incidents through the data center networks and across unrelated applications. As the data center networks contained only known servers for known applications we were able to implement bidirectional 'default deny' network security policy. In other words, servers within the data center could not connect to addresses on the Internet unless it was specifically permitted by firewall policy.

The strict firewall policy mitigated many of the common attack vectors to which other organizations had succumbed. By restricting an applications ability to connect out to random Internet IP addresses, we also lessened our dependency on application security - something which applications were (and still are) notorious for failing.

We developed a strong operating principle that "If it can surf the Internet it can not be secured". In other words, when securing our applications and data we did not trust our own desktops. This principle was and still is validated by even the most trivial following of desktop and application security.

By following this principle we were able to move nearly all critical user data off of desktops and on to data center servers, where we felt reasonably confident in our ability to secure the data. We came up with a methods for allowing remote access to data center servers and applications from what were relatively insecure desktops. We were able to shut off all direct desktop access to all database listeners by installing all applications that required access to a database listener onto remotely accessible servers configured to run the desktop application and manage the data that would normally have been downloaded to desktop. The data never left the data centers, so it was relatively easy to secure vs. had it been downloaded to desktops.

This was fairly complex and expensive to run, as it required thorough understanding of exactly how every application and technology worked - in many cases something that not even vendors who wrote the application understood. We oftentimes ran into vendors who told us that their application or technology could not be firewalled, or that if we were to attempt to firewall it they would not support us, or they told us how to firewall the app or technology but they were wrong -  they simply didn't know how their app or technology worked.

This also required  a significant effort to work with users to convince them that the inconveniences of having to remotely access their data in the data centers reduced security risk enough that their obligation and  responsibility towards the owners of the data would be well met. In most cases the user aspect of the problem was harder to solve than the technical aspect.