Skip to main content

Service Deprovisioning as a Security Practice

Deprovisioning of things other than users is, from the point of view of a security person or system manager, both interesting and relatively unexplored. Applications, servers, databases and devices have a finite life, even if the bad ones seem to hang around forever. The end-of-life of these non-person things is an event that needs the attention if future security and operational issues are to be avoided.

When one thinks of deprovisioning, one generally thinks of user deprovisioning. The search engines are full of interesting bits in the user deprovisioning space, most of which are more interesting than my thoughts. I'm interested in deprovisioning of things other than users.

To organize the concept of non-user deprovisioning, I'll invent something called 'Service Deprovisioning'. A temporary definition of Service Deprovisioning:

When an application, server, database or device is no longer required for system functionality, it and its dependencies are taken out of service using a structured methodology.
To ensure that happens consistently:

A cross boundary process must be established that assures that the service and its related dependencies, including application software, databases, firewall and load balancer configs, network, storage and server infrastructure are properly deprovisioned.
The concept assumes that system managers actually know when a service is no longer in use. In some cases, where the service is upgraded or re-deployed, the point at which the old service is no longer 'in production' is easy to determine. In other cases the service may gradually fall into disuse and not have a definitive date at which it may be deprovisioned. The concept also assumes that the various entities within an organization communicate to each other the services that are being deprovisioned, so that for example firewall administrators know that an application is no longer in use and DBAs know that a database can be dropped.

The high level tasks can be broken down into at least:

  • Firewall and Load Balancer rule deprovisioning
  • Software Deprovisioning
  • Database Deprovisioning
  • Physical Server Decommissioning
  • Virtual Sever Deprovisioning
  • Network Deprovisioning
  • Storage Deprovisioning
(If I think of more, I'll update this post.)

Firewall and Load Balancer rule deprovisioning is probably the most critical task associated with service deprovisioning. The outward facing interface to the service normally should be the first item on the deprovisioning check list. Unused or dead rules have a high probability of causing future security incidents as new services are deployed in the same network address space.

In the best case, all the rules associated with an application, server or database are well documented and well understood. Deprovisioning the rules then follows the service related documentation. A more likely scenario is that the rules are not fully understood and a discovery process must be used to determine which rules are associated with a service. Some discovery is straight forward. If a host is decommissioned, all rules associated with that host IP address should be candidates for removal. For organizations that use URL based load balancing rules, the load balancing rules must be unconfigured as elements of the web application are disabled or taken out of service.

Software Deprovisioning must occur whenever application software is no long required for production. Both the application software and its dependencies must be uninstalled. Uninstalling applications insures that the software cannot affect the availability or security of a system at some point in time in the future. Software deprovisioning is particularly important in the case of Internet facing applications. It isn't hard to imagine using as a search tool to find older, less secure versions of an application. If the old version is still hanging around on app servers, it would have a fair chance of being an unexpected conduit into the enterprise. (Of course that assumes that the old version of the application is more vulnerable to SQL injection, Cross Site Scripting, etc. The way things look today, that might not be a safe assumption.)

In some cases, such as with Perl CPAN or in the case Java run time proliferation, it is difficult to determine the application dependencies, hence it is difficult to decide if a particular JRE or CPAN module is no longer in service and can be deleted. Even if the original installation is properly documented, applications installed on the same server after the deprovisioned application may (or likely will) have undocumented dependencies on modules that were part of the original application. It tends to be difficult to sort application interdependencies unless some form of structured software installation is used and application interdependencies are clearly determined at installation time.

In any cases where the uninstalled software had a network interface (listening on a TCP or UDP port) there presumably is an associated firewall rule. In theory, every time a Windows service or Unix daemon is taken out of service, there should be corresponding firewall and/or load balancer change. Over the years I have had the opportunity to investigate many successful web server hacks. In one case, the hacker was kind enough to leave a message on the server. 'You only left one port open. One port was all I needed.'

Database Deprovisioning follows the deprovisioning of the related applications. The process is straightforward. Full (cold) backups are taken, the database is dropped, the listener is stopped, related data files, control files, scripts and utilities are removed or modified. Both the firewall rules and storage related to the database are dropped. There should be a one to one relationship between dropping a database and removing a firewall rule set.

Physical Server Decommissioning occurs when the last service that depends on the sever has been deprovisioned. Decommissioning of servers requires not only the obvious un-racking, cable removal, asset disposal and inventory updates, but also requires DNS updates, network deprovisioning, changes to monitoring systems and related documentation. Server decommissioning should automatically trigger the related firewall, network, DNS and monitoring changes required to deprovision the server related dependencies. If the server was SAN attached, the related storage must aslo be deprovisioned.

Virtual Sever Deprovisioning follows roughly the same rules as physical servers, minus the un-racking and un-cabling, but with the additional task of ensuring the archiving or disposal of old virtual machine clones, and the related task of unconfiguring any virtualized network interfaces or virtual security devices.

Network Deprovisioning is required any time that network connected hardware is decommissioned. Presumably your network managers are routinely disabling unused network ports, and hence need notification of network connections that are no longer in use. If servers are un-racked, then network cabling must be pulled out and network documentation must be updated. Network deprovisioning is an essential part of ensuring that future re-use of the network interface doesn't cause unanticipated security or configuration issues.

Storage Deprovisioning tends to be straight forward, and almost certainly the task that happens most reliably. Storage mangers, who are almost always are struggling to find space on SAN's, tend be pretty good at finding unused LUN's. Decommissioning SAN attached servers requires related switch fabric zone changes, switch port deprovisioning and documentation updates. Tagged on to storage deprovisioning is the final disposition of the last set of full backup and archive tapes, and if spindles are freed up, the wiping and/or disposal of the spindles is required.

Service Deprovisioning is the tail end of the Least Bit Principle, and assumes that you have some form of Structured System Management.


Popular posts from this blog

Cargo Cult System Administration

Cargo Cult: …imitate the superficial exterior of a process or system without having any understanding of the underlying substance --Wikipedia During and after WWII, some native south pacific islanders erroneously associated the presence of war related technology with the delivery of highly desirable cargo. When the war ended and the cargo stopped showing up, they built crude facsimiles of runways, control towers, and airplanes in the belief that the presence of war technology caused the delivery of desirable cargo. From our point of view, it looks pretty amusing to see people build fake airplanes, runways and control towers  and wait for cargo to fall from the sky.
The question is, how amusing are we?We have cargo cult science[1], cargo cult management[2], cargo cult programming[3], how about cargo cult system management?Here’s some common system administration failures that might be ‘cargo cult’:
Failing to understand the difference between necessary and sufficient. A daily backup …

Ad-Hoc Versus Structured System Management

Structured system management is a concept that covers the fundamentals of building, securing, deploying, monitoring, logging, alerting, and documenting networks, servers and applications. Structured system management implies that you have those fundamentals in place, you execute them consistently, and you know all cases where you are inconsistent. The converse of structured system management is what I call ad hoc system management, where every system has it own plan, undocumented and inconsistent, and you don't know how inconsistent they are, because you've never looked.

In previous posts (here and here) I implied that structured system management was an integral part of improving system availability. Having inherited several platforms that had, at best, ad hoc system management, and having moved the platforms to something resembling structured system management, I've concluded that implementing basic structure around system management will be the best and fastest path to…

The Cloud – Provider Failure Modes

In The Cloud - Outsourcing Moved up the Stack[1] I compared the outsourcing that we do routinely (wide area networks) with the outsourcing of the higher layers of the application stack (processor, memory, storage). Conceptually they are similar:In both cases you’ve entrusted your bits to someone else, you’ve shared physical and logical resources with others, you’ve disassociated physical devices (circuits or servers) from logical devices (virtual circuits, virtual severs), and in exchange for what is hopefully better, faster, cheaper service, you give up visibility, manageability and control to a provider. There are differences though. In the case of networking, your cloud provider is only entrusted with your bits for the time it takes for those bits to cross the providers network, and the loss of a few bits is not catastrophic. For providers of higher layer services, the bits are entrusted to the provider for the life of the bits, and the loss of a few bits is a major problem. These …