Skip to main content

Virtualization Security - Reading List (update)

(Added and updated links) Here's a reading list of interesting posts in the virtualization+security space. There is a lot of thought going on about how virtualization in the data center affects application security. If you have VM's, you ought to be reading and thinking. Really thinking.

Start with Five Immutable Laws of Virtualization Security. Follow through to the Burton Blog for details. (Pete Lindstrom, Spire Security)

Maybe read “Virtualization and Security: The Full Story” (I'm still thinking about this one.) I'm done thinking. See comments. (Sara Peters, CSI)

Thoughts on auditing virtualized environments, separation of duties, and audit controls. Are your auditors going to declare all the vm's that share the same hardware cluster in scope for the audit of one of the vm's? Ours will. (Hoff, Rational Security)

Bits on PCI compliance (Hoff, Rational Security)

The operational issues surrounding virtualized servers, or - how are you going to manage the siloed operational domains of system management, network and security, when all of the above are in one box? (Hoff, Rational Security)

The performance issues that you should think about before dumping your virtualized network and security functions onto the same processors that serve up your application. Can we say context switches? I keep thinking about a spanning tree meltdown in a virtual switch. That would be amusing. Hint: There is no wire to pull. (Hoff, Rational Security)

An analysis of Patch frequency for VMWare ESX. Yep - you now have another platform to patch-manage, another patch repository, another patch management console, another set of patches to run through the patch test-QA-deploy cycle. (Ronald Oglesby and Dan Pianfetti @ GlassHouse Technologies)

A summary of the four big issues surrounding virtualization. (Hoff, Rational Security)

And last, but not least, Theo's thoughts on virtualization.

"x86 virtualization is about basically placing another nearly full
kernel, full of new bugs, on top of a nasty x86 architecture which
barely has correct page protection.  Then running your operating
system on the other side of this brand new pile of shit."




  1. Hey Mike... So I gotta ask. What about my story are you "still thinking about"?

    Inquiring me wants to know.


  2. Sara -- I'm still thinking about it because you wrote an article that requires a bit of thought. ;)

    General Comments:

    The introductory sections are great, and you've got an excellent collection of viewpoints on the topic. It is a very effective summary of the current state of virtualization and security. That makes it worth the read. I'm glad I ran across it and I'm glad you've posted it publically.

    Now - some specific comments:

    "Properly locking down a data center containing 1,000 servers is far pricier than locking down a data center with 10 servers."

    I disagree. To lock down 1000 server operating systems installed on real hardware is exactly the same cost as locking down 1000 virtualized servers. They all still need hardening, upgrades, anti-virus agents and routine patching. They all still need vulnerability scans & periodic auditing. Those costs are a wash. The associated network and firewall lock down cost is also the same, but the virtualized environment has an additional cost associated with locking down the hypervisors and the VM host operating system (Red Hat EL in the case of VMware).

    My take so far on virtualization is that you really only save on server hardware costs, power/cooling costs, and network costs. You have a significant increase in storage costs. At $800 per fiber channel HBA, and $2000 per fiber channel switch port, the SAN attach costs are approaching the cost of the server itself, and if you are going to do anything other than trivial virtualization, you'll have to get SAN attached (redundantly, of course). All the other costs related to system management, operations patching, etc. are either the same or higher.

    "It’d also be swell if I could quickly make a few copies of my corporate server"

    Making a copy of a corporate server and taking it with you on your trip would be a bad idea in my book. And we tend to leave copies of VM's laying around the datacenter, as leftovers from upgrades or config changes. Those copies are potential private data loss incidents sitting around waiting to happen.

    "When one server can so suddenly become three or four, the risk of server sprawl is very real, and can defeat the original purpose of server consolidation"

    I couldn't agree more. "Let's just spin up another VM.....".

    "'So perhaps the HR department uses one VM for Web browsing and another VM—one which is never connected to the Internet, nor even has a Web browser—that is devoted exclusively to the task of running the PII database software."

    I agree with the idea of separation of unprivileged apps (browsing) from privileged apps (HR). We've been doing similar sorts of desktop separation for quite a while. But for all practical purposes, a better method of separating the functions would be to put the secure app on a server running RDP or Citrix that is configured with minimal privileges, can't access the Internet, and is safely tucked away in a rack in the datacenter. It would be far easier to deploy a thin client/remote desktop solution than multiple VM's on a whole bunch of desktops, and you'd have the same privilege separation.

    "The wisest course of action may be to wait until 1) the virtualization security tools are more mature and 2) your organization is prepared to adequately invest in the superior security the virtual environment demands. Only then should you strap on your SCUBA gear and take the plunge."

    Yep - that's about where I am at on the topic. Virtualization where appropriate, without crossing security boundaries.

    Thanks for the great article.


  3. All good points, Mike.

    Just to be clear, when I said "It'd be swell if I could make a couple of copies of the corporate server" I was sarcastically speaking as though I were an average gal thinking only of convenience, and not the squirrely security geek that I am. The thought of someone actually doing such a thing makes me cringe.

    You say: "To lock down 1000 server operating systems installed on real hardware is exactly the same cost as locking down 1000 virtualized servers."

    This is an interesting point, and--not surprisingly--quite a different stance than that taken by many virtualization vendors.

    It sounds, though, that you're actually spending more on securing your virtualized servers, (since the virtual environment needs better security) and saving money only on the non-security operational costs. Is that fair to say?

  4. What is interesting about the 'copy of the corporate server' is not that you'd take one on the road with you, but rather that with VM's it is really easy to keep copies of servers laying around, sensitive data or not. It is so easy that it happens all the time. In some cases, such as DR or business continuity, we want copies of VM's stored off line or of site to achieve desired RPO/RTO goals. In other cases, the copies are left laying around for the convenience of the server manager. Those copies are a potential security problem, just like loose backup tapes. Releasing a copy of sensitive data generates the same headlines as releasing the original, but because the copies are scattered about uncatalogued, the probability of accidental release is higher.

    Your sarcasm make me think - ding! - I have to secure every copy of every VM just the same as the original host, and to do that I need security and auditing and change control on the spot where I store the 'spare' VM's that is at least equal to the security requirements of the most sensitive host copy that I store. I think my costs just went up. ;)

    So if I want to hack a datacenter and grab interesting data from a server, I have the virtualized server containing the data as a target, the host server that is hosting the live virtual server as a target, the tape backups (or backup-to-disk-pool) as a target, and the big cheap NAS device where the sys admin stores copies of the VM's off to the side 'just in case'. I'd aim for the latter first.

    On ROI, because of the uncertainty surrounding hypervisor and VM host security, I'm not yet willing to cross security compartments on shared hardware, so the ROI has to be based on the consolidation of servers within a netblock (or VLAN, or firewall interface, or what I call a security compartment). In the model I use to design a hosting infrastructure, I have lots of small vlans with lots of firewall interfaces, so the server count per compartment is low, and hence I have a hard time consolidating. I can't do 10:1 so I don't save enough on hardware to make an easy ROI on the increased person-cost of having to manage another platform (ESX). Others, who either are willing to cross security boundaries or who can run higher ratios will see different numbers. Of course in some cases, such as test/dev/prototyping/proof-of-concept, Vm's are clearly an easy ROI win.

    And - I tend to have a different take than vendors on a lot of technology. Their job is to sell, my job is to be skeptical. ;)


Post a Comment

Popular posts from this blog

Cargo Cult System Administration

Cargo Cult: …imitate the superficial exterior of a process or system without having any understanding of the underlying substance --Wikipedia During and after WWII, some native south pacific islanders erroneously associated the presence of war related technology with the delivery of highly desirable cargo. When the war ended and the cargo stopped showing up, they built crude facsimiles of runways, control towers, and airplanes in the belief that the presence of war technology caused the delivery of desirable cargo. From our point of view, it looks pretty amusing to see people build fake airplanes, runways and control towers  and wait for cargo to fall from the sky.
The question is, how amusing are we?We have cargo cult science[1], cargo cult management[2], cargo cult programming[3], how about cargo cult system management?Here’s some common system administration failures that might be ‘cargo cult’:
Failing to understand the difference between necessary and sufficient. A daily backup …

Ad-Hoc Versus Structured System Management

Structured system management is a concept that covers the fundamentals of building, securing, deploying, monitoring, logging, alerting, and documenting networks, servers and applications. Structured system management implies that you have those fundamentals in place, you execute them consistently, and you know all cases where you are inconsistent. The converse of structured system management is what I call ad hoc system management, where every system has it own plan, undocumented and inconsistent, and you don't know how inconsistent they are, because you've never looked.

In previous posts (here and here) I implied that structured system management was an integral part of improving system availability. Having inherited several platforms that had, at best, ad hoc system management, and having moved the platforms to something resembling structured system management, I've concluded that implementing basic structure around system management will be the best and fastest path to…

The Cloud – Provider Failure Modes

In The Cloud - Outsourcing Moved up the Stack[1] I compared the outsourcing that we do routinely (wide area networks) with the outsourcing of the higher layers of the application stack (processor, memory, storage). Conceptually they are similar:In both cases you’ve entrusted your bits to someone else, you’ve shared physical and logical resources with others, you’ve disassociated physical devices (circuits or servers) from logical devices (virtual circuits, virtual severs), and in exchange for what is hopefully better, faster, cheaper service, you give up visibility, manageability and control to a provider. There are differences though. In the case of networking, your cloud provider is only entrusted with your bits for the time it takes for those bits to cross the providers network, and the loss of a few bits is not catastrophic. For providers of higher layer services, the bits are entrusted to the provider for the life of the bits, and the loss of a few bits is a major problem. These …