Thursday, April 24, 2008

Virtualization Security - Reading List (update)

(Added and updated links) Here's a reading list of interesting posts in the virtualization+security space. There is a lot of thought going on about how virtualization in the data center affects application security. If you have VM's, you ought to be reading and thinking. Really thinking.

Start with Five Immutable Laws of Virtualization Security. Follow through to the Burton Blog for details. (Pete Lindstrom, Spire Security)

Maybe read Virtualization and Security: The Full Story (I'm still thinking about this one.) I'm done thinking. See comments. (Sara Peters, CSI)

Thoughts on auditing virtualized environments, separation of duties, and audit controls. Are your auditors going to declare all the vm's that share the same hardware cluster in scope for the audit of one of the vm's? Ours will. (Hoff, Rational Security)

Bits on PCI compliance (Hoff, Rational Security)

The operational issues surrounding virtualized servers, or - how are you going to manage the siloed operational domains of system management, network and security, when all of the above are in one box? (Hoff, Rational Security)

The performance issues that you should think about before dumping your virtualized network and security functions onto the same processors that serve up your application. Can we say context switches? I keep thinking about a spanning tree meltdown in a virtual switch. That would be amusing. Hint: There is no wire to pull. (Hoff, Rational Security)

An analysis of Patch frequency for VMWare ESX. Yep - you now have another platform to patch-manage, another patch repository, another patch management console, another set of patches to run through the patch test-QA-deploy cycle. (Ronald Oglesby and Dan Pianfetti @ GlassHouse Technologies)

A few bits on securing ESX. Really basic and obvious, but likely not followed by most vm system managers. Certainly not a substitute for separating vm's by security classification. (Amol Sarwate, SC Magazine)

A few basic rules you to match up your vm infrastructure to your security containers. Really important rules. (Rich Mogull, Securosis)

A summary of the four big issues surrounding virtualization. (Hoff, Rational Security)

And last, but not least, Theo's thoughts on virtualization.

"x86 virtualization is about basically placing another nearly full
kernel, full of new bugs, on top of a nasty x86 architecture which
barely has correct page protection. Then running your operating
system on the other side of this brand new pile of shit."




  1. Hey Mike... So I gotta ask. What about my story are you "still thinking about"?

    Inquiring me wants to know.


  2. Sara -- I'm still thinking about it because you wrote an article that requires a bit of thought. ;)

    General Comments:

    The introductory sections are great, and you've got an excellent collection of viewpoints on the topic. It is a very effective summary of the current state of virtualization and security. That makes it worth the read. I'm glad I ran across it and I'm glad you've posted it publically.

    Now - some specific comments:

    "Properly locking down a data center containing 1,000 servers is far pricier than locking down a data center with 10 servers."

    I disagree. To lock down 1000 server operating systems installed on real hardware is exactly the same cost as locking down 1000 virtualized servers. They all still need hardening, upgrades, anti-virus agents and routine patching. They all still need vulnerability scans & periodic auditing. Those costs are a wash. The associated network and firewall lock down cost is also the same, but the virtualized environment has an additional cost associated with locking down the hypervisors and the VM host operating system (Red Hat EL in the case of VMware).

    My take so far on virtualization is that you really only save on server hardware costs, power/cooling costs, and network costs. You have a significant increase in storage costs. At $800 per fiber channel HBA, and $2000 per fiber channel switch port, the SAN attach costs are approaching the cost of the server itself, and if you are going to do anything other than trivial virtualization, you'll have to get SAN attached (redundantly, of course). All the other costs related to system management, operations patching, etc. are either the same or higher.

    "It’d also be swell if I could quickly make a few copies of my corporate server"

    Making a copy of a corporate server and taking it with you on your trip would be a bad idea in my book. And we tend to leave copies of VM's laying around the datacenter, as leftovers from upgrades or config changes. Those copies are potential private data loss incidents sitting around waiting to happen.

    "When one server can so suddenly become three or four, the risk of server sprawl is very real, and can defeat the original purpose of server consolidation"

    I couldn't agree more. "Let's just spin up another VM.....".

    "'So perhaps the HR department uses one VM for Web browsing and another VM—one which is never connected to the Internet, nor even has a Web browser—that is devoted exclusively to the task of running the PII database software."

    I agree with the idea of separation of unprivileged apps (browsing) from privileged apps (HR). We've been doing similar sorts of desktop separation for quite a while. But for all practical purposes, a better method of separating the functions would be to put the secure app on a server running RDP or Citrix that is configured with minimal privileges, can't access the Internet, and is safely tucked away in a rack in the datacenter. It would be far easier to deploy a thin client/remote desktop solution than multiple VM's on a whole bunch of desktops, and you'd have the same privilege separation.

    "The wisest course of action may be to wait until 1) the virtualization security tools are more mature and 2) your organization is prepared to adequately invest in the superior security the virtual environment demands. Only then should you strap on your SCUBA gear and take the plunge."

    Yep - that's about where I am at on the topic. Virtualization where appropriate, without crossing security boundaries.

    Thanks for the great article.


  3. All good points, Mike.

    Just to be clear, when I said "It'd be swell if I could make a couple of copies of the corporate server" I was sarcastically speaking as though I were an average gal thinking only of convenience, and not the squirrely security geek that I am. The thought of someone actually doing such a thing makes me cringe.

    You say: "To lock down 1000 server operating systems installed on real hardware is exactly the same cost as locking down 1000 virtualized servers."

    This is an interesting point, and--not surprisingly--quite a different stance than that taken by many virtualization vendors.

    It sounds, though, that you're actually spending more on securing your virtualized servers, (since the virtual environment needs better security) and saving money only on the non-security operational costs. Is that fair to say?

  4. What is interesting about the 'copy of the corporate server' is not that you'd take one on the road with you, but rather that with VM's it is really easy to keep copies of servers laying around, sensitive data or not. It is so easy that it happens all the time. In some cases, such as DR or business continuity, we want copies of VM's stored off line or of site to achieve desired RPO/RTO goals. In other cases, the copies are left laying around for the convenience of the server manager. Those copies are a potential security problem, just like loose backup tapes. Releasing a copy of sensitive data generates the same headlines as releasing the original, but because the copies are scattered about uncatalogued, the probability of accidental release is higher.

    Your sarcasm make me think - ding! - I have to secure every copy of every VM just the same as the original host, and to do that I need security and auditing and change control on the spot where I store the 'spare' VM's that is at least equal to the security requirements of the most sensitive host copy that I store. I think my costs just went up. ;)

    So if I want to hack a datacenter and grab interesting data from a server, I have the virtualized server containing the data as a target, the host server that is hosting the live virtual server as a target, the tape backups (or backup-to-disk-pool) as a target, and the big cheap NAS device where the sys admin stores copies of the VM's off to the side 'just in case'. I'd aim for the latter first.

    On ROI, because of the uncertainty surrounding hypervisor and VM host security, I'm not yet willing to cross security compartments on shared hardware, so the ROI has to be based on the consolidation of servers within a netblock (or VLAN, or firewall interface, or what I call a security compartment). In the model I use to design a hosting infrastructure, I have lots of small vlans with lots of firewall interfaces, so the server count per compartment is low, and hence I have a hard time consolidating. I can't do 10:1 so I don't save enough on hardware to make an easy ROI on the increased person-cost of having to manage another platform (ESX). Others, who either are willing to cross security boundaries or who can run higher ratios will see different numbers. Of course in some cases, such as test/dev/prototyping/proof-of-concept, Vm's are clearly an easy ROI win.

    And - I tend to have a different take than vendors on a lot of technology. Their job is to sell, my job is to be skeptical. ;)