Skip to main content

The Cloud - A New Provider Failure Mode

swat-jpg I certainly would not have thought of this failure mode. A law enforcement agency raids a datacenter and grabs the hardware that your provider uses to host your business critical application.

The FBI has seized all equipment belonging to our customers. Many customers went to the data center to try and retrieve their equipment, but were threatened with arrest.

Let’s assume that some customer in a cloud or co-lo is doing something bad. The law enforcement agency doesn’t understand clouds, virtualization VMotion, or hypervisors. They understand computers. Which one is it? Who knows. Grab them all.

I’m not clear on the details of what happened in this particular event. It’s possible that the provider was a legitimate target of law enforcement. From the point of view of someone who has a critical application tied up in the mess, the details are not as important as the concept. If someone, somewhere in the same co-lo or cloud as you commits some crime, what will law enforcement do, and how will you protect yourself against heavy ham handed agents and judges that don’t understand technology?

I’m thinking that we need to add a couple more questions to the cloud/co-lo RFP & vendor qualification forms:

  1. Does your security staff include former law enforcement, and if so, do they still socialize with current law enforcement agents?
  2. How many full time lawyers do you employ?

Question #1 gets you fair warning of the impending raid, and #2 gets your servers back after the raid.

In the mean time, you presumably have recovered your operations at some other co-lo, in some other cloud.


  1. #1 - Many LE seizures are under court seal, so that doesn't necessarily cover.

    #2 - LE seizures almost always result in return of assets unless other circumstances come into play, but you might be able to make it faster.

  2. I don't think a court seal necessarily stops data leakage amongst the close fraternity of law enforcement officers.

    Having said that, the action items were somewhat tongue in cheek. The key is that if one values availability, one must assume that the provider can disappear with no notice, and one must plan accordingly.


Post a Comment

Popular posts from this blog

Cargo Cult System Administration

Cargo Cult: …imitate the superficial exterior of a process or system without having any understanding of the underlying substance --Wikipedia During and after WWII, some native south pacific islanders erroneously associated the presence of war related technology with the delivery of highly desirable cargo. When the war ended and the cargo stopped showing up, they built crude facsimiles of runways, control towers, and airplanes in the belief that the presence of war technology caused the delivery of desirable cargo. From our point of view, it looks pretty amusing to see people build fake airplanes, runways and control towers  and wait for cargo to fall from the sky.
The question is, how amusing are we?We have cargo cult science[1], cargo cult management[2], cargo cult programming[3], how about cargo cult system management?Here’s some common system administration failures that might be ‘cargo cult’:
Failing to understand the difference between necessary and sufficient. A daily backup …

Ad-Hoc Versus Structured System Management

Structured system management is a concept that covers the fundamentals of building, securing, deploying, monitoring, logging, alerting, and documenting networks, servers and applications. Structured system management implies that you have those fundamentals in place, you execute them consistently, and you know all cases where you are inconsistent. The converse of structured system management is what I call ad hoc system management, where every system has it own plan, undocumented and inconsistent, and you don't know how inconsistent they are, because you've never looked.

In previous posts (here and here) I implied that structured system management was an integral part of improving system availability. Having inherited several platforms that had, at best, ad hoc system management, and having moved the platforms to something resembling structured system management, I've concluded that implementing basic structure around system management will be the best and fastest path to…

The Cloud – Provider Failure Modes

In The Cloud - Outsourcing Moved up the Stack[1] I compared the outsourcing that we do routinely (wide area networks) with the outsourcing of the higher layers of the application stack (processor, memory, storage). Conceptually they are similar:In both cases you’ve entrusted your bits to someone else, you’ve shared physical and logical resources with others, you’ve disassociated physical devices (circuits or servers) from logical devices (virtual circuits, virtual severs), and in exchange for what is hopefully better, faster, cheaper service, you give up visibility, manageability and control to a provider. There are differences though. In the case of networking, your cloud provider is only entrusted with your bits for the time it takes for those bits to cross the providers network, and the loss of a few bits is not catastrophic. For providers of higher layer services, the bits are entrusted to the provider for the life of the bits, and the loss of a few bits is a major problem. These …