Skip to main content

When the weather map look like this….


Odds are the traffic map will look something like this:


I’m sure there is a parallel between the DOS attacks that mother nature periodically foists us and internet security. I’ll take a stab at describing the parallels.

Predictability: Snow storms and hurricanes are very predictable (compared to tornadoes, where one has 0-10 minutes warning and rarely has accurate predictions). It is possible to prepare for weather that can be predicted. In certain regions, snow storms or hurricanes are a high enough probability event that you will certainly experience them. The probability of a major snow storm  hitting my house in a particular winter is close enough to ‘one’ that it might as well be ‘one’. Tornadoes, on the other hand, even though there are dozens per year in my region, are localized enough that I probably will never experience a direct hit on my house.

I might tend to be prepared for a predictable event (snow storm), but rest assured that I have not taken any significant precautions for a tornado. I’m playing the odds on that one.

Preparation: Many people prepare for predictable events, some do not. I’m a lifetime veteran of snowstorms, yet I was replacing shear pins and changing oil on my snow blower in the middle of the DOS attack (snow storm). I could have done that in summer, but man was it hot last summer. Way too hot to be changing oil on a snow blower. On the other hand, 2/3 of my vehicles are true 4wd and my 4wd’s have dedicated winter tires, so I normally can get to where I have to go whether I clean my driveway or not. Local governments spend a fortune on DOS (snow storm) preparation.  They have snow removal equipment, snow removal planning, emergency notification, etc. Their preparation allows me to function fairly well even when I am not prepared.

Preparation costs money though, as I can attest when I fill up the gas tanks on my 4wd’s. They cost money every day I drive them; they have twice as many driveline parts and are expected to incur significant driveline maintenance costs, but I only really need them a couple days per year.

Preparation has limits though. Even though I may be able to make it down an unplowed street, my neighbor may not have made it down that street & may be blocking my path, or worse, my neighbor may lose control of his car and whack my car, disabling me in spite of my preparation. In rural areas – the wide open prairie around here – you will be limited by visibility (white out), not traction, so your 4wd vehicle will only serve to get you deep enough into trouble that you can’t dig yourself out.

Don’t ask me how I know. 

As it turned out, 4wd vehicle #1, a Subaru with 7in of ground clearance ,was expected to be operable in an event of magnitude ‘n’ (4-5” of snow), or perhaps marginally operable in an event of ‘2n’ (8-10 inches of snow). It was not expected to be operable in a ‘4n’ event (16-20” of snow) and predictably was not operable on unplowed roads last weekend. My 4wd vehicle #2 on the other hand, a robust pickup truck, was expected to be operable in a ‘4n’ event. After a half hour of trying to get the vehicle out to a plowed road so I could take my pharmacist neighbor to work at the 24 hour pharmacy, I concluded that getting 4wd vehicle #2 back into my driveway would be a far more reasonable outcome. Apparently I’ve either configured 4wd vehicle #2 wrong, or I didn’t have an adequate pre-purchase test plan. The currently working theory is that even though it was purchased for a ‘4n’ event, it is only configured for a ‘3n’ event.

There is no doubt that one could prepare for a ‘4n’ event like last weekend. I’d like to think that someone made a serious calculation on dollars spent versus level of preparation. Odds are though, that nobody did. It probably went more like ‘here is how much money you can have, be as prepared as possible given that constraint’.

Or – in my case – spend whatever is necessary to prepare for a ‘4n’ event, but then configure it wrong and inadequately test it, and watch as it fails to manage the event.

Incident handling: During a storm of magnitude ‘n’, a prepared person might conduct business as usual, perhaps with reduced capacity or response time. For example, one  might still get to work on time, but suffer a longer commute. A storm of magnitude ‘2n’ might cause a prepared person to have degraded operations, cancelling non-essential activities. A storm of magnitude ‘4n’ might cause most activity to come to a halt. Preparation can affect the value of ‘n’. A snow storm that would shut down Washington DC would probably have only a minor effect in Minneapolis or Buffalo.

Last weekends storm might have been a ‘4n’ event – something that maybe occurs every 20 years or so. The round red signs in the above image are closed roads. You get a ticket if you try to drive on them. Odds also are pretty good that you’d fail to make it to the other end of that particular road. The MSP airport has probably the best winter capability of any airport, yet they ended up more closed than not. Most of the municipal snow plowing was halted during the worst of the storm, buses were halted, and even fire trucks and ambulances were severely affected. Operations were certainly degraded during the DOS.

Degraded operations: During the DOS attack, most local governments have some ability to operate in a degraded mode. For example, the State Patrol may close roads, shedding load by restricting traffic to emergency vehicles only. Snow plows may stop plowing streets and only venture out to open up streets for emergency vehicles, airports may restrict incoming flights, schools and business may close, etc. Degraded operations are an accepted outcome of large scale DOS attacks (snow storms), and most entities have a pretty good idea what service need to be maintained during a DOS attack (snow storm).

In my case, degraded operations mode means avoiding travel, maintaining power, heat, Internet and food in approximately that order. Food is the easiest. I still have my Y2K stash in the basement. Year 2038 is just around the corner & I’d hate to be caught unprepared.


Popular posts from this blog

Cargo Cult System Administration

Cargo Cult: …imitate the superficial exterior of a process or system without having any understanding of the underlying substance --Wikipedia During and after WWII, some native south pacific islanders erroneously associated the presence of war related technology with the delivery of highly desirable cargo. When the war ended and the cargo stopped showing up, they built crude facsimiles of runways, control towers, and airplanes in the belief that the presence of war technology caused the delivery of desirable cargo. From our point of view, it looks pretty amusing to see people build fake airplanes, runways and control towers  and wait for cargo to fall from the sky.
The question is, how amusing are we?We have cargo cult science[1], cargo cult management[2], cargo cult programming[3], how about cargo cult system management?Here’s some common system administration failures that might be ‘cargo cult’:
Failing to understand the difference between necessary and sufficient. A daily backup …

Ad-Hoc Versus Structured System Management

Structured system management is a concept that covers the fundamentals of building, securing, deploying, monitoring, logging, alerting, and documenting networks, servers and applications. Structured system management implies that you have those fundamentals in place, you execute them consistently, and you know all cases where you are inconsistent. The converse of structured system management is what I call ad hoc system management, where every system has it own plan, undocumented and inconsistent, and you don't know how inconsistent they are, because you've never looked.

In previous posts (here and here) I implied that structured system management was an integral part of improving system availability. Having inherited several platforms that had, at best, ad hoc system management, and having moved the platforms to something resembling structured system management, I've concluded that implementing basic structure around system management will be the best and fastest path to…

The Cloud – Provider Failure Modes

In The Cloud - Outsourcing Moved up the Stack[1] I compared the outsourcing that we do routinely (wide area networks) with the outsourcing of the higher layers of the application stack (processor, memory, storage). Conceptually they are similar:In both cases you’ve entrusted your bits to someone else, you’ve shared physical and logical resources with others, you’ve disassociated physical devices (circuits or servers) from logical devices (virtual circuits, virtual severs), and in exchange for what is hopefully better, faster, cheaper service, you give up visibility, manageability and control to a provider. There are differences though. In the case of networking, your cloud provider is only entrusted with your bits for the time it takes for those bits to cross the providers network, and the loss of a few bits is not catastrophic. For providers of higher layer services, the bits are entrusted to the provider for the life of the bits, and the loss of a few bits is a major problem. These …