Apparently there is no redundancy.
I’m not thinking about this because I care about the commuters who were stranded, but rather because of how it relates to network and server redundancy and availability. My group delivers state wide networking, firewalling, ERP and eLearning applications to a couple hundred thousand students and tens of thousands of employees.
- Availability is expensive
- We hear about it when our systems suck
- We have no data that can tell us how much an outage costs. We are an .edu. Our students don’t switch vendors if they can’t access our systems for a few hours.
In that environment, how do you make a cost vs. availability decision?
Anecdote:
Years ago (circa 2001) we found a carrier that would offer us OC-3 (150mbps) for what was essentially the same price as the incumbent telco (Qwest) would charge us for two T1’s (3mbps).
- The new carrier had less experience with data. They were a cable TV provider.
- We expected availability to be worse than the incumbents telco class T1’s.
- We knew full well that the carrier had a single point of failure on a fiber path through a small town, and if there was a fiber cut in that town, 50,000 students would get knocked off the network. Unlike most carriers, they told us where they were weak.
I hesitated, figuring that it’d be best to get feedback from the campuses that were currently starved for bandwidth, so I posed the question at our quarterly CIO meeting.
Me: “If I have to make a choice, would you trade availability for bandwidth?”
CIO’s: “Yep.”
(At least that’s how I remember it…)
The carrier deliver excellent service and availability. Service was awesome, availability was at least as good as the expensive incumbent. We gained confidence in the low cost carriers ability to deliver, and year by year we migrated more campuses to the low cost carrier.
Life was good. We traded 3Mbps for 150mbps without increasing the budget. Hail to the hero (that’d be me).
Fast forward 4 years. The small town in the path of the low cost carrier’s non-redundant leg decided to build a road. The town cut the fiber, a dozen campuses disappeared from our network map for 8 hours, and to put it mildly, the campuses were not happy.
It gets better worse though. The carrier patched up the busted fiber late that day then scheduled a plow crew to come out a week later, bury a new fiber parallel to the old fiber and move us to the new fiber.
Yep, the carriers crew cut their own fiber. Another half day outage.
It gets even better worse. The carrier didn’t have facilities all they way to our core where we needed them so they leased a big carrier’s circuit to the big city. A few weeks after the two fiber outages, the big carrier in the big city smoked a network interface.
Outage number three. Let’s all throw rocks at the goat (that’d be me).
The next quarterly meeting wasn’t fun.
Suffice to say that we now make a different calculation on the relative value of bandwidth vs. availability.
Back to the broken train. A four hour outage, construction cost of a hundred million dollars per mile and no redundancy?
Damn.
