Skip to main content

Posts

Showing posts from 2008

2008 Year End Summary

It’s been almost a year since I started blogging. Sam Buchanan, who has been blogging since 2001, tried to get me started a couple times years ago, but I didn’t really think I had anything to write about, or maybe I thought that nobody would read what I wrote, or maybe I wouldn’t admit that I didn’t really know what  blog was, so I never started. My boss finally convinced me to start writing, and this blog is the result. I’m probably late to the party, as the trend seems to be shifted toward micro-blogs or Tweets. I’m a fan of well written, original thoughts in the longer blog format though, so that’s what I’ve tried to present here on this blog.Here’s a short summary of the interesting posts from the first year. Security related posts include a post on protecting yourself from your own applications.  It’s a concept that we’ve used for years that unbeknownst to me is closely related to Biba and BLP. Hopefully I’ve recorded the essence. I also wrote some thoughts on de-provisioning as …

If it can browse the Internet, it cannot be secured

Tired of IE’s vulnerabilities?

You could switch to Firefox, but if you were honest, you’d have to admit that you still can’t declare yourself secure. Or you could try Opera, but then you’d have to manage critical patches also, though perhaps less frequently. There is nothing about Chrome or Safari that indicates that using them will make you secure. They may have fewer vulnerabilities, or it may be that fewer of their vulnerabilities have been discovered and published. You may be more vulnerable or less vulnerable by switching browsers, but you will still be vulnerable. Throw in cross platform vulnerabilities and the combined vulnerabilities of the various third party browser addons & the menu looks pretty bleak.

Frankly, as the threats from the Internet have evolved over the last decade or so, I’ve not seen a huge difference between the security profiles of the various browsers. Some have fewer vulnerabilities, some have more; some have an easier selection of somewhat more secur…

Startups and Early Adopting Customers

Any product, no matter whose it is, will only meet a fraction of your needs. Working with early startups gives you the ability to influence a product early in its life cycle and increase that fraction. You get to nudge a product in a direction that matters to you, while the startup gets unvarnished, raw, but valuable product feedback.

In a recent post on Security for All, Joseph Webster describes risk to innovation that startups face when transitioning to established corporations. From the point of view of a customer of startups, the transition that the startup needs to make is also interesting:


In a small startup everyone is intimately familiar with the customers, whereas large corporations have to make concerted efforts to allow a design engineer to even have marginal contact with a customer - and that’s usually second hand through either a sales or marketing initiative.[1]
As a customer, I’ve seen both ends of the spectrum. My team was one of the early customers of LogLogic b…

The Cloud – Provider Failure Modes

In The Cloud - Outsourcing Moved up the Stack[1] I compared the outsourcing that we do routinely (wide area networks) with the outsourcing of the higher layers of the application stack (processor, memory, storage). Conceptually they are similar:
In both cases you’ve entrusted your bits to someone else, you’ve shared physical and logical resources with others, you’ve disassociated physical devices (circuits or servers) from logical devices (virtual circuits, virtual severs), and in exchange for what is hopefully better, faster, cheaper service, you give up visibility, manageability and control to a provider. There are differences though. In the case of networking, your cloud provider is only entrusted with your bits for the time it takes for those bits to cross the providers network, and the loss of a few bits is not catastrophic. For providers of higher layer services, the bits are entrusted to the provider for the life of the bits, and the loss of a few bits is a major problem. The…

Janke’s Official 2009 Technology Predictions

I’ll take Anton’s bait.Here they are:Prediction 1: The rate of adoption of IPV6 will greatly accelerate. Estimates of the final shutdown date for the last v4 global route will be moved up from ‘when hell freezes over’ to ‘long after I’m retired’, placing the problem right next to the Year 2038 Unix timestamp problem on CTO’s priority lists.Prediction 2: Gadget freaks will continue to search for the holy grail of multifunction all-in-one gadgets. They will continue to be disappointed. Prediction 3: Apple will announce a new product. The product will be generate a media frenzy. Apple fans will crash servers looking for the latest product leaks or fuzzy prototype pics, and arguing via blog comments the merits of the features the product may or may not have. Unfortunately the product will be missing cut and paste.Prediction 4: Hardware and network vendors will continue making faster and cheaper bits at a rate that matches Moore's law. Software will continue to bloat at a rate just sli…

The Power Consumption of Home Electronics

I learned something last week. Xbox and Playstation Game consoles are pathetically bad at energy consumption. The Wii doesn’t suck (power) quite as badly.

The Data:

The Natural Resources Defense Council did an interesting study[1] of game consoles and attempted to estimate annual energy usage and cost.

The good part:

Ouch. Unlike half watt wall warts, a hundred and some odd watts might actually show up on your monthly electric bill. And from what NRDC can tell, the game consoles are not real good at powering themselves off when unused, which makes the problem worse.

This is really discouraging. The idea that energy consuming devices should automatically drop themselves down into a low-power state when idle isn’t new, yet we continue to build (and buy) devices with poor power management. I suspect that part of the problem is that there isn’t sufficient information available to consumers at the time of purchase to make a rational ‘green’ decision. Unlike refrigerators, clothes washers, an…

Amusing log messages

I give Cisco credit for fully documenting firewall log messages. In theory this gives users the ability to set up a system for catching interesting log messages and ignoring uninteresting messages. More vendors should be so bold as to actually acknowledge that their products log messages, and that those messages need to be documented.

This level of disclosure has an interesting side effect. I'm not sure what I'd do if one of our ASA's  logged this error:
Error Message    %ASA-2-716515:internal error in: function: OCCAM failed to allocate memory for AK47 instanceExplanation    The OCCAM failed to allocate memory for the AK47 instance.Or this error:
Error Message    %ASA-2-716508: internal error in: function: Fiber scheduler is scheduling rotten fiber. Cannot continuing terminatingExplanation    The fiber scheduler is scheduling rotten fiber, so it cannot continue terminating. 
Fiber rot?

An AK47 instance?

No doubt those messages mean something to someone  at the TAC. For …

The Cloud – Outsourcing Moved up the Stack

Why is outsourcing to ‘the cloud’ any different than what we’ve been doing for years?
The answer: It isn’t.
We’ve been outsourcing critical infrastructure to cloud providers for decades. This isn’t a new paradigm, it’s not a huge change in the way we are deploying technology. It’s pretty much the same thing we’ve always been doing. It’s just moved up the technology stack.
We’ve been outsourcing layer 1 forever (WAN circuits), layer 2 for a couple decades (frame relay, ATM, MPLS), and sometimes even layer 3 (IP routing, VPNs’) to cloud providers. Now we have something new – outsourcing layers 4 through 7 to a cloud provider.
So we are scratching our heads trying to figure out what this ‘new’ cloud should look like, how to fit our apps into a cloud and what the cloud means[1] for security, availability and performance. Heck we’re not even sure how to patch the cloud[2], or even who is responsible for patching a cloud.
I’ll argue that outsourcing CPU, database or storage to a server/appl…

The Patch Cycle

The patch cycle starts again, this time with a bit of urgency.  A 'patch now' recommendation has hit the streets for what seems to be an interesting Windows RPC bug.

What does 'patch now' mean this time? Hopefully it means a planned, measured and tested patch deployment, but at an accelerated schedule.

It's a Microsoft patch, and that's a good thing.  The routine of monthly Microsoft security patches has been honed to a fine art in most places, making Windows OS patches by far the simplest and most trouble free of the platforms that we manage. This one appears to be no exception, at least so far.

Just for grins I drew up a picture of what a typical Microsoft Windows patch cycle looks like. The patch kits show up once a month. Most months have at least one 'important' patch, so most monthly patches get applied. Life is easier if you can fit the patch cycle into a one month window, just because the probability of missing a patch or patching out of order is g…

Missing the Point

ExtremeTech reviewed the new Fit-PC Slim.

Conclusion:

CompuLabs really needs to step up to a more modern platform if it wants to stay competitive in the rapidly growing market for small, net-top PCs.[1]
They missed the point. It's not a "net-top" or desktop replacement, it's an extremely low wattage home server.

The spec that matters:
Power consumption: 4-6W[2]Compare that to the 50-100w of typical desktops that are used as home servers & left running 24 hours per day, or the 20+ watts of a typical notebook. Even an Eee PC uses 15 watts.

If what you need is a home server to use as a samba share, a web server or similar always-on device, a 5 watt brick looks pretty interesting. That's 500kwh/yr less power, 400kg less CO2, and $50 less on your electric bill per year than the old desktop-turned-server that you have stuffed under your desk.

And don't whine about the 500mhz processor and 500mb RAM. We ran LAMP stacks that served up more users than your house ever wil…

Wide Area Network Outage Analysis

The following is an brief analysis of unplanned network outages on a large state wide network with approximately 70 sites at bandwidths from DS3 to GigE. The data might be interesting to persons who need to estimate expected availability of wide area networks.

The network is standard core, hub, spoke, leaf. The core is fully redundant. The hubs have redundant circuits connecting to multiple hubs or cores, redundant power and partially redundant hardware. The spokes and leaf sites are non-redundant. 

The source or the data was a shared calendar where outages were recorded as calendar events. The data was gathered at analysis time and is subject to omissions and misinterpretation. Errors likely are undercounts. Raw data, by approximate cause88 Total Outages 290 Total Hours of Outage 2 years calendar time Failures by type and duration

Cause# of Incidents Percent # of Hours Percent Circuit Failures34 39% 168 58% Equip Failures24 66% 60 79% Power Failures22 91% 53 97% Unknown5 97% 7 99% Other3 100% 2 100% Total…

There are some things about computers I really don’t miss…

There are some things about computers I don’t think I’m ever going to miss. Nostalgia has limits.I’m not going to miss:Programming machine tools using paper tape and a Flexowriter, and copying the paper tape to Mylar tape for production. But only if it was a good program, one that didn't drill holes in the wrong place on an expensive casting or smash the machine tool spindle into the tooling fixture and break really expensive stuff. Submitting a punch card deck to the mainframe operators, waiting four hours for the batch scheduler to compile and run the program, only to find a syntax error. Especially for a required assignment the last week of the semester. Waiting for a goofy little homemade PDP-8 to assemble, link and load a 50 line assembler program (about 40 minutes of watching tape cartridges spin, if memory serves.) Booting CAD/CAM systems by toggling bits and loading instructions from front panel switches. And then programming complex machine tools using a strange path desc…

Bank of America SafePass Authorization

Unlike American Express, Bank of America seems to have pretty decent account claiming, user id and password requirements. Additionally, BofA allows account holders to set up SMS alerts on various types of account activity.

The login process can be tied to SafePass® SMS based authentication. To complete the login process, BofA sends a six digit code to your cell phone. The code and your normal password are both required for access to your on line account.Additionally, BofA automatically uses the SMS based SafePass® for changes to the account, including alerts, e-mail address changes, account claiming etc. You also can set up your account to send SMS alerts on significant account activity and any/all changes to account profiles, including on line charges, charges greater than a specific amount and international charges.The user id and passwords are also allowed to be significantly more complex than American Express, allowing more than 8 characters and permitting various non-alphanumeric…

Trivial Account Reset on American Express Accounts (Updated)

2008-10-06 Update: I did eventually get an e-mail notice sent to the e-mail associated with the account about 6 hours after I reset my password.It still looks to me like the account can be hijacked, and the password restrictions and suggested examples are pathetic.Account claiming is an interesting problem. The tradeoffs necessary to balance ease of use, security and help desk call volume are non-trivial.2008-10-05 9:59 PM:I'm a bit disappointed how easy it was to recover online access to my American Express account.Enter the card number Enter the four digit card ID number on the front of the card Enter my mothers maiden name That's all you need. The first two numbers are obtainable by possession of the card, the third is readily available from on line searches. Enter those three bits of info and you get a screen with your user name and the option to set a new password. Set up a new password and you have full access, including the ability to request new cards, change e-mail a…

Essential Complexity versus Accidental Complexity

This axiom by Neal Ford[1] on the 97 Things wiki:It’s the duty of the architect to solve the problems inherent in essential complexity without introducing accidental complexity.should be etched on to the monitor of every designer/architect. The reference is to software architecture, but the axiom really applies to designing and building systems in general. The concept expressed in the axiom is really a special case of ‘build the simplest system that solves the problem', and is related to the hypothesis I proposed in Availability, Complexity and the Person Factor[2]:When person-resources are constrained, highest availability is achieved when the system is designed with the minimum complexity necessary to meet availability requirements.Over the years I’ve seen systems that badly violate the essential complexity rule. They’ve tended to be systems that were evolved over time without ever really being designed, or systems where non-technical business units hired consultants, contractor…

Unplug Your Wall Warts and Save the Planet?

Do wall warts matter?

(09/29-2008 - Updated to correct minor grammatical errors. )

Let's try something unique. I’ll use actual data to see if we can save the planet by unplugging wall transformers.

Step one – Measure wall wart power utilization.
Remember that Volts x Amps = Watts, and Watts are what we care about. Your power company charges you for kilowatt-hours. (One thousand watts for one hour is a kWh).

Start with one clamp-on AC ammeter, one line splitter with a 10x loop (the meter measures 10x actual current)and one wall wart (a standard Nokia charger for an N800).

And we have zero amps on the meter.

OK - That meter is made for measuring big things, so maybe I need a different meter. Lesson one Wall warts don't draw much current. They don't show up on the ammeters' scale even when amplified by a factor of 10.

Try again - this time with an in-line multimeter with a 300mA range.


Children - don't try this at home - unless you are holding on to your kid brother and…

The Path of Least Resistance Isn't

09/29-2008 - Updated to correct minor grammatical errors.

When taking a long term view of system management

The path of least resistance is rarely the path that results in the least amount of work.
As system managers, we are often faced with having to trade off short term tangible results against long term security, efficiency and stability. Unfortunately when we take the path of least resistance and minimize near term work effort, we often are left with systems that will require future work effort to avoid or recover from performance, security and stability problems. In general, when we short cut the short term, we are creating future work effort that ends up costing more time and money than we gained with the short term savings.

Examples of this are:
Opening up broad firewall rules rather than taking the time to get the correct, minimal firewall openings, thereby increasing the probability of future resource intensive security incidents.Running the install wizard and calling it pr…

Privacy, Centralization and Security Cameras

09/29-2008 - Updated to correct minor grammatical errors.

The hosting of the Republican National Convention here in St Paul has one interesting side effect. We finally have our various security and traffic cameras linked together:
http://www.twincities.com/ci_10339532
“The screens will also show feeds from security cameras controlled by the State Patrol, Minnesota Department of Transportation, and St. Paul, Minneapolis and Metro Transit police departments.
Before the RNC, there was no interface for all the agencies' cameras to be seen in one place. Local officials could continue to use the system after the RNC.” (Emphasis mine)So now we have state and local traffic cameras, transit cameras and various police cameras all interconnected and viewable from a central place. This alone is inconsequential. When however, a minor thing like this is repeated many times across a broad range of places and technologies and over a long period of time, the sum of the actions are significant. In th…

Scaling Online Learning - 14 Million Pages Per Day

Some notes on scaling a large online learning application.

09/29-2008 - Updated to correct minor grammatical errors.
Stats:29 million hits per day, 700/second 14 million .NET active content pages per day[1]900 transactions per second 2000 database queries per second 20 million user created content files Daily user population of over 100,000 Database server with 16 dual core x64 CPU's, 128GB RAM, Clustered Nine IIS application servers, load balanced The largest installation of the vendors product Breadth and complexity. The application is similar to a comprehensive ERP application, with a couple thousand stored procedures and thousands of unique pages of active web content covering a full suite of online learning applications, including content creation and delivery, discussions, quizzing, etc. The application has both breadth and depth, and is approximately as complex as a typical ERP application. This makes tuning interesting. If a quarter million dollar query pops up, it can be t…