A Shack in the Woods, Crumbling at the Core


Imagine a run down shack. The kind I’m thinking of is like the cabin my grandfather bought in the late 60’s. It was about 16’x20’, built in the 30’s and then left to rot. It had wooden blocks for its foundation, it sagged about a foot in one corner as the wood blocks shifted and rotted, it was musty, moldy and damp. It was the very definition of deferred maintenance. A menagerie of non-human mobile life forms called it home. It was a shack, kind of like the one pictured here.



There are applications like that, lots of them. We probably all know where they are. Most of us will only have to look someplace in our data centers. Think of an old shack when you think of those applications.

What do you do with a shack like that? You could paint it, put new doors and windows in it, add on a bedroom and a porch, maybe even add plumbing and a bathroom. You could even secure it (from the rain) by fixing the roof. But what about the foundation? Sure you can remodel and add on to a house with a rotten foundation, but to do so is to negate much of the value of the other improvements. Your bathroom will have a sagged floor, your doors and windows won’t fit right, and your shiny new front porch will detach from the shack as the shack rots itself into the ground. You’ll have a maintenance headache that doesn’t really do what it should.

The obvious first step is to jack up the shack and build a concrete foundation. If you bite the bullet and fix the foundation first, the rest of the improvements can be built on something solid and long lasting. If the foundation is solid and square your improvements will be cheaper and easier to build. Skip that part and everything else you do will be difficult and expensive to build and poorly done at the end anyway. It’ll be a square addition added on to a crooked cabin.

Back to your rotting application - the one with the unsupported operating system and database, broken data model, the dead, archaic, insecure code that nobody understands. Are you re-skinning it to look nice? Are you grafting on features and business logic onto the sagging foundation? Or are you digging out a new footing and pouring concrete?

The business unit doesn’t know or care about the foundation, they only care about the user interface and the features. Our job is to convince them that they need to spend time and money on the foundation, not just the features.

Tell them the story about the shack in the woods.

Minimal Installations & the Consultants who Have no Clue

I've got a simple mantra that I sometimes express as the 'Least Bit Principle'. It's not complicated. In any system, install and configure the least number of services, packages, configurable entities, permissions or privileges necessary for system functionality. This goes way back to the days when one needed to disable defaults services while hardening a new server (chargen anyone?) and it applies to any complex system, such as operating systems, databases, routers, firewalls, etc. It's not new, it's not radical.

The fundamentals of this principle are self evident.
  • What is the minimum software needed to support the application functionality? (Hint: It's not 'Entire Distribution')
  • What are the minimum file system privileges necessary for application functionality? (Hint: It's not 'Everyone/Full Control')
  • What are the minimum database privileges necessary for application functionality? (Hint: It's not 'DBA' or 'DBO')
  • What are the minimum services that need to be running to support the application functionality? (Hint: You don't need chargen, rsh, rcmd or IPX)
For software installation, general expression of this principle is that if a package or feature is not installed, it does not have to be maintained, patched or upgraded, and more importantly, if the package or feature is not installed it cannot be accidentally enabled in an unconfigured or partially configured insecure state. When code red and slammer hit, how many of the victims  knew they were running SQL server or IIS? Many of them didn't know that the vulnerable software was even installed and running, much less that they had to patch it or they'd be screwed.

This is extremely valuable for Solaris and Oracle. For both of those, we are able to minimize the installations and defer a significant number of patch cycles simply because the vulnerable feature or package is not installed.  If the vulnerable software is not installed, we do not have to consider the vulnerability. It's even on Microsoft's radar. With server 2008, it is finally possible to install a minimized version of the operating system. I dream of the day when my IIS server will not have a GUI browser, and I'll be able to ignore vulnerabilities and patches that infect the pathetically insecure userland software that infests my servers.

So a vendor (Sun) offers to help out with a proof of concept. They delegate the actual install to a VAR. The consultant paid by the VAR (or Sun) shows up and starts to build an 'Entire Installation' server. We insist that 'Entire Installation', which includes software that we will never, ever use on that server, is not appropriate and does not meet our standards. We declare to the consultant that what we need is 'Reduced Networking Core System Support'. The vendor (Sun) provides and supports minimized installation options for the software (Solaris) and we expect the consultant to perform a minimal installation plus the specific packages necessary for supporting the intended application. What's so hard about figuring out a dependency tree and installing the minimum software necessary to resolve the dependencies? The consultant balked.

In this case, fatigued from having to deal with clueless consultants, we said screw it. We'll end up running the proof of concept with an 'Entire Installation', throwing it away and doing the minimal installation later when & if it moves to production. It shouldn't have to be that way though. It's 2009 and I expect consultants to think and act like it's the 21st century.

Why are all my 'vendor' posts also tagged as 'annoying'?

SMS as Two Factor Authentication Spoofable?

Speculation as to why there is an apparent rush to buy a certain model of an old Nokia cell phone.
'The 1100 can apparently be reprogrammed to use someone else's phone number, which would also let the device receive text messages. That capability opens up an opportunity for online banking fraud."
If true, then an SMS to a cell phone would no longer be reliable for two factor authentication. Impersonate a person's phone long enough to log into their bank account? That'd be amusing.

Firewall Complexity Versus Time

As a follow up to Firewall Rule (Mis)Management, I created a few simple charts showing the growth of a firewall config over time. The Y-axis is simply the size of the config in bytes. The X-axis represents time from 2002 to present.

Firewall-Config-Size-Annotated

This particular firewall is at a data center that is being phased out, so as applications get deprovisioned or moved, the configuration size shrinks.

The second chart is for a firewall for another data center that was spun up in 2005. The X-axis is time from 2005 through today. The steep change in size at the left is the initial provisioning of the apps in the new data center.

Firewall-Config-Growth

The configuration size has grown continuously since 2005. I’m expecting that it will continue to grow as more apps get hosted. There are not too many scenarios were a configuration would shrink unless major applications were phased out or the firewall manger decided to simplify the problem with a few ‘permit any any’ rules.

At some point in time it’ll be too large to mange if it isn’t already. Presumably the probability of human error increases with complexity (configuration size), so one might suppose that if firewall configuration errors cause a decrease security, then the data center becomes less secure over time.

Entropy is the enemy of security.

Anonymity is not Privacy

Let's pretend that I want to do something subversive like participate in a social network with persons who don't like the current form of government, or maybe I want to be really subversive and prevent advertisers from tracking my social network habits. In either case, preventing someone from de-anonymizing me is essential. The University of Texas paper demonstrates that by using anonymous data from multiple social networks it is possible to de-anonymize individuals and reliably identify them based on a comparison of the links between the persons in the social networks.

QWERTY is Mainstream?

Touch screens and QWERTY keyboards on mobile devices are finally mainstream just about the time when they really should be obsolete. That’s too bad, really. Voice control should be mainstream, not qwerty keypads. I should be able to have basic mobile phone control and mobile communications without reaching for my phone and poking around on it.

It’s 2009. For mobile devices, touches, swipes, swirls, stabs and keypads are archaic. Phones should be heard, not seen.

How close are we?

Microsoft Voice Command is a partial solution. It allows basic phone control with speaker independent voice. It works with Bluetooth so I can tap the headset and say things like ‘call Jake Botts at home’ or ‘dial 612 555 1212’ and it generally figures out what I want. It also allows voice access to the calendar (‘What’s my schedule’) and can navigate menus and contacts (‘show Jake Botts’ or ‘start solitaire’ or ‘start google maps’). Additionally, it reads incoming SMS’s & messages and announces incoming phone calls by name or number. It’s not comprehensive though. Once the command executes, Voice Command drops out of the picture. So I’m still stuck with viewing and touching the phone.

Motorola (and probably others) have similar features. Motorola’s speaker independent voice dialing was great at recognizing my voice with a bluetooth headset at 80mph in a convertible. But then it would sometimes get confused in a quiet room. Go figure.

Microsoft Live Search does fairly decent voice recognition on a limited set of tasks. It suffers from a few flaws. It doesn’t use bluetooth and it doesn’t speak back to me. It also fails the ‘context’ test. I tell it ‘traffic’ and instead of showing me a traffic map, it searches for driving schools. And in most cases, it’s still touch dependent.

I know about Jott and Nuance, but haven’t tried either one, and my carrier appears to offer an add-on service that has some interesting features. Vlingo has something that looks close, but it is still touch dependent. My vision is to be able to do what Vlingo can do without taking my phone from my pocket. Adando tries to solve the problem using your home computer as the smart part of the equation. Your phone bridges your voice back to your home PC. Your PC does all the work. Clever, I guess.

I think Apple sent us down the wrong path by perfecting the touch-swipe-stab-pinch-flick interface. I can’t do any of those while driving in a convertible at 80 mph. (Well…I can, but I really shouldn’t…)

Its about time we re-think phone interfaces, isn’t it?

Vendors That Don’t Support VM’s?

From a large software vendor:

VMWare support: <vendor> applications running on VMware are supported as long as the OS running in the virtual machine is certified by <vendor> (see <vendor> Platform Availability Matrix).

That’s good.

However, if <vendor> is unable to reproduce an issue the customer may be asked to reproduce the problem running natively on the certified OS, without the use of VMware. <vendor> does not explicitly test or certify products running in a VMware environment.

That’s not good.

We’ve run into this a handful of times. In this case, I’m not sure how to interpret the nuanced difference between support and certification, but it’s pretty clear that <vendor> wants to leave open the option of blaming the vm in cases where they can’t figure out what’s broke. It’s the old ‘I have no clue what’s happening, so I’ll blame the network/firewall’ problem. In theory, one would have to maintain a non-vm'd test environment to use in the event that the vendor blames your vm'd environment for their problems.

We’ve also run into an application from a major network vendor that ran fine in a vm until we applied a minor patch (something like from version 3.4.1 to 3.4.2). After the patch, the app quit working. The vendor told us ‘no vm’s’. We ended up installing a dedicated server.

Somewhere along the line software vendors are going to have to provide the same level of support for virtualized environments as they do for non-virtualized environments.

This virtualization stuff is mainstream, isn’t it?

The Cloud - A New Provider Failure Mode

swat-jpg I certainly would not have thought of this failure mode. A law enforcement agency raids a datacenter and grabs the hardware that your provider uses to host your business critical application.

The FBI has seized all equipment belonging to our customers. Many customers went to the data center to try and retrieve their equipment, but were threatened with arrest.

Let’s assume that some customer in a cloud or co-lo is doing something bad. The law enforcement agency doesn’t understand clouds, virtualization VMotion, or hypervisors. They understand computers. Which one is it? Who knows. Grab them all.

I’m not clear on the details of what happened in this particular event. It’s possible that the provider was a legitimate target of law enforcement. From the point of view of someone who has a critical application tied up in the mess, the details are not as important as the concept. If someone, somewhere in the same co-lo or cloud as you commits some crime, what will law enforcement do, and how will you protect yourself against heavy ham handed agents and judges that don’t understand technology?

I’m thinking that we need to add a couple more questions to the cloud/co-lo RFP & vendor qualification forms:

  1. Does your security staff include former law enforcement, and if so, do they still socialize with current law enforcement agents?
  2. How many full time lawyers do you employ?

Question #1 gets you fair warning of the impending raid, and #2 gets your servers back after the raid.

In the mean time, you presumably have recovered your operations at some other co-lo, in some other cloud.

A New Model for Product Purchasing

Most organizations follow an acquisition model similar to:
  • Identify Business Need
  • Gather Requirements
  • Evaluate Products
  • Select Product
  • Purchase Product
  • Implement Product
This model has generally proven to be costly and unreliable. Significant resources are spent identifying needs and requirements. Working with users to determine requirements is generally unreliable and often results in recursive need/requirement loops. Product evaluation and selection is time consuming and expensive, often requiring RFP’s, live demos and test installations. The end product rarely meets the users needs or requirements, simply because the unreliability of the user results in documented needs and requirements that do not represent actual user requirements.
The above process may be significantly streamlined by simply eliminating the unnecessary steps and reordering the remaining. The new model looks like this:
  • Purchase Product
  • Implement Product
  • Gather Requirements
  • Identify Business Need
This minor modification of the standard model has the following benefits.
  • Formal product evaluation by highly paid technical staff is eliminated. Products are instead evaluated by VP’s and executives at vendor provided lunches, social gatherings, sporting events, etc. Product selection is likewise eliminated, as it derives directly from product purchase.
  • Product requirements can be extracted from vendor documentation and product brochures more efficiently than developing requirements from scratch. Likewise, when an RFP or similar formal purchasing process is required, the process can be significantly streamlined by extracting requirements from vendor product brochures. Care must be taken to avoid the ‘ransom note look’ when copying and pasting vendor materials. Pasting as plain text into a preformatted document is preferred.
  • Business needs are easily derived from post implementation product walk through and demonstrations. Business users will realize their needs as they explore the product user interface and features during product training. An entire layer of analysis is thereby eliminated.
  • Entire teams of business analysts and architects are redundant in this model. The vendor’s analysts and architects have already provided much of the functionality that these roles normally perform. Substantial cost savings will result from the elimination of these roles.
  • Project risk is minimized. When needs and requirements are derived from the documentation of an already purchased product, there are no unmet needs or requirement gaps except in cases where the vendor product documentation is inaccurate. Change orders can be executed to correct the mismatch. Project success is virtually assured.
In the particular case of highly technical products where customers are unlikely to realize that they have an unmet need (security products, for example) this method assures that customer needs are met even when customer awareness of needs is insufficient to result in customer initiated product evaluation and purchase.

A parallel model may be used for software development. In the software development derivative of this model, developers perform requirements gathering as they write the software. As in the product purchase model, business needs are derived from the functionality of the completed software. Individual developers assume the roles of business analyst and architect. Their familiarity with the functionality of the software assures that the software functions as requirements dictate, as all requirements are derived from existing software functionality.

The matching of functionality to requirements is simplified by careful construction of module and function names. This will allow use of regular expression matching and word substitution to automatically build requirements from source code. For example, for a source file containing the following pseudo code[1]:
function store_customer_records 
{ 
code.... 
}
This simple command:
cat <source file> </ source file>| sed -n \
'/^function/s/\(function\)\(.*\)/The product will \2\./p' | tr '_' ' ' 

Will extract the following requirement:

The product will store customer records.
The post-processing of requirements can easily be managed as a part of normal software deployment. As new versions of the software are deployed, the updated requirements can be automatically extracted and e-mailed to project managers by simple post-processing scripts attached to commonly available automated deployment systems.

A significant fraction of all product purchase and software development already follows derivatives of this model. Additional refinement and wider implementation of this model should result in improvements in efficiency, productivity and user satisfaction.


UPDATE 04/09/2009 After testing the new model on a recent large software purchase, it’s clear that I’ve obviously left out a critical step in the proposed model. An additional step should be added to the tail end of the process:
  • Analyze and Document Security Requirements
The complete model should be:
  • Purchase Product
  • Implement Product
  • Gather Requirements
  • Identify Business Need
  • Analyze and Document Security Requirements
Adding security requirements to the end of the document does not significantly alter the procurement process. The requirements are derived from existing functionality, making the process largely pro forma.

[1]Note: Regex matches on object oriented code should extract public methods only. The publishing of private methods is a security violation.