Thursday, April 30, 2009

A Shack in the Woods, Crumbling at the Core

Imagine a run down shack. The kind I’m thinking of is like the cabin my grandfather bought in the late 60’s. It was about 16’x20’, built in the 30’s and then left to rot. It had wooden blocks for its foundation, it sagged about a foot in one corner as the wood blocks shifted and rotted, it was musty, moldy and damp. It was the very definition of deferred maintenance. A menagerie of non-human mobile life forms called it home. It was a shack, kind of like the one pictured here.


There are applications like that, lots of them. We probably all know where they are. Most of use will only have to look someplace in our data centers. Think of an old shack when you think of those applications.

What do you do with a shack like that? You could paint it, put new doors and windows in it, add on a bedroom and a porch, maybe even add plumbing and a bathroom. You could even secure it (from the rain) by fixing the roof. But what about the foundation? Sure you can remodel and add on to a house with a rotten foundation, but to do so is to negate much of the value of the other improvements. Your bathroom will have a sagged floor, your doors and windows won’t fit right, and your shiny new front porch will detach from the shack as the shack rots itself into the ground. You’ll have a maintenance headache that doesn’t really do what it should.

The obvious first step is to jack up the shack and build a concrete foundation. If you bite the bullet and fix the foundation first, the rest of the improvements can be built on something solid and long lasting. If the foundation is solid and square your improvements will be cheaper and easier to build. Skip that part and everything else you do will be difficult and expensive to build and poorly done at the end anyway. It’ll be a square addition added on to a crooked cabin.

Back to your rotting application - the one with the unsupported operating system and database, broken data model, the dead, archaic, insecure code that nobody understands. Are you re-skinning it to look nice? Are you grafting on features and business logic onto the sagging foundation?  Or are you digging out a new footing and pouring concrete?

The business unit doesn’t know or care about the foundation, they only care about the user interface and the features. Our job is to convince them that they need to spend time and money on the foundation, not just the features.

Tell them the story about the shack in the woods.

Tuesday, April 28, 2009

Unsolicited E-mail Containing Security Advice

body-logo[1] Here’s one for the what-were-they-thinking files.

I recently received an e-mail from a vendor that I’ve never heard of and with whom I’ve never done business:

From: "USA.NET" <>

To: <***.***@***.edu>

Date: 3/6/2009 11:28 AM

Subject: Weekly Security Update

Another wave of scam emails are circulating disguised as warnings from the U.S. Federal Reserve Bank. They lead to a fraudulent website that installs phishing/malware software or a Trojan virus that will send out critical information such as user names, passwords, or file contents of sensitive documents.

Always use best security practices when reviewing your email and do not open any email attachments from unknown senders.,***,***,****,*****,***,vn

To unsubscribe, send an email to: with the address: ***.***@***.edu in the subject line.

An unsolicited e-mail from an unknown sender advising me not to click on attachments in e-mails from unknown senders? The e-mail contains a link however, which presumably I’m supposed to follow. That’s amusing. According to this vendor, links must be safe, whereas attachments are not. The subject is generic, and there is no indicator that the sender knows who I am.

This really is indistinguishable from a phishing attempt.

Monday, April 27, 2009

NYPD: Attacks or Noise?

Gasp! Panic! 70,000 attacks per day against the NYPD! From China no less! It must be a grand conspiracy! Black Red Helicopters!

Either that or it's just the normal background noise of the Internet. Let me check.

Let's try 'cat *.log | grep Deny | wc -l' on today's logs:

00:00 - 01:00 = 2,437,222
12:00 - 13:00 = 4,071,284
13:00 - 14:00 = 3,323,089

That's enough data.  This is a blog post, not peer reviewed research.

Figure 3 million per hour * 24 hours = umm... big  numbers...let me get my calculator.....Start/Run/calc.....3 * 24 = 72...

If an 'attack' is a denied packet, then seventy-two million 'attacks' per day is normal for a medium sized network ( two /16's and a  /19.  My guess is that some of them even come from China.

I think it's just noise.

Don't tell anyone though. NYPD is  probably going after Homeland Security funding.

Oh, and one more question. How do they know 'all the attempts have failed'.

Thursday, April 23, 2009

Minimal Installations & the Consultants who Have no Clue

I've got a simple mantra that I sometimes express as the 'Least Bit Principle'. It's not complicated. In any system, install and configure the least number of services, packages, configurable entities, permissions or privileges necessary for system functionality. This goes way back to the days when one needed to disable defaults services while hardening a new server (chargen anyone?) and it applies to any complex system, such as operating systems, databases, routers, firewalls, etc. It's not new, it's not radical.

The fundamentals of this principle are self evident.
  • What is the minimum software needed to support the application functionality? (Hint: It's not 'Entire Distribution')
  • What are the minimum file system privileges necessary for application functionality? (Hint: It's not 'Everyone/Full Control')
  • What are the minimum database privileges necessary for application functionality? (Hint: It's not 'DBA' or 'DBO')
  • What are the minimum services that need to be running to support the application functionality? (Hint: You don't need chargen, rsh, rcmd or IPX)
For software installation, general expression of this principle is that if a package or feature is not installed, it does not have to be maintained, patched or upgraded, and more importantly, if the package or feature is not installed it cannot be accidentally enabled in an unconfigured or partially configured insecure state. When code red and slammer hit, how many of the victims  knew they were running SQL server or IIS? Many of them didn't know that the vulnerable software was even installed and running, much less that they had to patch it or they'd be screwed.

This is extremely valuable for Solaris and Oracle. For both of those, we are able to minimize the installations and defer a significant number of patch cycles simply because the vulnerable feature or package is not installed.  If the vulnerable software is not installed, we do not have to consider the vulnerability. It's even on Microsoft's radar. With server 2008, it is finally possible to install a minimized version of the operating system. I dream of the day when my IIS server will not have a GUI browser, and I'll be able to ignore vulnerabilities and patches that infect the pathetically insecure userland software that infests my servers.

So a vendor (Sun) offers to help out with a proof of concept. They delegate the actual install to a VAR. The consultant paid by the VAR (or Sun) shows up and starts to build an 'Entire Installation' server. We insist that 'Entire Installation', which includes software that we will never, ever use on that server, is not appropriate and does not meet our standards. We declare to the consultant that what we need is 'Reduced Networking Core System Support'. The vendor (Sun) provides and supports minimized installation options for the software (Solaris) and we expect the consultant to perform a minimal installation plus the specific packages necessary for supporting the intended application. What's so hard about figuring out a dependency tree and installing the minimum software necessary to resolve the dependencies? The consultant balked.

In this case, fatigued from having to deal with clueless consultants, we said screw it. We'll end up running the proof of concept with an 'Entire Installation', throwing it away and doing the minimal installation later when & if it moves to production. It shouldn't have to be that way though. It's 2009 and I expect consultants to think and act like it's the 21st century.

Why are all my 'vendor' posts also tagged as 'annoying'?

Wednesday, April 22, 2009

SMS as Two Factor Authentication Spoofable?

Speculation as to why there is an apparent rush to buy a certain model of an old Nokia cell phone.
'The 1100 can apparently be reprogrammed to use someone else's phone number, which would also let the device receive text messages. That capability opens up an opportunity for online banking fraud."
If true, then an SMS to a cell phone would no longer reliable for two factor authentication. Impersonate a persons phone long enough to log into their bank account? That'd be amusing.

Firewall Complexity Versus Time

As a follow up to Firewall Rule (Mis)Management, I created a few simple charts showing the growth of a firewall config over time. The Y-axis is simply the size of the config in bytes. The X-axis represents time from 2002 to present.


This particular firewall is at a data center that is being phased out, so as applications get deprovisioned or moved, the configuration size shrinks.

The second chart is for a firewall for another data center that was spun up in 2005. The X-axis is time from 2005 through today. The steep change in size at the left is the initial provisioning of the apps in the new data center.


The configuration size has grown continuously since 2005. I’m expecting that it will continue to grow as more apps get hosted. There are not too many scenarios were a configuration would shrink unless major applications were phased out or the firewall manger decided to simplify the problem with a few ‘permit any any’ rules.

At some point in time it’ll be too large to mange if it isn’t already. Presumably the probability of human error increases with complexity (configuration size), so one might suppose that if firewall configuration errors cause a decrease security, then the data center becomes less secure over time.

Entropy is the enemy of security.

Friday, April 17, 2009

Breaker One-Nine This Here's a Tweet!

"Ah, breaker one-nine, this here's the Oprah. You gotta copy on me, Shaq-Man, c'mon? ... Yeah, that's a big 10-4 there,  Shaq."

What's your '20 good buddy?

It's a fad, no doubt.

What's next?  Lot Lizards?

Over and out.

Tuesday, April 14, 2009

Anonymity is not Privacy

Let's pretend that I want to do something subversive like participate in a social network with persons who don't like the current form of government, or maybe I want to be really subversive and prevent advertisers from tracking my social network habits. In either case, preventing someone from de-anonymizing me is essential. The University of Texas paper linked here demonstrates that by using anonymous data from from multiple social networks it is possible to de-anonymize individuals and reliably identify them based on a comparison of the links between the persons in the social networks.

This is relevant in daily online life. If for example, my personal online presence could jeopardize my professional standing (a firearms advocate in a liberal University comes to mind), I would need to maintain separation of personal and professional online presence. Individuals who need to fear loss of freedom based on their personal social network affiliation would be another example. This research indicates that in order to maintain privacy in a social network, one would need to maintain complete separation between the questionable social network and any other social network.

The research is very interesting. If I need or desire to maintain separate networks, and I decide to use a network like Google for professional presence, then I can't allow Google or any Google related network to be associated with my personal presence. For personal presence I need to pick social networks that not only don't cooperate with Google, but actually fear or despise Google. (Microsoft, for example). This only works if I never share the same relationships across the different social networks. If I do, the digraphs intersect and identification becomes possible. Maintaining separation would mean that I would have no overlapping 'friends' across personal and professional networks. The separation also fails if the corporations that own my private and personal networks merge, get sold to another data empire, use the same advertising empires or succumb to the agents of rouge governments.

The research confirms what I've always suspected, that if I desire separation between personal and professional online presence, I am restricted my ability to participate in social networks. The restriction may be for no other reason than before I can participate, I have to decide if a particular network will be professional or personal. No network can be allowed to be both personal and professional, simply because someone, somewhere will be able to use the hybrid network to link the rest of them together. Once I decide to participate in a network I can't switch the network from personal to professional or vice versa. Even though it is theoretically possible to maintain dual identities (and dual news readers, browsers and cookie managers), it’s tough to do, and I suspect that maintaining that level of separation is doomed to failure.
Fortunately for me, my personal life is conducted largely in real time, face to face. Internet based social networks add very little value to my personal life, so I don't have any significant online personal network relationships. I figure that the worlds largest data vacuum can hoover up all my professional related online data without too much concern simply because as an employee of a public entity, my professional life open to the world anyway. But that decision prevents me from using really cool things like Goog411 or Google Voice for my personal life.

I would assume that the conclusions are generic across any network of nodes and links. Presumably e-mail address books are similar in that a person looking at the connections between people based on e-mail exchanges or address books could identify individuals using only anonymized data, and I suspect that anonymized RSS subscription data could be used in a similar manner.

The following are quotes from the paper De-anonymizing Social Networks by Arvind Narayanan and Vitaly Shmatikov, The University of Texas at Austin
The main lesson of this paper is that anonymity is not sufficient for privacy when dealing with social networks. We developed a generic re-identification algorithm and showed that it can successfully de-anonymize several thousand users in the anonymous graph of a popular microblogging service (Twitter), using a completely different social network (Flickr) as the source of auxiliary information…
…We demonstrated feasibility of successful re-identification based solely on the network topology and assuming that the target graph is completely anonymized. In reality, anonymized graphs are usually released with at least some attributes in their nodes and edges, making de-anonymization even easier…
…We do not believe that there exists a technical solution to the problem of anonymity in social networks. Specifically, we do not believe that any graph transformation can (a) satisfy a robust definition of privacy, (b) withstand de-anonymization attacks described in our paper, and (c) preserve the utility of the graph for common data-mining and advertising purposes. Therefore, we advocate non-technical solutions…
…First, the false dichotomy between personally identifiable and non-personally identifiable information should disappear from privacy policies, laws, etc. Any aspect of an individual's online personality can be used for de-anonymization, and this reality should be recognized by the relevant legislation and corporate privacy policies...
…Second, social-network operators should stop relying on anonymization as the "get out of jail" card insofar as user privacy is concerned. They should inform users when their information is disclosed to third parties, even if this information has been anonymized, and give them an opportunity to opt out…


Thursday, April 9, 2009

QWERTY is Mainstream?

Touch screens and QWERTY keyboards on mobile devices are finally mainstream just about the time when they really should be obsolete. That’s too bad, really. Voice control should be mainstream, not qwerty keypads. I should be be able to have basic mobile phone control and mobile communications without reaching for my phone and poking around on it.

It’s 2009. For mobile devices, touches, swipes, swirls, stabs and keypads are archaic. Phones should be heard, not seen.

How close are we?

Microsoft Voice Command is a partial solution. It allows basic phone control with speaker independent voice. It works with Bluetooth so I can tap the headset and say things like ‘call Jake Botts at home’ or ‘dial 612 555 1212’ and it generally figures out what I want. It also allows voice access to the calendar (‘What’s my schedule’), and can navigate menus and contacts (‘show Jake Botts’ or ‘start solitaire’ or ‘start google maps’). Additionally, it reads incoming SMS’s & messages and announces incoming phone calls by name or number. It’s not comprehensive though. Once the command executes, Voice Command drops out of the picture. So I’m still stuck with viewing and touching the phone.

Motorola (and probably others) have similar features. Motorola’s speaker independent voice dialing was great at recognizing my voice with a bluetooth headset at 80mph in a convertible. But then it would sometimes get confused in a quiet room. Go figure.

Microsoft Live Search does fairly decent voice recognition on a limited set of tasks. It suffers from a few flaws. It doesn’t use bluetooth and it doesn’t speak back to me. It also fails the ‘context’ test. I tell it ‘traffic’ and instead of showing me a traffic map, it searches for driving schools. And in most cases, it’s still touch dependent.

I know about Jott and Nuance, but haven’t tried either one, and my carrier appears to offer an add-on service that has some interesting features. Vlingo has something that looks close, but it is still touch dependent. My vision is to be able to do what Vlingo can do without taking my phone from my pocket. Adando tries to solve the problem  using your home computer as the smart part of the equation. Your phone bridges your voice back to your home PC. You PC does all the work. Clever, I guess.

I think Apple send us down the wrong path by perfecting the touch-swipe-stab-pinch-flick interface. I can’t do any of those while driving in a convertible at 80 mph. ( Well…I can, but I really shouldn’t… )

Its about time we re-think phone interfaces, isn’t it?

Tuesday, April 7, 2009

Vendors That Don’t Support VM’s?

From a large software vendor:

VMWare support: <vendor> applications running on VMware are supported as long as the OS running in the virtual machine is certified by <vendor> (see <vendor> Platform Availability Matrix).

That’s good.

However, if <vendor> is unable to reproduce an issue the customer may be asked to reproduce the problem running natively on the certified OS, without the use of VMware. <vendor> does not explicitly test or certify products running in a VMware environment.

That’s not good.

We’ve run into this a handful of times. In this case, I’m not sure how to interpret the nuanced difference between support and certification, but it’s pretty clear that <vendor> wants to leave open the option of blaming the vm in cases where they can’t figure out what’s broke. It’s the old ‘I have no clue what’s happening, so I’ll blame the network/firewall’ problem. In theory, one would have to maintain a non-vm'd test environment to use in the event that the vendor blames your vm'd environment for their problems.

We’ve also run into an application from a major network vendor that ran fine in a vm until we applied a minor patch (something like from version 3.4.1 to 3.4.2). After the patch, the app quit working. The vendor told us ‘no vm’s’. We ended up installing a dedicated server.

Somewhere along the line software vendors are going to have to provide the same level of support for virtualized environments as they do for non-virtualized environments.

This virtualization stuff is mainstream, isn’t it?

Saturday, April 4, 2009

The Cloud - A New Provider Failure Mode

swat-jpg I certainly would not have thought of this failure mode. A law enforcement agency raids a datacenter and grabs the hardware that your provider uses to host your business critical application.

The FBI has seized all equipment belonging to our customers. Many customers went to the data center to try and retrieve their equipment, but were threatened with arrest.

Let’s assume that some customer in a cloud or co-lo is doing something bad. The law enforcement agency doesn’t understand clouds, virtualization VMotion, or hypervisors. They understand computers. Which one is it? Who knows. Grab them all.

I’m not clear on the details of what happened in this particular event. It’s possible that the provider was a legitimate target of law enforcement. From the point of view of someone who has a critical application tied up in the mess, the details are not as important as the concept. If someone, somewhere in the same co-lo or cloud as you commits some crime, what will law enforcement do, and how will you protect yourself against heavy ham handed agents and judges that don’t understand technology?

I’m thinking that we need to add a couple more questions to the cloud/co-lo RFP & vendor qualification forms:

  1. Does your security staff include former law enforcement, and if so, do they still socialize with current law enforcement agents?
  2. How many full time lawyers do you employ?

Question #1 gets you fair warning of the impending raid, and #2 gets your servers back after the raid.

In the mean time, you presumably have recovered your operations at some other co-lo, in some other cloud.

Wednesday, April 1, 2009

A New Model for Product Purchasing

Most organizations follow an acquisition model similar to:
  • Identify Business Need
  • Gather Requirements
  • Evaluate Products
  • Select Product
  • Purchase Product
  • Implement Product
This model has generally proven to be costly and unreliable. Significant resources are spent identifying needs and requirements. Working with users to determine requirements is generally unreliable and often results in recursive need/requirement loops. Product evaluation and selection is time consuming and expensive, often requiring RFP’s, live demos and test installations. The end product rarely meets the users needs or requirements, simply because the unreliability of the user results in documented needs and requirements that do not represent actual user requirements.
The above process may be significantly streamlined by simply eliminating the unnecessary steps and reordering the remaining. The new model looks like this:
  • Purchase Product
  • Implement Product
  • Gather Requirements
  • Identify Business Need
This minor modification of the standard model has the following benefits.
  • Formal product evaluation by highly paid technical staff is eliminated. Products are instead evaluated by VP’s and executives at vendor provided lunches, social gatherings, sporting events, etc. Product selection is likewise eliminated, as it derives directly from product purchase.
  • Product requirements can be extracted from vendor documentation and product brochures more efficiently than developing requirements from scratch. Likewise, when an RFP or similar formal purchasing process is required, the process can be significantly streamlined by extracting requirements from vendor product brochures. Care must be taken to avoid the ‘ransom note look’ when copying and pasting vendor materials. Pasting as plain text into a preformatted document is preferred.
  • Business needs are easily derived from post implementation product walk through and demonstrations. Business users will realize their needs as they explore the product user interface and features during product training. An entire layer of analysis is thereby eliminated.
  • Entire teams of business analysts and architects are redundant in this model. The vendor’s analysts and architects have already provided much of the functionality that these roles normally perform. Substantial cost savings will result from the elimination of these roles.
  • Project risk is minimized. When needs and requirements are derived from the documentation of an already purchased product, there are no unmet needs or requirement gaps except in cases where the vendor product documentation is inaccurate. Change orders can be executed to correct the mismatch. Project success is virtually assured.
In the particular case of highly technical products where customers are unlikely to realize that they have an unmet need (security products, for example) this method assures that customer needs are met even when customer awareness of needs is insufficient to result in customer initiated product evaluation and purchase.

A parallel model may be used for software development. In the software development derivative of this model, developers perform requirements gathering as they write the software. As in the product purchase model, business needs are derived from the functionality of the completed software. Individual developers assume the roles of business analyst and architect. Their familiarity with the functionality of the software assures that the software functions as requirements dictate, as all requirements are derived from existing software functionality.

The matching of functionality to requirements is simplified by careful construction of module and function names. This will allow use of regular expression matching and word substitution to automatically build requirements from source code. For example, for a source file containing the following pseudo code[1]:
function store_customer_records 
This simple command:
cat <source file> </ source file>| sed -n \
'/^function/s/\(function\)\(.*\)/The product will \2\./p' | tr '_' ' ' 

Will extract the following requirement:

The product will store customer records.
The post-processing of requirements can easily be managed as a part of normal software deployment. As new versions of the software are deployed, the updated requirements can be automatically extracted and e-mailed to project managers by simple post-processing scripts attached to commonly available automated deployment systems.

A significant fraction of all product purchase and software development already follows derivatives of this model. Additional refinement and wider implementation of this model should result in improvements in efficiency, productivity and user satisfaction.

UPDATE 04/09/2009 After testing the new model on a recent large software purchase, it’s clear that I’ve obviously left out a critical step in the proposed model. An additional step should be added to the tail end of the process:
  • Analyze and Document Security Requirements
The complete model should be:
  • Purchase Product
  • Implement Product
  • Gather Requirements
  • Identify Business Need
  • Analyze and Document Security Requirements
Adding security requirements to the end of the document does not significantly alter the procurement process. The requirements are derived from existing functionality, making the process largely pro forma.

[1]Note: Regex matches on object oriented code should extract public methods only. The publishing of private methods is a security violation.