Skip to main content


Showing posts from August, 2008

Privacy, Centralization and Security Cameras

The hosting of the Republican National Convention here in St Paul has one interesting side effect. We finally have our various security and traffic cameras linked together:
“The screens will also show feeds from security cameras controlled by the State Patrol, Minnesota Department of Transportation, and St. Paul, Minneapolis and Metro Transit police departments.
Before the RNC, there was no interface for all the agencies' cameras to be seen in one place. Local officials could continue to use the system after the RNC.” (Emphasis mine)So now we have state and local traffic cameras, transit cameras and various police cameras all interconnected and viewable from a central place. This alone is inconsequential. When however, a minor thing like this is repeated many times across a broad range of places and technologies and over a long period of time, the sum of the actions are significant. In this case, what’s needed to turn this into something signifi…

Scaling Online Learning - 14 Million Pages Per Day

Some notes on scaling a large online learning application.

09/29-2008 - Updated to correct minor grammatical errors.
Stats:29 million hits per day, 700/second 14 million .NET active content pages per day[1]900 transactions per second 2000 database queries per second 20 million user created content files Daily user population of over 100,000 Database server with 16 dual core x64 CPU's, 128GB RAM, Clustered Nine IIS application servers, load balanced The largest installation of the vendors product Breadth and complexity. The application is similar to a comprehensive ERP application, with a couple thousand stored procedures and thousands of unique pages of active web content covering a full suite of online learning applications, including content creation and delivery, discussions, quizzing, etc. The application has both breadth and depth, and is approximately as complex as a typical ERP application. This makes tuning interesting. If a quarter million dollar query pops up, it can be t…


Two acronyms worth remembering.

RGE:RGE: (Resume Generating Event) – An event that forces a person, or the persons manger to generate an updated resume. An  RGE is something most of us don't want to experience, at least not too often. RGEs are often followed by changes in income, housing, marital status, etc.

HGE:HGE: (Headline Generating Event) – An event that causes news reporters to write stories and generate headlines. HGEs can be either positive or negative, depending on the causes and effects of the event, although with the exception of dot-com startups, most IT initiated HGEs are negative events related to system or project failures of some sort.

HGEs are often followed by RGEs.

Obviously a goal of system mangers, security people and IT folks in general is to make sure that that acronyms like the above don’t show up unexpectedly. Those of us in public service are particularly sensitive to HGEs. There are not too many circumstances where public service IT organizations can ge…

Design your Failure Modes

In his axiom 'Everything will ultimately fail', Michael Nygard writes that in IT systems, one must:
"Accept that, no matter what, your system will have a variety of failure modes. Deny that inevitability, and you lose your power to control and contain them. [....]  If you do not design your failure modes, then you will get whatever unpredictable---and usually dangerous---ones happen to emerge."I'm pretty sure that I've seen a whole bunch of systems and applications where that sort of thinking isn't on the top of the software architects or developers stack. For example, I've seen:
apps that spew out spurious error messages to critical logs files at a rate that makes 'tail -f' useless. Do the errors mean anything? Nope - just some unhandled exceptions. Could the app be written to handle the exceptions? Yeh, but we have deadlines......apps that log critical application server/database connectivity error messages back to the database that caused the…

Using the DNS Question to Carry Randomness - a Temporary Hack?

I read Mark Rothman's post Boiling the DNS Ocean. This lead me to a thought (just a thought), that somewhere within the existing DNS protocol, there has to be a way of introducing more randomness in the  DNS question and get that randomness back in the answer, simply to increase the probability that a resolver can trust an authoritative response. Of course having never written a resolver, I'm not qualified to analyze the problem -- but this being the blogosphere, that's not a reason to quit posting.

So at the risk of committing bloggo-suicide....Here goes......

Plan 'A' was to figure out if the unused bits in the ancount, nscount or ascount of the question could be used to send more random bits to the authoritative DNS. The ADNS would have to return those bits somewhere, and not in the same fields, because in the answer those bits are already used. Perhaps in an additional RR? It sounds hard to implement, and it would require that the ADNS and resolver both b…

Patch Now - What Does it Mean?

When security researchers/bloggers announce to the world 'patch now', are they are implying that the world should 'patch now without consideration for testing, QA, performance or availability'? Or are they advising an accelerated patch schedule, but in a change managed, tested, QA’d rollout of a patch that considers security and availability? And when they complain about others not patching fast enough, are they assuming that the foot draggers are incompetent? Or are they ignoring the operational realities of making untested changes to critical infrastructure?

Consider that:
All patches have a probability of introducing new bugs. That probability is always > 0  and <= 1. The probability is never equal to zero. (And for a certain large database vendor, our experience is that the probability of introducing new bugs is very close to one).There are many, many bugs that are only relevant under high loads. A patch that corrupts data, as in databases or file system…

The Cat is Out of the Bag - Defcon or not

Apparently Massachusetts Transit thinks that it is actually possible to retract information from the public once it leaks.

They've convinced a judge to issue a restraining order to prevent MIT students from presenting at Defcon. But they missed the fact that the presentation is already downloadable from public sources (MIT's student newspaper), is available on the Defcon CD, and then a "copy of the entire presentation was entered as evidence in publicly available court records".


The source code for the nifty toolkit might be harder to obtain. But because is was once public, odds are that someone grabbed a copy.

In any case, the physical security at MBTA was apparently weak enough that many opportunities for mischief were available without going through the trouble of hacking the cards.

Safe browsing - Websense says fuggetaboutit!

It would sure be nice if an ordinary mortal could buy a computer, plug it in, and safely surf the web. Websense doesn't think so. I don't either. Apparently neither does CNN.According to Websense:
75 percent of Web sites with malicious code are legitimate sites that have been compromised [...] 60 percent of the top 100 most popular Web sites  have either hosted or been involved in malicious activity in the first half of 2008.
Ordinary precautions, like 'don't surf pr0n' , 'don't run P2P', and 'don't download screen savers' are of marginal value when legitimate web sites are part of the malware content distribution network.It's 2008. So now that we have the wonderful world of Web 2.0, Websense says:
The danger is that users typically associate the content they are viewing from the URL in the address bar, not the actual content source. The URL is no longer an accurate representation of the source content from the Web page. (Emphasis is mine.…

A Down Side to Open Wireless

An interesting byproduct of maintaining an open wireless network? Apparently a man in Mumbai had his life disrupted by a raid on his property when authorities suspected his open wireless as being the source for messages relating to a recent bombing.

So what happens when a crime is alleged to have been committed using your open network?

For individuals who happen to get caught up in a crime involving something like what happened in Mumbai, and who don't have the backing of corporate legal departments, I suspect that the process wouldn't be much fun.

A person has open wireless because:
They don't know it is open.
They know it is open but don't know how to secure it. They know how, but are to lazy to bother securing it. They know how to secure it, but they don't really care.
They know how, and they are leaving it unsecured on principle.If someone gets caught up in something like this, I hope they are in one of the latter categories, rather than in one of the first coup…

The crud moved up the stack

A long time ago (in internet time), in a galaxy really nearby, a few large software companies attached their buggy, unsecured operating systems to the Internet.

Havoc ensued.

The overall quality of software, as measure by MTTB (Mean Time to Badness), or 'if I connect to the internet, how long 'till bad things happen' was pathetic. Cow tipping was an amusing past time, Land attacks, Ping of Death attacks, Smurf attacks, were a daily event. Toss a few malformed packets into a campus, watch the campus roll over and die. Build a worm that can hack a zillion web servers in a week, or sling out a UDP packet that can turn hundreds of thousands of database servers into corporate network wrecking zombies, affecting even the companies that wrote the software. Sysadmins made the problem worse by connecting any old crap to the internet without the slightest thought for securing it.

It was chaos.

But software vendors recognized the problem, or at least most of them did, and implemented ne…