Skip to main content


Showing posts from 2010

It is a Platform or a Religion?

Blog posts like this annoy me. "Anyone who was ever fool enough to believe that Microsoft software was good enough to be used for a mission-critical operation..."I’m annoyed enough to keep that link in my ‘ToBlog’ notebook for over a year. That’s annoyed, ‘eh? Apparently the system failed and the blogger decided that all failed systems that happen to be running on Windows fail because they run on windows. A word from the blind. I've been known as 'anti-Microsoft', having had a strong preference for Netware and Solaris on the server side and OS/2 & Solaris on desktops. At home I went for half to a decade without an MS product anywhere in the house. Solaris on SunRays with Star Office made for great low energy, low maintenance home desktops that ran forever. My anti-Microsoft attitude changed a bit with NT4 SP3, which even though it had a badly crippled UI, was robust enough to replace my OS/2 desktop at work. My real work still got done on Solaris though. …

ToBlog Dump – Time to Clean House

Geeze – Even after periodic culling, I still have twenty+ notes in Google Notebook, fifty-odd notes in Ubernote, and a whole bunch of Google Reader starred items, all waiting to be turned into blog posts. Ain’t gonna happen. Time to clean house. I’ll dump the most interesting ones into a few posts & cull the rest.Obviously tracking this sort of thing would be better served by a bookmarking service, but I’ve decided that my professional Internet presence will be Google and Google related apps. I use a combination of Yahoo & for things that I don’t want associated with my professional presence, and I try hard not to mix them. The only interesting bookmark service is a Yahoo property (for now, at least) so I don’t have a public bookmarking service. Lame? Yes. I don’t have Twitter or Facebook accounts either. Really lame. Maybe even lame2. I still would rather read blogs posts than tweets. Is that lame3 ?Disclaimer – most of these links are more than a year old, but they’v…

Thomas Limoncelli: Ten Software Vendor Do’s and Don’ts

From a panel discussion at a recent CHIMIT (Computer-Human Interaction for Management of Information Technology), summarized and published at the Association for Computing Machinery. A good read, right through the comments.Thomas covers non-GUI, scripted and unattended installation, administrative interfaces, API’s, config files, monitoring, data restoration, logging, vulnerability notification, disk management, and documentation. The comments cover more. Comments on the above:API’s: In our latest RFP’s, we ask ‘What percentage of your application functionality is exposed via API’s?’ These can RFP’s can have an $8-digit tail on them, so odds are that they actually read them. I like sending messages to vendors.Installation layout, location: I really like non-OS software to be completely contained in something like /opt/<application>. I don’t like third party software mucking up /etc, /var, or /usr. When I’m done with the software, I want to be able to pkgrm and rm –rf and have a …

When the weather map look like this….

Odds are the traffic map will look something like this:
I’m sure there is a parallel between the DOS attacks that mother nature periodically foists us and internet security. I’ll take a stab at describing the parallels. Predictability: Snow storms and hurricanes are very predictable (compared to tornadoes, where one has 0-10 minutes warning and rarely has accurate predictions). It is possible to prepare for weather that can be predicted. In certain regions, snow storms or hurricanes are a high enough probability event that you will certainly experience them. The probability of a major snow storm  hitting my house in a particular winter is close enough to ‘one’ that it might as well be ‘one’. Tornadoes, on the other hand, even though there are dozens per year in my region, are localized enough that I probably will never experience a direct hit on my house.I might tend to be prepared for a predictable event (snow storm), but rest assured that I have not taken any significant precautions …

The flaw has prompted the company to consider changes in its development process

The recent WSJ article on banks releasing mobile banking software that stores user names, passwords and bank accounts unencrypted on phones has opened up a sore topic for me.Apparently we have very, very large corporations chocked full of highly paid analysts, architects, developers and QA staff believing that it is perfectly OK to store banking credentials in plain text on a mobile device a decade into the 21st century. Something is broke. Possibilities include:
the bank's analysts, architects, developers and QA staff are unaware of the state of application security in the 21st century. They have no idea that a fraction of the worlds population enjoys compromising other peoples systems and they use the information to steal peoples money.  In other words  - they are unconscious of the environment to which they are deploying their application. They are sufficiently unconscious of their environment that they didn't know that there may be some sort of best practice on the storage…

Application Security Challenges

Assume that all your application security challenges are conquered. You've got smart people and you've trained them well. They catch all their exceptions, they bound their arrays, they else their if's and sanitize their inputs and outputs.Congratulations. You've solved your biggest security problem.Maybe.How about your crufty old apps?
You’ve got de-provisioning down pat, right? No old apps laying around waiting to be exploited? Nobody would ever use the wayback machine to find out where your app used to be, would they?
An associate of mine did the forensics on one like that. Yes, you can upload a Unix rootkit to a blob in a SQL server, execute it directly from the database process and target a nearby Unix server. Heck – you can even load a proxy on the SQL server, poison the Unix servers ARP cache, and proxy all it’s traffic. No need to root it. Just  proxy it. Network segmentation anyone?
Any old libraries laying around?
Yech. I have no clue how someone who downloads…

Log Reliability & Automotive Data Recorders

When are logs reliable? Toyota's official answer seems to be either It depends or "The data retrieved from the EDR is far from reliable", unless the data exonerates them, in which case "the EDR information obtained in those specific incidents is accurate".There’s got to be a blog post somewhere in that. Accuracy:Did the log record what actually happened. Did the log record when something actually happened? Do the logs represent the events in the order that they occurred? Are the time stamps accurate? Time syncing all you systems is fundamental, obvious and a best practice for the last fifteen years or so, but unless you log time sync failures, you don't necessarily know if the time stamp on a logs is accurate. I like syslog capable systems that time stamp the logs at the source and syslog servers that time stamp them again as they are caught and written. That helps verify the accuracy of time stamps.

Completeness:Are there gaps in the logs? If so, can we d…

I’m not sure what it is, but I’m sure it’s rootable

I have no clue why anyone would still run RealPlayer. I’ve pretty much forgot that it existed.  But I know that those who know what it is and still run it are screwed. If they even know they are running it. They probably don’t. That makes them extra screwed. If you accidently configure RDS in your Linux kernel, you’ve got something to fix.  From what I can see, we can blamethank Oracle for RootableReliable Datagram Sockets. You’d think that be now we’d be able to introduce something new and interesting without making the old & stable rootable.I guess not.If obscure media players and Infiniband protocols are rootable, the most popular OS in the world must be rootable, right? Yep, it’s rootable. Again. Damn. It’s probably also running Java, which makes it double-extra rootable. Speaking of Java, Microsoft thinks that there is an unprecedented wave of Java exploitations. I wonder who wrote the operating system that allows itself to be exploited by such an unprecedented wave. Waves ar…

DNS RPZ - I like the idea

An opt-in real time black hole list for untrustworthy domain names?


Some thoughts:

I certainly don't think that offering the capability is a bad thing. Nobody is forced to use it.

Individual operators can decide what capability to enable and which blacklists to enable. ISP's could offer their customers resolvers with reputation filters and resolvers without.  ISP's can offer blacklisted/greylisted resolvers for their 'family safe' offerings. Corporations/enterprises can decide for themselves what they blacklist.

A reputation based white list would be interesting. Reputation could be determined by the registrar, perhaps based on the registrar having a valid, verified street address, phone and e-mail for the domain owner. A domain that has the above and has been registered for a month or so could be part of a white list. A domain that hasn't met the above could be gray listed. Operators could direct those to an internal 'caution' web pa…

Are we creating more vulnerabilities than we are fixing?

Thoughts on Application Logging

As a follow on to:How to Do Application Logging Right by Anton Chuvakin and Gunnar PetersonApplication Security Logging by Colin Watson and the Common Event Expression (CEE) Architecture Overview [PDF],I have a few semi-random thoughts on application logging.Things I like to see in logs are:Machine parseable (yet human readable) format. I need to be able to write a regex that cleanly separates interesting messages and pipe them into sed/awk and extract critical fields from the messages. I typically use sed/awk/perl to strip out uninteresting parts of the message and sort/count pipe-to-Excel the rest of the fields. I also use logsurfer to catch real time events and alert interested parties. Even organizations with sophisticated tools still need to be able to parse the logs. Bonus points if all messages of a particular type have the same number of fields - or if variable word fields are at the end of the message. Single line events. No XML. I'm not going to write a custom multi-lin…

ZFS and NFSv4 ACL’s

I've been doing granular file access control lists since Netware 2.0. I'm used to being able to specify (for example) permissions such that a file can be modified, but not renamed or deleted, or setting permissions on a file so that it can be executed, but not read - (Yes, Netware could do that). And of course, it's obvious that more than one user or group permission should be allowable. I'm also used to having some control over inheritance, so that I can 'kneecap' permissions on a nested directory.Obviously I've been very unimpressed with Unix's trivial rwxr-x--- style permissions. Sun band-aided the decades old rwxr-x--- up with POSIX getfacl and setfacl. That was a start. We now have NFSv4 style ACL’s on ZFS. It looks like they are almost usable.For an experiment, I decided to clean up a few 'home directories' where the existing permissions are a mess of randomness left over from a decade of ufsdump/ufsrestore, POSIX ACL's, tar, cpio, pax…

Engineering by Roomba’ing Around

A simple random walk algorithm:
Start out systematically Hit an obstacle Change direction Hit another obstacle Change direction Eventually cover the problem space. As applied to the problem of cleaning a floor, the algorithm seems to work OK, particularly if you are willing to ignore the parts of the problem space that the device cannot solve (corners, low furniture, complex spaces).I sometimes see similar algorithms used by IT engineers. They start out systematically, hit an obstacle, head off in a random direction, hit an obstacle, head off in a different direction, and (usually) solve the problem (eventually). Unfortunately many IT engineers troubleshoot this way. It could be worse – Some engineers start out systematically, hit an obstacle, and instead of changing direction, they just keep on banging into the obstacle. They haven’t figured out that even a random direction change is better than no direction change.I also see IT engineers ignore the problems that their tool or proj…

Bogus Drivers Licenses, Fake Passports

The State of Minnesota is running a facial recognition algorithm on Minnesota drivers licenses and state ID’s.Partial results:Ran the algorithm on 11 million license photosFlagged 1 million for manual reviewOf the 100,000 reviewed so far, 1200 licenses were cancelledBy simple extrapolation of the numbers, there could be as many as 10,000 bogus state issued ID’s or licenses out of the pool of 11 million. There isn’t enough data in the media to know if a simple extrapolation is valid, so the number could be less.Meanwhile, Government Accounting Office investigators were able to obtain US passports with fake identification in three out of seven attempts.I think there is a house of cards here somewhere.

Just another day in Internet-land

So I’m goofing off at work, gambling with other peoples money using my fully patched but rootable browser, running on a fully patched but rootable operating system, occasionally downloading digitally signed malware while I contemplate the possibility that my medical records are on a P2P network somewhere, knowing that I really should be patching the remotely exploitable database that I just installed on my shiny new sever that was thoughtfully preloaded with malware, and I’m thinking to myself: “What’s new and interesting today?”Nothing. Just another day in Internet-land.

Oracle Continues to Write Defective Software, Customers Continue to Buy it

What’s worse:Oracle continues to write and ship pathetically insecure software. Or:Customers continue to pay for it. From the July 2010 Oracle CPU pre release announcement:Oracle ProductVulnerabilityRatingLicense Cost/ServerDatabase ServerRemote, No Auth[1]7.8/10$167,000[2]Awesome. For a mere $167,000[2] I get the privilege of installing poorly written, remotely exploitable, defective database software on a $5,000 2-socket Intel server.Impressive, isn’t it. I’m not sure what a ‘Times-Ten’ server is – but I’m glad we don’t have it installed. The good news is that it’s only half the price of an Enterprise Edition install. The bad news is that it is trivially exploitable (score of 10 on a scale of 1-10).Oracle ProductVulnerabilityRatingLicense Cost/ServerTimes-Ten ServerRemote, No Auth[1]10/10$83,000[3]From what I can see from the July 2010 pre-release announcement, their entire product catalog is probably defective. Fortunately I only need to be interested in the products that we have i…

Let’s Mix Critical Security Patches and Major Architecture Changes and see What Happens.

Is re-architecting key functionality on an N.n.n release unusual?“Yes, this was an unusual release, and an experiment in shipping new features quicker than our major release cycle normally allows.”On version 3.6.n, plugins shared process space. On 3.6.n+1, plugins do not.The experiment appears to have suffered a setback.The problem? “…we are seeing an increasing number of reports that some users are unable to play Farmville, because Farmville hangs the browser long enough for out timeout to trigger and kill it.”Apparently the “crashed plugin” timer needs to be long enough that Farmville can finish loading. Ten seconds isn’t long enough.How did they originally arrive at a 10 second timeout? “Originally a 10s timeout made a lot of sense considering that we had no actual data to go with.”It looks like none of the Mozilla developers or testers play Farmville, or they’d have caught the problem prior to release.Why make major changes to a minor release? To improve the customer experience, o…

Sun/Oracle Finally Announces ZFS Data Loss Bug

If you’ve got a Sun/Oracle support login, you can read that an "Abrupt System Reboot may Lead to ZFS Filesystem Data Integrity Issues" on all Solaris kernels up through April 2010.“Data written to a Solaris ZFS filesystem and confirmed by fsync(3C) may be lost in the event of an abrupt system reboot.”This announcement came too late for us though.If I am a customer of an ‘enterprise’ vendor with millions of dollars of that vendors hardware/software and hundreds of thousands in annual maintenance costs, I expect that vendor will proactively alert me of storage related data loss bugs. I don’t think that’s too much to expect, as vendors with which I do far less business with have done so for issues of far less consequence.Sun failed. Hopefully Oracle will change how incidents like this are managed.

Another Reason for Detailed Access Logs

Another poorly written application, another data leak. Not new, barely news. This statement is interesting though:“[company spokesperson] said it's unclear how many customers' information was viewed, but that letters were sent to 230,000 Californians out of an "abundance of caution.”Had there been sufficient logging built into the application, Anthem Blue Cross would have known the extent of the breach and (perhaps) could have avoided sending out all 230,000 breach notifications. That’s a view on logging that I’ve expressed to my co-workers many times. Logs can verify what didn’t happen as well as what did happen, and sometimes that’s exactly what you need.There are a couple of other interesting things in the story:“the confidential information was briefly accessed, primarily by attorneys seeking information for a class action lawsuit against the insurer.”That’ll probably cost Anthem a bundle. Letting lawsuit-happy attorneys discover your incompetence isn’t going to be th…

Would You Give up Your Credit Card Number for an Hour of Free Wireless?

Cool: The City of Minneapolis has city-wide WiFi.Cooler: The City of Minneapolis is offering free WiFi hotspots at selected spots in the city. Coolest: It works. Uncool: To use the free hot spots, you have to surrender a credit card number.Would you give up a CC number just to get free WiFi?This probably isn’t any worse that handing your card to a waiter at a restaurant.I don’t know what else one could request that would provide a bit of verification of a users identity. A drivers license?One could simply decide to not care who uses the free hotspots. Our big brothers at the University don’t. They offer free guest wireless with only an e-mail address. For them, is an e-mail address.It’s entirely possible that the vendor that built and owns the the network is PCI-DSS SAQ-whatever compliant. I don’t think that I’d give up a card number just to get free WiFi.I will assert though, that most people will.

What’s an Important Update?

Windows update runs (good).Windows update classifies some updates as important, and some updates as optional (good).Windows update decides that a Silverlight update is important. It appears security related (good) but also add features (maybe good, maybe bad).Windows update decides that a security definition update is optional (bad).How can a definition update for a signature based security product be optional? That’s annoying, ‘cause now I have to make sure to check optional updates just in case they’re important.

Where are your administrative interfaces, and how are they protected?

One of the many things that keeps me awake at night:For each {application|platform|database|technology} where are the administrative interfaces located, and how are they protected?I've run into administrative interface SNAFU's on both FOSS and purchased software. A common problem is applications that present an interface that allows access to application configuration and administration via the same ports and protocols as the application user interface. A good example is Twitter,  where hacker-useful support tools were exposed to the Internet with ordinary authentication.In the case of the Pennsylvania school spy cam caper, the 'administrative interface' that the school placed on the laptops apparently is relatively easy to exploit, and because they sent the students home with the district laptops, the interface is/was exploitable from the Internet. Years ago one of our applications came with a vendor provided Tomcat install configured with the Tomcat management interf…

IPv6 Tunnels & Solaris

Following Dan Anderson’s instructions I set up an IPv6 tunnel and put my home network on IPv6. It was surprisingly easy. I have an OpenSolaris server acting as the tunnel end point and IPv6 router, with IPv6 tunneled to Hurricane Electric, and didn’t spend much more than an hour doing it.Following Dan’s instructions, I:Signed up at Hurricane Electrics tunnel broker service, requested a /64 & created a tunnel Configured my OpenSolaris server as a tunnel end point Configured Solaris’s IPv6 Neighbor Discovery Protocol (NDP) service & reloaded itPointed my devices at HE’s DNS’s‘Bounced’ the wireless adapters on my various notebooks, netbooks and Mac’s I didn’t have to reboot anything – and better yet – when I did reboot the various devices, IPv6 still worked.I’m not sure why I needed to use HE’s name servers, but things started working a lot better when I did, and their name servers seem to work as good as anyone’s.I think I got lucky – my DLINK DIR-655 home router/access point ro…

Oracle/Sun ZFS Data Loss – Still Vulnerable

Last week I wrote about how we got bit by a bug and ended up with lost/corrupted Oracle archive logs and a major outage. Unfortunately, Oracle/Sun’s recommendation – to patch to MU8 – doesn’t resolve all of the ZFS data loss issues.There are two distinct bugs, one fsync() related, the other sync() related. Update 8 may fix 6791160 zfs has problems after a panic , butBug ID 6880764 “fsync on zfs is broken if writes are greater than 32kb on a hard crash and no log attached”is apparently not resolved until 142900-09 released on 2010-04-20.DBA’s pay attention: Any Solaris 10 server kernel earlier than Update 8 + 142900-09 that is running any application that synchronously writes more than 32k chunks is vulnerable to data loss on abnormal shutdown.As best as I can figure – with no access to any information from Sun other than what’s publicly available – these bugs affect synchronous writes large enough to be written directly to the pool instead of indirectly via the ZIL. After an abnormal …

We do not retest System [..] every time a new version of Java is released.

This post’s title is a quote from Oracle technical support on a ticket we opened to get help running one of their products on a current, patched JRE. Oracle’s response:“1. Please do not upgrade Java if you do not have to
2. If you have to upgrade Java, please test this on your test server before implemeting [sic] on production
3. On test and on production, please make a full backup of your environment (files and database) before upgrading Java and make sure you can roll back if any issue occurs.”In other words – you are on your own. The hundreds of thousands of dollars in licensing fees and maintenance that you pay us don’t do you sh!t for security.Let’s pretend that we have a simple, clear and unambiguous standard: ‘There will be no unpatched Java runtime on any server’.There isn’t a chance in hell that standard can be met. This seems to be a cross vendor problem. IBM’s remote server management requires a JRE on the system that has the application that connects to the …

Bit by a Bug – Data loss Running Oracle on ZFS on Solaris 10, pre 142900-09 (was: pre Update 8)

We recently hit a major ZFS bug, causing the worst system outage of my 20 year IT career. The root cause:Synchronous writes on ZFS file systems prior to Solaris 10 Update 8 are not properly committed to stable media prior to returning from fsync() call, as required by POSIX and expected by Oracle archive log writing processes.On pre Update 8 MU8 + 142900-09[1], we believe that a programs utilizing fsync() or O_DSYNC writes to disk are displaying buffered-write-like behavior rather than un-buffered synchronous writes behavior. Additionally, when there is a disk/storage interruption on the zpool device and a subsequent system crash, we see a "rollback" of fsync() and O_DSYNC files. This should never occur, as write with fsync() or O_DSYNC are supposed to be on stable media when the kernel call returns. If there is a storage failure followed by a server crash[2], the file system is recovered to an inconsistent state. Either blocks of data that were supposedly synchronously writ…