Saturday, July 10, 2010

Oracle Continues to Write Defective Software, Customers Continue to Buy it

What’s worse:

  • Oracle continues to write and ship pathetically insecure software.

Or:

  • Customers continue to pay for it.

From the July 2010 Oracle CPU pre release announcement:

Oracle Product Vulnerability Rating License Cost/Server
Database Server Remote, No Auth[1] 7.8/10 $167,000[2]

Awesome. For a mere $167,000[2] I get the privilege of installing poorly written, remotely exploitable, defective database software on a $5,000 2-socket Intel server.

Impressive, isn’t it.

I’m not sure what a ‘Times-Ten’ server is – but I’m glad we don’t have it installed. The good news is that it’s only half the price of an Enterprise Edition install. The bad news is that it is trivially exploitable (score of 10 on a scale of 1-10).

Oracle Product Vulnerability Rating License Cost/Server
Times-Ten Server Remote, No Auth[1] 10/10 $83,000[3]

From what I can see from the July 2010 pre-release announcement, their entire product catalog is probably defective. Fortunately I only need to be interested in the products that we have installed and have an Oracle’s CVSS of 6 or greater & are remotely exploitable (the really pathetic incompetence).

If I were to buy a Toyota for $20,000, and if anytime during the first three years the Toyota was determined to be a smoldering pile of defective sh!t, Toyota would notify me and offer to fix the defect at no cost to me other than in inconvenience of having to drive to their dealership and wait in their lobby for a couple hours while they replace the defective parts. If they didn’t offer to replace or repair the defects, various federal regulatory agencies in various countries would force them the ‘fess up to the defect and fix it at no cost to me. Oracle is doing a great job on notification. But unfortunately they are handing me the parts and telling me to crawl under the car and replace them myself.

An anecdote: I used to work in manufacturing as a machinist, making parts with tolerances as low as +/-.0005in (+/-.013mm). If the blueprint called for a diameter of 1.000” +/-.0005 and I machined the part to a diameter of 1.0006” or .9994”, the part was defective. In manufacturing, when engineers designed defective parts and/or machinists missed the tolerances and made defective parts, we called it ‘scrap’ when the part was un-fixable or ‘re-work’ if it could be repaired to meet tolerances. We wrote it up, calculated the cost of repair/re-machining and presented it to senior management. If we did it too often, we got fired. The ‘you are fired’ part happen often enough that myself, my foreman and the plant manager had a system. The foreman invited the soon to be fired employee into the break room for coffee, the plant manager sat down with the employee, handed him a cup of coffee and delivered the bad news. Meanwhile I packed up the terminated employees tools, emptied their locker and set the whole mess out in the parking lot next to their car. The employee was escorted from the break room directly to their car.

Will that happen at Oracle? Probably not.

Another anecdote: Three decades ago I was working night shift in a small machine shop. The owner was a startup in a highly competitive market, barely making the payroll. If we made junk, his customers would not pay him, he’d fail to make payroll, his house of cards would collapse and 14 people would be out of work. One night I took a perfectly good stack of parts that each had hundreds of dollars of material and labor already invested in them and instead of machining them to specification, I spent the entire shift machining them wrong & turning them into un-repairable scrap.

  • One shift’s worth of labor wasted ($10/hour at the time)
  • One shift’s worth of CNC machining time wasted ($40/hour at the time)
  • Hundreds of dollars per part of raw material and labor from prior machining operations wasted
  • Thousands of dollars wasted total (the payroll for a handful of employees for that week.)

My boss could have (or should have) fired me. I decided to send him a message that hopefully would influence him. I turned in my timecard for the night (a 10 hour shift) with ‘0’ in the hours column.

Will that happen at Oracle? Probably not.

One more anecdote: Three decades ago, the factory that I worked at sold a large, multi-million dollar order of products to a foreign government. The products were sold as ‘NEMA-<mumble-something> Explosion Proof’. I’m not sure what the exact NEMA rating was. Back in the machine shop, we just called them ‘explosion proof’.

After the products were install on the pipeline in Siberia[4], the factory sent the product out for independent testing & NEMA certification. The product failed. Doh!

Too late for the pipeline in Siberia though. The defective products were already installed. The factory (and us peons back in the machine shop) frantically figured out how to get the dang gear boxes to pass certification. The end result was that we figure out how to re-work the gear boxes in the field and get them to pass. If I remember correctly,  the remedy was to re-drill 36 existing holes on each part 1/4” deeper, tap the holes with a special bottoming tap, and use longer, higher grade bolts. To remedy the defect, we sent field service techs to Siberia and had them fix the product in place.

The factory:

  1. Sold the product as having certain security related properties (safe to use in explosive environments)
  2. Failed to demonstrate that their product met their claims
  3. Figured out how to re-manufacture the product to meet their claims
  4. Independently certified that the claims were met.
  5. Upgraded the product in the field at no cost to the customer

Oracle certainly has met conditions #1, #2 and #3 above. Will they take action #4 and #5?

Probably not.


[1]Remotely exploitable - no authentication required implies that any system that can connect to the Oracle listener can exploit the database with no credentials, no session, no login, etc. In Oracles words: “may be exploited over a network without the need for a username and password”

[2]Per Core prices: Oracle EE $47,000, Partitioning $11,500, Advanced Security $11,500, Diag Pack $5000, Tuning Pack $5000, Patch Management $3500. Core factor of 0.5, discount of 50%  == $167,000. YMMV.

[3]All other prices calculated as list price * 8 cores * .5 core factor * 50% discount.

[4]I have no idea if the pipeline was the infamous pipeline that made the headlines in the early 1980’s or not, nor do I know if it is the one that is rumored to have been blown up by the CIA by letting the Soviets steal defective software. We made gearboxes that opened and closed valves, not the software that drove them. We were told by management that these were ‘on a pipeline in Siberia’.

5 comments:

  1. I think the comparison is a bit unfair. These are not faults that make the database unable to do its job. They are not comparable to dodgy brakes.

    If Toyota found people were dropping bricks from bridges onto their windscreens and shattering them, would they offer to upgrade every windscreen to military grade blast-proof glass ?

    These errors you describe as "pathetic incompetence" mean that someone (probably inside your company since most databases aren't publically accessible) can cause the thing to crash. Frankly, they are more likely to shove a roll of paper towels down the company toilet to make it flood.

    ReplyDelete
  2. Gary -

    In this case, it looks like the vulnerability is availability related. Any client that knows the magic bits can disrupt the database for applications that provide hundreds of thousands of users (in our case) with the things that they need to do whatever it is that they do. I'd call that important, and I'd say that a software vendor that continues to supply customers with products with that scale of defect is incompetent. As a comparison, if a sysadmin built a database server w/o redundant disk drives (an obvious availability problem), I'd say that sysadmin is also incompetent.

    Unfortunately we value availability, and therefore we have to run through a patch cycle anyway, with the full test/QA cycle. It'll tie us up for most of the next couple months and delay our work on things that are far more interesting and perhaps would even result in better service to our customers, or reduced cost to the taxpayers.

    In other words, the cost to us is the same as if it were data integrity related.

    You seem to be asserting that if it's only vulnerable from inside the security perimeter, it's not such a big deal. It's not safe, in my opinion, to assume that because it's inside a firewall, it's secure. Presumably the attack surface is any desktop with direct connections and any web application with insecure web app software. I presume that web apps are generally coded insecurely, and I'm presuming of course, that no large corporation is completely bot-free. That presumption is based on working in a large EDU, where we see pretty much everything you read about in the press.

    For non-edu's see: this -- Or better yet - ask Google how many of the 'APT' compromised desktops were inside their corporate security perimeter & check out the hints that they were not the only large corp to suffer the infestation. We've detected bot'd computers on fortune 500 networks in cases where we hosted the controllers, and I know staff at large corporations that detect compromised desktops on their networks often enough that they have a routine for handling it. I've also observed that when researchers publish details on large botnets, they commonly assert that large corporations are part of the botnets.

    I'm zeroing in on Oracle (but not excluding any other top tier vendor) because Oracle is particularly costly to purchase and maintain. If I'm paying a premium, I expect better software.

    No exceptions, no excuses, no bullshit.

    ReplyDelete
  3. Yes it sucks that Oracle have bugs in their products.

    But who has not? I mean, IBM has, HP has, etc. Everyone has. And every company behaves just like Oracle does. Nothing new here. Move along, nothing to see.

    ReplyDelete
  4. From my response to Gary Meyers:

    "I'm zeroing in on Oracle (but not excluding any other top tier vendor) because Oracle is particularly costly to purchase and maintain. If I'm paying a premium, I expect better software."

    "No exceptions, no excuses, no bullshit."

    ReplyDelete
  5. Michael,

    Your absolutely right you pay a premium price for a database product and you expect some quality with the product, and it should:

    > be easy to install
    > bugs are easy to patch
    > there are no anomalies in operation
    > easy to develop and deploy
    > easy to administrate

    where does oracle stand on each of these,

    then compare it other database vendors:

    sqlserver
    sybase
    teradata
    db2
    RDB -- also an oracle product (VMS only)

    I'm sure there are more databases out there. Just rate cost to quality with each of these.

    ReplyDelete