Failed Backups – Unrecoverable Service

A small but high profile social bookmarking site recently suffered catastrophic, unrecoverable data loss. The site’s creator and owner Larry Halff posted a video blog is which he talks about the failure and lessons learned.

[Since deleted]

Highlights from the vlog: 
  • Software RAID volume or database corruption was the original cause.
  • The site was self hosted.
  • Complex dependencies made moving the site to professional hosting difficult.
  • The only backups were a copy to an attached firewire drive.
  • There were no integrity checks or test restores.
  • The site was hosted on Apple xServe’s and Mac Mini’s. 
It’s a great ‘lessons learned’ for small startups. My take is that the people who create cool things on the Internet aren’t necessarily the ones that should be hosting those cool things. Those are rather different skill sets. The corollary is probably that people who are good at hosting the cool things on the Internet are likely not capable of creating them.

The big picture? When trusting others with your data, how do you know if they are taking appropriate steps to protect the data?

You don’t.