Dabble DB

The Dabble Blog

A bad case of the Mondays

On Monday, the server that hosted our main web site, including this blog and our user forum, had a major catastrophe and died. We spent the rest of the day getting a new server up and running to take its place. Perhaps it was a sort of comeuppance as we were recently congratulating ourselves on having launched Dabble DB a year ago and having had no system-wide outages — scheduled or otherwise — in that entire time.

Unfortunately, the server that went down was also responsible for directing requests to the various servers that host customer databases, so everyone lost access to their databases for a time too. No public web site, no blog, no forum, no database access... all in all, not our finest hour.

Naturally there are lessons to be learned after such disasters. We've identified some of the weaknesses in our old system and are taking steps to make sure that when the next server fails, it will be a less calamitous affair. Our systems will be more independent so that we can better communicate with you in the event of emergency. This blog, for instance, is now hosted off-site by the fine folks at TypePad, so it should always be available for us to give status updates. (You may want to change your bookmarks to http://blog.dabbledb.com/).

I'm proud to say that in all the mayhem, we lost absolutely no data. A little hair and some sanity on the other hand... aah, who needs 'em?
—Luke

6 comments

  1. Dave Myers says:

    Thanks for blogging about this. Can you comment on what steps are taken to ensure the security of user data?

  2. Avi says:

    Dave,

    All user data is backed up every two hours to two locations, one within our data center and one offsite.

  3. David Ulevitch says:

    And the more important question,

    How often do you try to do restores? Is this built into your *weekly* routine yet?

    Much like a datacenter should test their generator-sets frequently, and on a schedule, backups also need to be tested.

  4. Luke says:

    We actually do restores on a regular basis when users request it -- when they've accidentally deleted or altered a lot of data, for instance. So we're quite confident in that process although your point is taken: we will make sure our backup/restore process is tested regularly even if none of our users need it.

  5. James says:

    What would it take for you to add the one.org banner to your blog to support charity? I have added it to my own and would love to see other bloggers amplify the need to stomp out poverty.

    If the activism irritates you then I understand...

    James McGovern
    http://duckdown.blogspot.com/

  6. Andrew Denny says:

    I dunno about Dabble, but the activism would irritate me :-)

    But I notice it's not simply activism, it's also nationalistic and political. I'm a Dabble user and in the UK, and at least one of our clients is a little irritated that we aren't using a UK-based solution.

    One.org seems to be US-focussed, and thus too nationalistic.

The comments to this entry are closed.