Disaster (Un)Procedures

October 30, 2006

We had a major electrical fault at our backup data centre this weekend. Thankfully it was our backup data centre so we only had very minor glitches, and to our secondary services at that.

It was a good chance to put those disaster recovery plans into full swing. And at 5am in the morning this time was as good as ever.

Here’s what we learned:

  • It’s the small things that catch you out

our electricians needed access to the dry riser cupboards, but then also to our roof to access the external power feeds. Where were the keys?

  • We’d lost power to the data centre and the lights were off

put two and two together, we needed a torch! The stairwell lights had gone too.

  • We needed to call the electricians, the air con crews, the building owner, the generator hire company

have these numbers to hand. Plaster them all over the data centre. Don’t have the need to go scrambling

  • Our file servers had cooked! What about all our support files and documentation?

We couldn’t access them. Luckily we keep copies off-site, and everyones (well, most) contact details in our phones.

  • Procedures?

Yes, we have procedures for Africa. We had them off-site. Did we use them? Not once. When the clock is ticking and the pressure is you don’t have time. Well trained, well drilled staff is what you need.

  • Turn a negative into a positive

Be truthful with your customers. By informing them that we had a major power failure but that we were fully up and running within 4 hours, with only minor and secondary glitches to services, they’ll knew that we were onto it. They appreciated the effort. They’ll know we’re not trying to pull the wool over their eyes. Truth and honesty – customers like that.

Basically, and you should all be sucking eggs by now, test test and triple test your disaster recovery processes. Even though our recovery process went very well, in our debrief today we still came up with a list of over ten actions!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: