An error in judgement

| | Comments (0)
so yeah, I emailed 15 servers of users, told them they'd have an hour of downtime "Saturday Evening" 

I figured I'd do 5 servers at a go.   small number.   

So yeah, I came up with some spare rails, screwed 'em in, setup the PDU, etc... did all the usual things you do before you move servers.   But these things have a way of taking longer than you expect, and I also had to configure an unfamiliar switch (my force10 10gbe setup, which I still don't have configured) 

So after we setup the destination co-lo, we headed to the source co-lo (55 s. market.) - this means getting tickets to remove servers and arranging it so that the security folk know we are coming and verify our tickets before we shut things down (it always takes a few tries to get the ticket right to leave with hardware.)   so Anyhow, it was around 4:30AM local time when we started shutting things down.  It's 9:00am now.     I really should have called it off around 1:00am or so.   

I mean, the rest of our problems have to do with the fact that these are some of the oldest servers (both hardware wise and software wise) in the fleet.   Well, that, and I'm not exactly at my mental best at this point.

Anyhow, we're back up.  I'm sorry.  


Leave a comment

About this Entry

This page contains a single entry by luke published on May 5, 2013 9:05 AM.

network outage at 4:00AM pst was the previous entry in this blog.

dhcp currently (kindof) broken, will fix after sleep is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.