outage: November 2008 Archives

never again

| | Comments (0)
will I schedule co-lo work on a friday night.  Bad idea.  I'm tired from the work week, and I haven't had time to properly prepare and test everything.  Downtime was a little over 3 hours.  I'll figure out how that fits into the SLA  in the morning. 

But yeah.  Also, I'm never again going to buy a server without hot-swap drives.  this would have gone considerably faster with hot swap drives.

And furthermore, I am going to abandon the more complex mirrored LVM setup I currently have in favor of using md to do the mirroring and using LVM on top of my MD devices. 


downtime should be under 10 minutes.  you are being moved on to the new server, which means that our new,  low-priced plans are available.  (I'm not done tweaking the prices/ram, I will probably even out the curve a little, but the price per megabyte of ram won't be going up.) 

I will be removing two of our legacy servers to make room for our new server.   (one of them doesn't have customers on it, only customers on hydra should be impacted by this.) 

It has begun,  hind is going down now.  After that, hydra
Late Friday night, I'm going to head down to the co-lo and replace the failed drive on Boar's mirror.  If this task goes well, customers will notice nothing more than a short period of unreachability. the save/restore should put them back where they were without a reboot. 

ok starting now

´╗┐ok, bad drive is samsung w/ serial s13Uj1nq108089

always mark serials before replacing drives... this much I have learned.


About this Archive

This page is a archive of entries in the outage category from November 2008.

outage: September 2008 is the previous archive.

outage: December 2008 is the next archive.

Find recent content on the main index or look in the archives to find all content.