mares down again.

| | Comments (2)
definitely a hardware problem.  I'm going to reboot it now, and replace the hardware with a spare tonight.

Yeah, uh, it's not rebooting, even goosed from the PDU.  I've gotta swap the hardware.  expect another 2 hours of downtime. 

update:  ugh,  that was a lot longer than 2 hours.  We are almost done.  A comedy of errors; I left my rescue CD at home and the dhcp/pxeboot server I normally use to get rescue images in this case are on mares.   expect a good debriefing on how I'm going to prevent these problems going forward after I have it up. 

Note, yes, the dhcp server is down.  Customers that are on dhcp (which should be none of the customers I setup)   may be down now.  You can fix this by adding your IP address statically. 

I am kind of in and out of the #prgmr channel on as I work.

It is done.  Everyone is back.   I am cleaning up at the data center, then I will stop for a burger and clean up the support queue.   If you are still down, complain loudly and I will help.

The mares hardware is completely replaced (save for two harddrives that I needed to keep;  raid is rebuilding now.)     I will figure out what the problem was later. 


any ETA yet? my VM still seems down.

you should be back up. what's your username? Like I said, this impacted both users on mares and users that were using DHCP (we /will/ put in a backup dhcp server) so if you are a user that is not on mares but uses dhcp, you need to get a new dhcp lease. First kill the running dhclient then,

dhclient eth0

should do the trick. (a reboot also will work)

After that, you want to go assign the IP statically (that is, put it in /etc/network/interfaces if you are debian, or /etc/sysconfig/network-scripts/ifcfg-eth0 and gateway in /etc/sysconfig/network if you are centos)

Leave a comment

About this Entry

This page contains a single entry by luke published on February 2, 2012 12:02 PM.

mares crashed was the previous entry in this blog.

New update on servers is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.