power outage at he.net

| | Comments (0)
here is what my provider said:

"At H.E. Fremont 1 around 9pm, on the night of 11/20 during a thunderstorm, there was a power incident lasting approximately 3 seconds. Two UPSes failed. Power was
routed around them and power was restored"

a bit ago, he.net fremont 1 had a power outage.   all our KVM guests and some of our Xen guests did not come back properly and were down for several hours.   Turns out we misconfigured a PDU and it defaulted to leaving all outlets off after power was restored.  (this pdu caused most of the KVM downtime.   the PDUs the Xen servers were on are okay, most of the xen server reboot issues were caused by misconfiguration on the servers themselves.) 

Anyhow, everything should be up and back to normal now.  

Note, hydra is garbage.  we need to move everyone off hydra.  

Leave a comment

About this Entry

This page contains a single entry by luke published on November 21, 2010 2:00 AM.

ipv6 routing, bgp and quagga was the previous entry in this blog.

another power outage at he.net is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.