November 2010 Archives

ugh.  this needs to go in the pager system.  anyhow, I have several hundred messages to deal with tonight.  sorry for the delay.  

another power outage at

| | Comments (2)
Around 7:30 this morning, there was another power outage at Fremont. We will post more information when we have it, and now that power is back we will make sure everything boots up correctly. boar, hydra, stables, birds and hamper are back up but lion and the kvm servers are not. Luke and I are going to Fremont now to see about them. If your vps is on one of the servers that is back up now and is still not working, email

edit:  word has it that was in 'reduced redundancy' mode to work on a UPS when the power event occurred.  

power outage at

| | Comments (0)
here is what my provider said:

"At H.E. Fremont 1 around 9pm, on the night of 11/20 during a thunderstorm, there was a power incident lasting approximately 3 seconds. Two UPSes failed. Power was
routed around them and power was restored"

a bit ago, fremont 1 had a power outage.   all our KVM guests and some of our Xen guests did not come back properly and were down for several hours.   Turns out we misconfigured a PDU and it defaulted to leaving all outlets off after power was restored.  (this pdu caused most of the KVM downtime.   the PDUs the Xen servers were on are okay, most of the xen server reboot issues were caused by misconfiguration on the servers themselves.) 

Anyhow, everything should be up and back to normal now.  

Note, hydra is garbage.  we need to move everyone off hydra.  

ipv6 routing, bgp and quagga

| | Comments (2)
Our ipv6 router wasn't routing outbound for a few hours this morning because I moved the ipv6 default route configuration from into quagga and I mispasted it. We're starting to use quagga because we are having more static routes to manage and we are also planning to start using bgp. Today we will send a letter of agency to our provider to advertise some routes for our customers who have their own provider independent address blocks, and we plan to have our own block soon also (hopefully!). We should be able to multihome our provider address blocks too which will help reliability. Wish us luck!

Billing system woes

| | Comments (0)

I broke the billing system last Wednesday with an ill-advised and ill-planned upgrade of FreeSide. we're rolling back to Wednesday's backup. As external orders were shut down and freeside-daily was not run, the only effected users should be those who made changes to their accounts in the meantime. I believe we've pinpointed the users in question, and will be emailing them privately. If you made a change to your account over the weekend and do not get an email, please check with

It's not over, though. we still plan on upgrading to a newer version of freeside, only this time we will do it on a test server with test data before screwing about with real data and taking down the billing system again

Also note, if we do screw something up and double-bill a few of you, we send bills... we don't actually have the capability to withdraw money from your papal account, so if we do double bill you, please complain, we will correct the error, and usually give you some extra credit for the trouble, but know that this won't cause your other stuff to bounce.

And we're done. Ordering is back, please report any irregularities.