luke: November 2010 Archives

ugh.  this needs to go in the pager system.  anyhow, I have several hundred messages to deal with tonight.  sorry for the delay.  

power outage at

| | Comments (0)
here is what my provider said:

"At H.E. Fremont 1 around 9pm, on the night of 11/20 during a thunderstorm, there was a power incident lasting approximately 3 seconds. Two UPSes failed. Power was
routed around them and power was restored"

a bit ago, fremont 1 had a power outage.   all our KVM guests and some of our Xen guests did not come back properly and were down for several hours.   Turns out we misconfigured a PDU and it defaulted to leaving all outlets off after power was restored.  (this pdu caused most of the KVM downtime.   the PDUs the Xen servers were on are okay, most of the xen server reboot issues were caused by misconfiguration on the servers themselves.) 

Anyhow, everything should be up and back to normal now.  

Note, hydra is garbage.  we need to move everyone off hydra.  

Billing system woes

| | Comments (0)

I broke the billing system last Wednesday with an ill-advised and ill-planned upgrade of FreeSide. we're rolling back to Wednesday's backup. As external orders were shut down and freeside-daily was not run, the only effected users should be those who made changes to their accounts in the meantime. I believe we've pinpointed the users in question, and will be emailing them privately. If you made a change to your account over the weekend and do not get an email, please check with

It's not over, though. we still plan on upgrading to a newer version of freeside, only this time we will do it on a test server with test data before screwing about with real data and taking down the billing system again

Also note, if we do screw something up and double-bill a few of you, we send bills... we don't actually have the capability to withdraw money from your papal account, so if we do double bill you, please complain, we will correct the error, and usually give you some extra credit for the trouble, but know that this won't cause your other stuff to bounce.

And we're done. Ordering is back, please report any irregularities.

About this Archive

This page is a archive of recent entries written by luke in November 2010.

luke: September 2010 is the previous archive.

luke: December 2010 is the next archive.

Find recent content on the main index or look in the archives to find all content.