UPDATE 21:52 -0800: VMs are back up. Total downtime was approximately 4 hours. We’ve sent credit to those of you impacted.
Our power distribution unit (PDU) in rack 05-11 failed, causing the entire rack to lose power. We replaced it with our spare. We believed we’ve identified the equipment that caused the original fault, and will write up a more detailed retrospective soon.
Our ability to blog was impacted by this outage. I provided updates in #prgmr on freenode.net, and welcome you to stop by if you have any questions.
We are currently experiencing downtime in one or more racks, which we believe to be power loss. We’ll be on the ground at the data center shortly.