srn: April 2013 Archives

lozenges is down

| | Comments (0)
UPDATE 12:36 PST: The raid is rebuilding right now.  Since everyone is already down and it takes much longer to rebuild the raid when guests are running, we're going to wait another 2-3 hours and let the raid finish rebuilding before bringing up guests.  Everyone on lozenges will be getting a month credit.  We will post when the guests on lozenges have been brought back up.


As of 10:15 PST 2 disks in the same mirror were kicked out of the raid.  We're at 55 south market with 2 new hard drives.  Working on it.

UPDATE: it appears that one disk fell out of the array at Sat Apr 20 01:51:06 2013, the other at  Sat Apr 20 02:12:00 2013.  We aren't quite sure why there wasn't an alert sent until 10:15 this morning.
Commands run:
(Repair filesystem) 73 # mdadm --assemble /dev/md1
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
(Repair filesystem) 74 # mdadm --assemble --scan /dev/md1
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
(Repair filesystem) 75 # mdadm --assemble --force /dev/md1
mdadm: forcing event count in /dev/sdd2(3) from 15103 upto 15105
mdadm: clearing FAULTY flag for device 0 in /dev/md1 for /dev/sdd2
mdadm: /dev/md1 assembled from 3 drives - not enough to start the array.
(Repair filesystem) 76 # mdadm --assemble --force /dev/md1 --run
mdadm: /dev/md1 has been started with 3 drives (out of 4).
Partitioning new /dev/sdc:
(Repair filesystem) 86 # sfdisk --dump /dev/sda > sda.sfdisk
(Repair filesystem) 87 # sfdisk /dev/sdc < sda.sfdisk
(Repair filesystem) 93 # mdadm /dev/md0 -a /dev/sdc1
(Repair filesystem) 94 # cat /proc/mdstat
(Repair filesystem) 95 # mdadm /dev/md1 -a /dev/sdc2
(Repair filesystem) 96 # cat /proc/mdstat
UPDATE: another round of emails will be going out within the next few hours, please do not email support until (tomorrow) 2013-04-16 if you have not received an email already.
The following IP ranges outside of are NOT being renumbered at this time:

  • is owned by directly and will not be renumbered at all.

If your IP is outside of these ranges and you have not yet received an email regarding the new IP allocation, please contact

You will need to update your ipv6 address should you be using ipv6.  The new ipv6 address can be found the same way as the ipv4 address by running

host [username]

About this Archive

This page is a archive of recent entries written by srn in April 2013.

srn: November 2012 is the previous archive.

srn: May 2013 is the next archive.

Find recent content on the main index or look in the archives to find all content.