lozenges is down

| | Comments (0)
UPDATE 12:36 PST: The raid is rebuilding right now.  Since everyone is already down and it takes much longer to rebuild the raid when guests are running, we're going to wait another 2-3 hours and let the raid finish rebuilding before bringing up guests.  Everyone on lozenges will be getting a month credit.  We will post when the guests on lozenges have been brought back up.

---

As of 10:15 PST 2 disks in the same mirror were kicked out of the raid.  We're at 55 south market with 2 new hard drives.  Working on it.

---
UPDATE: it appears that one disk fell out of the array at Sat Apr 20 01:51:06 2013, the other at  Sat Apr 20 02:12:00 2013.  We aren't quite sure why there wasn't an alert sent until 10:15 this morning.
--
Commands run:
(Repair filesystem) 73 # mdadm --assemble /dev/md1
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
(Repair filesystem) 74 # mdadm --assemble --scan /dev/md1
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
(Repair filesystem) 75 # mdadm --assemble --force /dev/md1
mdadm: forcing event count in /dev/sdd2(3) from 15103 upto 15105
mdadm: clearing FAULTY flag for device 0 in /dev/md1 for /dev/sdd2
mdadm: /dev/md1 assembled from 3 drives - not enough to start the array.
(Repair filesystem) 76 # mdadm --assemble --force /dev/md1 --run
mdadm: /dev/md1 has been started with 3 drives (out of 4).
--
Partitioning new /dev/sdc:
(Repair filesystem) 86 # sfdisk --dump /dev/sda > sda.sfdisk
(Repair filesystem) 87 # sfdisk /dev/sdc < sda.sfdisk
(Repair filesystem) 93 # mdadm /dev/md0 -a /dev/sdc1
(Repair filesystem) 94 # cat /proc/mdstat
(Repair filesystem) 95 # mdadm /dev/md1 -a /dev/sdc2
(Repair filesystem) 96 # cat /proc/mdstat
--

Leave a comment

About this Entry

This page contains a single entry by srn published on April 20, 2013 11:47 AM.

Update: New (direct allocation) IP addresses was the previous entry in this blog.

serious issues on rehnquist is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.