lozenges down, same problem as Rutalidge last night

| | Comments (0)
which is to say, we swapped a hot-swap drive, and it hung during the RAID rebuild.    I suspect that we need to limit the raid rebuild speed, but I don't have hard evidence.   We will research this.

The annoying thing is that it hangs rather than printing something useful to the console.   

Anyhow, we took the box down and are rebuilding the raid in single user mode.   

sh-3.2# cat /proc/mdstat 
Personalities : [raid1] [raid10] 
md1 : active raid10 sda2[0] sdb2[1] sdc2[2] sdd2[4]
      1048578048 blocks 256K chunks 2 near-copies [4/3] [UUU_]
      [=>...................]  recovery =  6.9% (36467328/524289024) finish=71.9min speed=112928K/sec
md0 : active raid1 sdd1[3] sdc1[2] sdb1[1] sda1[0]
      10482304 blocks [4/4] [UUUU]
unused devices: <none>

So figure we'll be back in around an hour and a half.

edit at 20:40 PST:  rebooted.  guests are coming back up.  I expect no further problems (at least until the next disk fails; we will hopefully have figured out a solution by then.)

Leave a comment

About this Entry

This page contains a single entry by luke published on December 16, 2011 7:12 PM.

rutledge raid rebuild was the previous entry in this blog.

stone hung this morning, looks similar to the other two hangs is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.