So earlier today I was stuck trying to bounce a remote host that had a Software RAID volume. It turns out for some reason (I still suspect a faulty volume in the RAID1 volume) nothing was able to write to the mounted RAID volume. All processes kept getting hung and drove the load up to 25. Responsiveness was still good since this load was actually due to I/O waiting processes that kept getting some CPU time and not CPU-intensive processes trying to share a couple of processors.
In any case, after manually failing the partition that was causing problems with the RAID volume I still wasn't able to bounce the box since the RAID volume won't stop or sync and actually halted the shutdown process. I thought about the magic SysRq but realized I was ssh'ed in and that won't work. For some reason I didn't think of searching for a way too invoke the magic SysRq processes from the commandline. I looked for a way to force a shutdown and found this blog. It worked like a charm!
Intriguingly enough, when the host came back up the RAID volume remounted and both partitions that formed the RAID volume were fine as reported by mdadm. What's more all seemed well with the mounted volume and I was able to read/write to it just fine. I'll be keeping an eye on that for a bit.
No comments:
Post a Comment