I normally run Fedora on my personal systems at home.  I usually enjoy it and it helps keep me up to date with all the new toys that will eventually be a part of the RHEL machines I sysadmin as a professional.  I’ve been running Fedora 15 and had switched to XFCE as the new Gnome 3 user interface and I didn’t get along.  It was past time I updated to Fedora 16.

I’m a professional systems administrator.  If there is one thing that has taught me and I’d like to teach everyone else is that all hard drives fail.  Not if, but when.  So most machines I use (save for laptops) have 2 hard drives installed using Linux’s software RAID 1 to mirror them.  (Not two identical hard drives either.  Find ones that are different.)  Needless to say, my workstation at home is configured this way.  The /boot partition is md0 and everything else runs in LVM on top of md1.

I installed Fedora 16 on my workstation after backing up my data.  I normally do a clean install and reformat everything accept for the /home logical volume.  I get a brand new system and don’t lose my data.  Everything appeared to go well during the install.  When I rebooted I was greeted with an unfriendly Grub2 Rescue Mode.  The new boot loader couldn’t boot my system.

I’m quite familar with the older Grub and using its shell mode to recover my system.  Boy was I in for a suprise.  Grub 2’s command shell is completely different.  Unequipped with a “help” or “?” command to boot!  At this point the Grub2 rescue shell has 4 commands: ls, set, unset, and insmod.  Helpful isn’t it?

I was already using Google (from my smart phone).  There’s not a lot of quality documentation about recovering a system with Grub2.  There are quite a lot of Ubuntu articles.  This is a big problem.  A big problem in that Grub2 should have more visible documentation and a big problem that Fedora should have more visible documentation.

Learn More About Grub2 Rescue Mode

Running ls showed me the problem after I had figured out how Grub2 was working.  It only listed “(hd0) (hd1)”.  Grub didn’t see the /boot partitions because they were RAID 1 partitions.

Turns out Fedora has never “supported” /boot on a RAID 1 device.  Its worked for years and allowed me to recover broken systems many times.  The Fedora 16 installer does not have the Grub modules loaded to support Linux Software RAID devices.  (Yes, RAID 1 which you can mount one of the mirrors as normal ext4 filesystem in a pinch.)  There is a warning buried in my install logs that Grubby didn’t complete but the install itself had no errors.  Needless to say I am disappointed in many ways in the Fedora project.

Fortunately, the above Fedora 16 Common Bugs page had most of the solution.  After the install was complete and Grub2 ends up in the rescue shell this is how to recover your system.

1. Boot the Fedora 16 install media and use its Rescue Mode.

2. Tell the rescue mode to find and mount up your existing Linux system.  In my case the rescue mode couldn’t find my system.  I dropped down to the shell and ran

   # mdadm --assemble --scan
   # vgchange -ay
   

   This loaded up my Software RAID devices and the LVMs on top of them. Mount them properly under `/mnt/sysimage`.

3. You need to setup a chroot to run the Grub2 install program correctly.  Because the automated rescue didn’t locate my system and I mounted it my hand, I needed to create enough nodes in /mnt/sysimage/dev so I could continue.

   # cp -a /dev/* /mnt/sysimage/dev/
   

4. Do the chroot.

   # chroot /mnt/sysimage
   

5. Add the following lines to /boot/grub2/grub.cfg:

   insmod raid
   insmod mdraid09
   insmod mdraid1x
   

   At the top of the file is fine.

6. Now run the following commands:

   # grub2-install /dev/sda
   # grub2-install /dev/sdb
   

   At this point you should see a successful install of Grub 2 on both mirrors of the RAID 1. You should be able to reboot and have the system come up from the hard disks.