Bad Experiences With Fedora

Posted on by
Reply

I normally run Fedora on my personal systems at home.  I usually enjoy it and it helps keep me up to date with all the new toys that will eventually be a part of the RHEL machines I sysadmin as a professional.  I’ve been running Fedora 15 and had switched to XFCE as the new Gnome 3 user interface and I didn’t get along.  It was past time I updated to Fedora 16.

I’m a professional systems administrator.  If there is one thing that has taught me and I’d like to teach everyone else is that all hard drives fail.  Not if, but when.  So most machines I use (save for laptops) have 2 hard drives installed using Linux’s software RAID 1 to mirror them.  (Not two identical hard drives either.  Find ones that are different.)  Needless to say, my workstation at home is configured this way.  The /boot partition is md0 and everything else runs in LVM on top of md1.

I installed Fedora 16 on my workstation after backing up my data.  I normally do a clean install and reformat everything accept for the /home logical volume.  I get a brand new system and don’t lose my data.  Everything appeared to go well during the install.  When I rebooted I was greeted with an unfriendly Grub2 Rescue Mode.  The new boot loader couldn’t boot my system.

I’m quite familar with the older Grub and using its shell mode to recover my system.  Boy was I in for a suprise.  Grub 2′s command shell is completely different.  Unequipped with a “help” or “?” command to boot!  At this point the Grub2 rescue shell has 4 commands: ls, set, unset, and insmod.  Helpful isn’t it?

I was already using Google (from my smart phone).  There’s not a lot of quality documentation about recovering a system with Grub2.  There are quite a lot of Ubuntu articles.  This is a big problem.  A big problem in that Grub2 should have more visible documentation and a big problem that Fedora should have more visible documentation.

Learn More About Grub2 Rescue Mode

Running ls showed me the problem after I had figured out how Grub2 was working.  It only listed “(hd0) (hd1)“.  Grub didn’t see the /boot partitions because they were RAID 1 partitions.

Turns out Fedora has never “supported” /boot on a RAID 1 device.  Its worked for years and allowed me to recover broken systems many times.  The Fedora 16 installer does not have the Grub modules loaded to support Linux Software RAID devices.  (Yes, RAID 1 which you can mount one of the mirrors as normal ext4 filesystem in a pinch.)  There is a warning buried in my install logs that Grubby didn’t complete but the install itself had no errors.  Needless to say I am disappointed in many ways in the Fedora project.

Fortunately, the above Fedora 16 Common Bugs page had most of the solution.  After the install was complete and Grub2 ends up in the rescue shell this is how to recover your system.

  1. Boot the Fedora 16 install media and use its Rescue Mode.
  2. Tell the rescue mode to find and mount up your existing Linux system.  In my case the rescue mode couldn’t find my system.  I dropped down to the shell and ran 
    # mdadm --assemble --scan
# vgchange -ay

    This loaded up my Software RAID devices and the LVMs on top of them. Mount them properly under /mnt/sysimage.

  3. You need to setup a chroot to run the Grub2 install program correctly.  Because the automated rescue didn’t locate my system and I mounted it my hand, I needed to create enough nodes in /mnt/sysimage/dev so I could continue. 
    # cp -a /dev/* /mnt/sysimage/dev/
  4. Do the chroot. 
    # chroot /mnt/sysimage
  5. Add the following lines to /boot/grub2/grub.cfg: 
    insmod raid
insmod mdraid09
insmod mdraid1x

    At the top of the file is fine.

  6. Now run the following commands: 
    # grub2-install /dev/sda
# grub2-install /dev/sdb

    At this point you should see a successful install of Grub 2 on both mirrors of the RAID 1. You should be able to reboot and have the system come up from the hard disks.

Getting Started with Python and Genshi

Posted on by
Reply

I’ve used the Genshi templating language toolkit for a few years now.  Its a great templating engine for producing XML, HTML, and plain text output.  In fact, its the engine that powers my Web-Kickstart system at work.

I recently stumbled across some of the first “Hello World” python code I wrote to figure out how Genshi works.  (I was also experimenting with out Genshi would handle different behaviors with custom classes.) I thought this might be useful to others, especially for folks looking to automate plain text files (like Red Hat Kickstarts).

IPTables: The MARK Target

Posted on by
Reply

Load balancing for High Availability and Disaster Recovery with LVS and Keepalived is fun, and quite powerful.  One of the most useful aspects is that you can use IPTables with the MARK target to select what traffic is routed to a set of real servers.  Its a lot more powerful than simple IP or IP/port combinations.

For example, a specific service may have a web site as well as another protocol.  Printing uses the IPP protocol and we have a web site documenting our printing system.  With the above trick you can create one virtual IP and have web traffic directed to a pool of web servers doing virtual hosting of many sites.  IPP traffic on a different port gets routed to a pool of Cups servers that do not maintain any web infrastructure.  End users only have to remember one DNS name.

However, remember that the MARK target is one of IPTables’ non-terminating-targets.  It doesn’t stop the packet from being processed by later iptables rules and possibly other MARK targets.  So your iptables rules need to be in order from least specific to most specific.  With the above example, let’s say all traffic goes to the Cups pool and only web traffic gets redirected to the Apache pool.  Your snippet that lives in /etc/sysconfig/iptables will look something like this:

*mangle
-A PREROUTING -d 10.0.0.5 -p tcp -m tcp -j MARK --set-mark 0xc
-A PREROUTING -d 10.0.0.5 -p tcp -m tcp --dport 80 -j MARK --set-mark 0xA
-A PREROUTING -d 10.0.0.5 -p tcp -m tcp --dport 443 -j MARK --set-mark 0xA

Where 10.0.0.5 is the print service’s Virtual IP and our LVS configuration is set to direct firewall mark 0xC to the Cups pool and 0xA to the web pool.

Yum API: Reloading Repos

Posted on by
Reply

While working with the Yum driver for Bcfg2 (a configuration management tool) I came across an interesting problem.  Bcfg2 would load the driver and Yum would read all its configuration and .repo files.  Next, Bcfg2 would attempt to install files and packages according to its specification.  Bcfg2 would gladly install additional .repo files for Yum, but the Yum driver would have no knowledge of them and wouldn’t be able to install packages from the new repo.  You would have to run the Bcfg2 client again for the Yum driver to proper install the new packages.  A bug, and it was time to fix it.

Turns out, using the Yum API, there isn’t a way to reset or reload the .repo files without a lot of work.  All my attempts just removed all repos from Yum’s little mind rendering it unable to install any package.  A suggestion from the Yum-devel mailing list was to simply reinstantiate the Yum API. So,

import yum
b = yum.YumBase()
...do interesting things...
b = yum.YumBase()
...do interesting things with updated .repo files...

As I was cleanly between transactions at that point in the code, this worked very well.

My Bcfg2 patches and work can be found at GitHub: https://github.com/jjneely/bcfg2

Repairing Users’ Accounts

Posted on by
Reply

How does one go about repairing a user’s UNIX/Linux account?  We very often run into this issue were things have gone wrong in a user’s account.  A corrupted Firefox profile, a bad .xsession script.  Or any number of problems.  How do other people go about resetting a user’s account back to its default configuration at scale?

We have several tools available on the web or through the GDM session manager that all end up calling one very small script.  This script has a collection of files and directories it moves safely out of the way and attempts to repair Firefox and Chrome profiles.

repair_dotfiles.sh

What does your IT shop do?

Configuring kdump on RHEL 6

Posted on by
Reply

Some quick notes for configuring kdump on RHEL 6.  Kdump produces a vmcore on a kernel panic, oops, or other condition that our friends a Red Hat support can use to debug kernel level issues.

  1. Make sure /var/crash has space for vmcores.  You need to have enough space for an entire dump of RAM just to be safe.
  2. Add crashkernel=128M to your kernel command line in /boot/grub/grub.conf
  3. Setup /etc/kdump.conf to save vmcores to the right place.  I normally have /var in a separate logical volume so I need to change the default location.  We also setup what memory pages to leave out and to use compression. 
    # cat /etc/kdump.conf
ext4 /dev/mapper/Volume00-var
path /crash
core_collector makedumpfile -c --message-level 1 -d 31
  4. Make sure the kdump service is set to start on boot and restart the system.
  5. Check that there is an initrd in /boot created for kdump.  It will have “kdump” in the file name.
  6. Test your configuration. 
    # echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger
  7. Configure the system to kernel panic on oops or NMI depending on the problem you are attempting to capture. Add these lines to /etc/sysctl.conf and then run sysctl -p as root. 
    kernel.panic_on_oops = 1
kernel.unknown_nmi_panic=1
kernel.panic_on_unrecovered_nmi=1

Dumb Tricks with gPXE

Posted on by
Reply

For my first bit of magic with gPXE I decided to replace the boot ISOs I have for folks that are unable to install machines via PXE.  With gPXE I don’t need my RHEL initrd and kernel image to boot strap myself into an install from a CD or USB stick.  I’ve encoded the TFTP server and the file name to grab and execute into the gPXE image so as long as the machine can get any type of DHCP lease it will load up my PXELINUX environment.  This makes the boot CD images work identically to doing a real PXE boot…because you are.

Step 1:  I grabbed the gPXE distribution and unpacked it.  I patched its autoboot functionality as described here.  This lets me DHCP automatically even if the first ethernet device is not the one connected to the network.  For gPXE 1.0.1 you can use my patch instead.

Step 2: Make an embedded script file.  This just supplies the information to gPXE that a normal PXE boot would get from the next-server and filename options in the DHCP response.

#!gpxe
autoboot
chain tftp://FQDN/pxelinux.0

Yup, we have DNS support so just add the FQDN of your TFTP server. In my setup I have pxelinux.0 in the root of my TFTP server.

Step 3: Build gPXE with your embedded script.

make EMBEDDED_IMAGE=path/to/your/script

Step 4: Burn the resulting ISO onto a CD and PXE boot a PXE-less machine.