Python, Linux, and Operations

Monitoring Free Memory

One thing I've always have difficulty keeping straight is what the free command in Linux tells you about buffers and cache memory. Is that buffers and cache used and free (making you do the math of how much real free memory you have) or does the free command do the math for you? In tuning a Nagios setup, I researched it to make sure I had it right.

Misunderstandings about how Linux uses RAM are common and those misconceptions can lead to quite a few false positives in monitoring machines for memory pressure conditions. Linux doesn't eat RAM, although it will try to use as much RAM as possible to cache block devices which makes your machine much faster.

Remember, free does the math for you. Its goal is to inform you about free memory, not make you do math.

$ free -m
             total       used       free     shared    buffers     cached
Mem:        257948     256570       1377          0       3336     238963
-/+ buffers/cache:      14270     243677
Swap:       262127          1     262126

The -/+ buffers/cache line's two values are:

  1. Buffer/cache memory used subtracted from total used memory.
  2. Buffer/cache memory used added to total free memory.

So, in the above example, the value 243677 is the value you want to monitor with Nagios / Graphite and what us humans would like to see as the free memory metric.

More precisely the two values above are calculated like so:

  1. Total Used Memory - Buffers - Cached
  2. Total RAM - Value obtained in #1

See /proc/meminfo for the gory details.

Thinking About Keyboards

I've been thinking about my keyboards of late. This is mine.

IBM Model M Manufactured in 1988

I've had a collection of IBM Model M keyboards since before college and I've picked up a few more since. My keyboards are old enough to drink. They click at each other in the ABC store. They feel like home.

Fortunately, what I have known for years (that mechanical keyboards are superior for us that code or work in operations all day) has gone main stream. There are actually quite a few options for new keyboards. A friend of mine recently got a Das Keyboard (the completely blank and black one). I have been looking at the more expensive CODE Keyboard. The backlit keys and minimalistic design strike me.

Are they as good as ye olde Model Ms? Less key travel to actuation perhaps? Not quite as loud? (I'm on the headset a lot with the new job.) Or are buckling springs the best for a RSI free life?

As I've been thinking about this through the last mouse button fails.


Nothing ever quite stays the same. The best SAs understand this almost unconsciously. I no longer work for NC State University and have started with a small company called 42 Lines. Its been a challenge, and I've been drinking from the fire-hose. Instead of teaching and advising I've been learning.

Clearly, this calls for a website make-over. So, welcome to the new LinuxCzar powered by Pelican. I've been wanting to move to something other than Wordpress for lots of reasons. Its time to get back to Python, lose the ever annoying comment spam, and make one less Wordpress install to maintain. Best of all, the website can live in Git as all things should.

For a Python fan, I am using the Octopress theme ported to Pelican. This them has stood out to me as fantastic design for a long time. Google fonts and the works. However, it does require Ruby tools to modify the CSS and doesn't seem to have a support for a lot of reStructuredText directives. So it may not last. Know of a better theme Pelican theme?

However, there is nothing better than being able to write in reStructuredText.

Past Projects

This page documents some past projects that I have been involved with. If something looks of interest to you, check it out. Also, you are welcome to ask any questions or even submit a patch!


Public projects, random source, and backups often live in the LinuxCzar Git repository linked below.

If you have any questions including that about licensing, please let me know.

Realm Linux

Realm Linux was my main reason for life at a former employer, NC State University. My involvement ended in late 2013. This is a modified version of Red Hat's operating systems (as of this writing RHEL) that integrates seamlessly into NC State's computing environment.  This includes Kerberos, hesiod, OpenAFS, automated Kickstart installs, and configuration management with Puppet.  You may also be interested in the more general website for Campus Linux Services as it contains links to more projects and code created in my role at NC State.

Repositories of Note:


I was the lead maintainer of Current from 2004 to 2007.  Current aimed to support the Yum and RHN protocols and develop an Open Source errata and repository management system.  This project was superseded by Red Hat's release of Spacewalk.

Partitions: 1981 Technology We Need To Ditch

I don't like partitions.  MS-DOS partition tables or GUID Partition Tables (GPT) alike.  We use them because you always partition a disk, right?  We use them without understanding the ramifications.  The MS-DOS partition table was designed in 1981.  We still blindly use it on almost every machine today.  Do we still surf the Internet with a Commodore 64?

MS-DOS partition tables cannot handle drives of more than 2TiB.  Intel designed GPT in the 1990s to deal with this and add more than 4 (count 'em, 4) primary partitions.  Its better, and is now part of the EFI standard.  For booting your hardware, a partition table can be quite handy.  Or even required.

However, I've long advised folks not to wrap partition tables around their dedicated storage arrays. Growing storage arrays comes up multiple times a week for me.  An outage is required if there are partition tables in use.  A longer outage is needed if we have to convert the array to GPT to cross the 2 TiB barrier.  I have always advised folks use raw LVM physical volumes on their storage arrays.  LVM handles all the manipulation for us in a smooth and consistent way in this case.  No reboots either as long as you instruct Linux to rescan the SCSI bus after a resize.

echo 1 > /sys/class/scsi_device/DEVICE/rescan

Then grow the LVM physical volume:

pvresize /dev/DEVICE

Finally, you may resize the logical volumes and your file systems:

lvresize -l+100%FREE -r /dev/mapper/Volume00-foobar

No outage to handle growth in your dedicated storage array.

With so much infrastructure based on VMs (virtual machines) like KVM or VMWare we can take these concepts further.  Growing a root file system on a VM isn't hard and shouldn't require a Linux expert to handle.  Many cloud providers have tools that do this for their customers.  Take into consideration:

  • VMWare and other VM tools are fully capable of doing all of our storage virtualization that us Linux folks commonly do with LVM.
  • Certain enterprise Linux distributions turn off the kernel's feature to re-read partition tables for safety.  In any case, the only way to safely grok a modified partition table is to reboot.  We can manipulate file systems without an outage, so why do we need an outage here?
  • Depending on the VM's partitioning and layout, it can require a large amount of skill to move and resize partitions to extend file systems.  If the VM uses LVM that helps in this case.
  • What if, by using a standard method of deployment, your worst case scenario is your VM guys extend the file space, and the customer is told to run one or two commands as root?  What if by using standard methods of deployment you could automate this process?

I've been attempting to build a better way for us to deploy VMs.  Each of these VM has 3 different virtual disks:

  1. 512MiB for /boot.  This is partitioned with an MS-DOS table so the machine can boot and the MBR is protected for the bootstrap procedure.  Rarely does /boot need to become larger.
  2. Swap.  2GiB.  Vary according to your needs.  Not partitioned!  Raw swap.  Depending on memory load, resizing swap can be done online.
  3. 30GiB, more, or less depending on your needs.  Not partitioned.  This is a raw LVM physical volume used to build your logical volumes for how you would like your file system separated out.

The Red Hat installer doesn't support this.  So creating and using an image as a template for your VM farm can be very handy.  But whether you use images or, like me, use Kickstart, you need to get the Installer to actually install this layout.  Below are the relevant Kickstart snippets that will install into the above configuration -- at least with RHEL 6. But once the above can be reached, its a simple matter to grow the third virtual HD and use pvresize and lvresize to extend the native filesystem without an outage to your systems. (My devices here are /dev/vda, /dev/vdb, and /dev/vbc.)

clearpart --drives vda --all

part /boot --size 1 --grow --ondisk vda

volgroup Volume00 vdc --useexisting

logvol /          --size 8704 --fstype ext4 --vgname=Volume00 --name=root
logvol /tmp       --size 2048 --fstype ext4 --vgname=Volume00 --name=tmp
logvol /var       --size 7168 --fstype ext4 --vgname=Volume00 --name=var

# Clean up any possible left overs...
vgremove -v -f Volume00

# Whipe any partition table
dd if=/dev/zero of=/dev/vdb bs=512 count=1
dd if=/dev/zero of=/dev/vdc bs=512 count=1

# Create an LVM Volume Group
pvcreate -y /dev/vdc
vgcreate Volume00 /dev/vdc

# Create swap device
mkswap /dev/vdb -L vmswap
swapon -L vmswap

# Setup swap space in fstab
echo "LABEL=vmswap  swap    swap    defaults  0 0" >> /etc/fstab