Python, Linux, and Operations

Writing Documentation

Documentation is every IT professional's job. I keep a 3x5 notecard with a generic layout I use to write documentation and all of my IT related stuff follows this pattern. I'm actually tired of keeping up with the card, so I'm going to put it here.

I lifted and simplified this layout Tom Limoncelli's Ops Report Card section on documentation.

Overview or Summary

  • A summary of what this is.
  • Where does this service live?
  • Why do we need it?
  • Upstream documentation
  • Design (Perhaps its own section outright)
  • Diagrams of logic or data flow
  • Other moving parts that make the whole
  • Subject Matter Expert contacts

Common Tasks or Process

  • Common tasks needed for care and feeding.
  • If this is a documented process that process, step by step, goes here.

Deployment or Building

  • Do we build the software locally, how do we do it
  • How do we deploy more of these machines or replace busted ones
  • Where are our configs in Puppet/Chef/Ansible
  • Hardware Requirements

Pager Playbook

  • How might the system fail?
  • What does failure mean?
  • What risks do we run?
  • What to do to restore each service or part
  • What side effects happen when specific parts are down or malfunctioning

Disaster Recovery Plans

  • How is (or isn't) this system recoverable from a disaster situation?
  • What disasters have we planned for?
  • HA plans can fit here too
  • Steps that need to happen to recover
  • Risks
  • Service Level Agreement (either real/legal or social)


  • Any notes about the service
  • Things that don't fit well above
  • Future to do or improvements
  • Uncommonly needed tasks

Logging with Docker 1.0.1

Docker encourages its users to build containers that log to standard out or standard error. In fact, its now common practice to do so. Your process controller (uWSGI/Gunicorn) should combine the logs from all processes and do something useful with them. Like write them all to a file without your app having to worry about locking. Or, maybe even to Syslog.

Docker supports this practice and collects logs for us. In JSON to add missing timestamps and to work well with LogStash. But, what is a show stopping issue for us is that these files grow boundlessly. You cannot use the logrotate utility with them because the Docker daemon will not re-open the file. Well, unless you stop/start the specific container. Docker logging issues are an ongoing topic and this is clearly an area where Docker will improve in the future.

There are two other widely accepted ways of working around this:

  • Bind mount in /dev/log and off load logs to the host's Syslog
  • Mount a volume from the host or a different container where logs will be processed.

The second point is out. Same problem of not being able to easily tell the app to re-open files for log rotation without restarting the container.

Using /dev/log and off loading logs to the system's log daemon sounds like a good idea. The Docker host can provide this service arbitrarily to all containers. Containers need not deal with (much) logging complexity inside them.

This approach has multiple problems.

Off loading logs to the host's Syslog most likely means that you want to add some additional configuration to rsyslog which requires a restart of the rsyslog daemon. (Say, you want to stick your logs in a specific, app-specific file.) The first thing rsyslog does when it starts is (re-)create the /dev/log socket. At this point, any running Docker container that has already bind mounted /dev/log now has an old socket not the newly created one. In any case, rsyslog is no longer listening to any of the currently running containers for logs. Full stop. This method doesn't pass the smoke test.

What ended up working for me was using the network, but it added complexity to the Docker host. I'm managing Docker hosts with Ansible so this wasn't a huge problem. I'd rather tune my Docker hosts than alter each image and container. I set the network range on the docker0 bridge interface to a specific and private IP range. Now, my Docker hosts always have a known IP address that my Docker containers can make connections to. In /etc/default/docker:

DOCKER_OPTS="--ip --bip"

I configured rsyslog on the host to listen for UDP traffic and bind only to this private address:

$ModLoad imudp
$UDPServerRun 514

I then built my image to run the process with its output piped to logger using the -n option to specify my syslog server. Guess what. No logs.

The util-linux in Ubuntu Trusty (and other releases) is 2.20 which dates from 2011-ish. The logger utility has known bugs. Specifically that the -n option is ignored silently unless you also specify a local UNIX socket to write to. This version of util-linux also does not have the nsenter command which is very handy when working with Docker containers either. (See here for nsenter.) This is a pretty big frustration.

The final solution was to make my incantations in my Dockerfiles slightly more complex for apps that do not directly support Syslog. But, it works.

CMD foobar-server --options 2>&1 \
    | logger -n -u /dev/null -t foobar-server -p local0.notice

I promise I'm not logging to /dev/null.

Packages in Their Glory

I've been thinking about and wanting to write about packages for a long time. DEBs. RPMs. Pip for Python. CPAN for Perl. Galaxy for Ansible. Registry and Docker. Puppet modules from Puppet Forge. Vagrant Boxes. Every technology comes with its own distribution format and tool it seems.

My recent transition from RHEL to Ubuntu has made one thing very clear. This mess of packages is intractable. No package format is aware of the others yet they usually have dependencies that interconnect different package types. Pip has no knowledge of C libraries required by many Python packages to build. Us SAs usually end up crossing the streams to produce a working environment. Or we spend hours building packages of one specific type. (Only to spend even more time on them later.) The end result is often different package management systems stepping on each other and producing an unmaintainable and unreproducible system.

I've spent, probably, years of my career doing nothing but packaging. The advantages of packages are still just as relevant today as they were in the past. Its a core skill set for running large infrastructures.

Recently, I've just about given up trying to deal with packages. Throw-away VMs. Isolation environments. Images. Advanced configuration management tools. Applications with conflicting requirements. Does maintaining a well managed server even matter any more?

I believe it does. A well managed host system keeps things simple and the SAs sane. However, I believe that there should be a line drawn in the sand to keep the OS -- and tools that manage the OS -- separate from the applications running on that machine or VM. On the OS side of the line, RPMs or DEBs rule. Configuration management has an iron fist. Your configuration management and automation should also deploy your application containers. But now we find the line in the sand.

Your applications, its crazy requirements, as well as whatever abominable package management scheme needed to get the job done should live in Docker containers. Here, your configuration management is a git repo where you can easily rebuild your images. Here, we can use the tools we need that work the best for the situation at hand without causing harm to the host system or another application.

Perhaps Docker "packages" are, finally, the one packaging system to rule them all.

There's just one thing that itches. I know Fedora out right bans it. Packaging libraries with your applications means that when OpenSSL has a security vulnerability, you have to patch your OS -- and find everywhere else that library has been stuffed. Itch. Docker containers seem reasonable about this, but it still means rebuilding and restarting all containers. Itch.

Ansible and EC2

The last several months have been a deep dive into Ansible. Deterministic. Simplistic. Ideally push based. Uses standard YAML. (I've never been much for inventing your own language.) Most of this work has been with Amazon's EC2 service and Ansible's included dynamic inventory plugin for EC2 has been indispensable.

However, there's a lot more this inventory plugin could do. I've developed a series of patches, some of which I've submitted pull requests for. All of which can be had by cloning the master branch of my GitHub Ansible fork.

  • Do you have more than one AWS account? Need to manage machines in all of them? The multiple-aws-accounts branch teaches to query multiple accounts given the proper IAM credentials.
  • Making groups from various tags and attributes in AWS is handy. But I wanted a way to just make arbitrary groups happen. The arbitrary_groups branch supports reading groups from the ansible_groups AWS tag on your EC2 instances.
  • Need additional information about the EBS stores mapped to your instnaces? The block_device_mapping branch exposes this as inventory variables from

Monitoring Free Memory

One thing I've always have difficulty keeping straight is what the free command in Linux tells you about buffers and cache memory. Is that buffers and cache used and free (making you do the math of how much real free memory you have) or does the free command do the math for you? In tuning a Nagios setup, I researched it to make sure I had it right.

Misunderstandings about how Linux uses RAM are common and those misconceptions can lead to quite a few false positives in monitoring machines for memory pressure conditions. Linux doesn't eat RAM, although it will try to use as much RAM as possible to cache block devices which makes your machine much faster.

Remember, free does the math for you. Its goal is to inform you about free memory, not make you do math.

$ free -m
             total       used       free     shared    buffers     cached
Mem:        257948     256570       1377          0       3336     238963
-/+ buffers/cache:      14270     243677
Swap:       262127          1     262126

The -/+ buffers/cache line's two values are:

  1. Buffer/cache memory used subtracted from total used memory.
  2. Buffer/cache memory used added to total free memory.

So, in the above example, the value 243677 is the value you want to monitor with Nagios / Graphite and what us humans would like to see as the free memory metric.

More precisely the two values above are calculated like so:

  1. Total Used Memory - Buffers - Cached
  2. Total RAM - Value obtained in #1

See /proc/meminfo for the gory details.