I’ve updated Buckytools, my suite for managing at scale consistent hashing Graphite clusters, with a few minor changes.

Sparse File Support

The buckyd daemon now supports working with sparse Whisper DB files on disk. In this case its assumed that you have carbon-cache.py daemons running with:

WHISPER_SPARSE_CREATE = True

Any new Whisper files that buckyd copies into place will also be checked, in 4KiB blocks, for areas that can be made sparse. Therefore, when running bucky rebalance files that were sparse on one server can be moved to a new server and recreated as sparse files.

Using bucky tar works as before but the generated archives do not have the GNU sparse types set and if expanded by hand will not automatically result in sparse files.

The bucky du command works as before and reports the apparent size of the Whisper files on disk. Similar to:

du -hs --apparent-size

Restoring tarballs with bucky restore attempts to create sparse files in the cluster.

To enable support for sparse Whisper DB files run the daemon with the -sparse option:

description "Buckyd, the Buckytools daemon"
author      "Jack Neely <[email protected]>"

start on startup
stop on shutdown

setuid graphite

exec /usr/bin/buckyd --sparse \
    graphite010:2104:a \
    graphite011:2104:a \
    graphite012:2104:a

Bucky Restore Bug Fixes

I can tell that we restore tarballs a lot using these tools. Oops! I’ve corrected bucky restore to properly ignore directories in the tarballs rather than create 0 length Whisper DBs in the cluster.

bucky-pickle-relay

This tool that listens for Graphite’s Pickle Protocol and emits Graphite’s text protocol has had some more verbose debugging added. This tool is on my short list to work on some improvements to, such as not storing things in memory as strings. Go’s UTF8 strings are very resource and memory intensive.