Buckytools Updates: Sparse Files, Restorations
I’ve updated Buckytools, my suite for managing at scale consistent hashing Graphite clusters, with a few minor changes.
Sparse File Support
buckyd daemon now supports working with sparse Whisper DB files on disk.
In this case its assumed that you have
carbon-cache.py daemons running with:
WHISPER_SPARSE_CREATE = True
Any new Whisper files that
buckyd copies into place will also be checked,
in 4KiB blocks, for areas that can be made sparse. Therefore, when running
bucky rebalance files that were sparse on one server can be moved to a new
server and recreated as sparse files.
bucky tar works as before but the generated archives do not have the GNU
sparse types set and if expanded by hand will not automatically result in
bucky du command works as before and reports the apparent size of the
Whisper files on disk. Similar to:
du -hs --apparent-size
Restoring tarballs with
bucky restore attempts to create sparse files in
To enable support for sparse Whisper DB files run the daemon with the
description "Buckyd, the Buckytools daemon" author "Jack Neely <email@example.com>" start on startup stop on shutdown setuid graphite exec /usr/bin/buckyd --sparse \ graphite010:2104:a \ graphite011:2104:a \ graphite012:2104:a
Bucky Restore Bug Fixes
I can tell that we restore tarballs a lot using these tools. Oops! I’ve
bucky restore to properly ignore directories in the tarballs
rather than create 0 length Whisper DBs in the cluster.
This tool that listens for Graphite’s Pickle Protocol and emits Graphite’s text protocol has had some more verbose debugging added. This tool is on my short list to work on some improvements to, such as not storing things in memory as strings. Go’s UTF8 strings are very resource and memory intensive.