2015 Holiday Updates
Life is busy around the holidays with much family and new family additions. If there is one thing I wish I could remind the world it is that Christmas does last 12 days from Christmas Day until January 6th. So, Merry Christmas and a Happy 2016 to all! Here are some updates in no particular order.
Google Jump Hash Support in Buckytools
It looks like my Graphite cluster will be used for long term storage as we
migrate toward Prometheus – which means I get no relief in my ever-scaling
cluster. So, I’m adding Google’s Jump Hash to Buckytools. With this
hashing algorithm you cannot remove a node in the middle of the hash ring
without affecting all nodes afterwards in the ring so full support of
Graphite’s replication factor will find its way into Buckytools as well.
If I’ve not merged yet, take a look at the
jump branch. The plan here
is to be directly compatible with the FNV1a-Jump hashing now implemented
New Graphite Storage Backend
To support my Graphite work I’m moving toward a new storage backend that uses a Ceres-like approach. One or multiple files per metric using a columnar data format so the files grow based on the number of data points stored. Implementing this in Go will give incredible performance improvements and the new file format will give a marked reduction in storage space used. Some code for this is in my new Journal project.
Also, key to this project is being able to regularly compress older data. There are some interesting choices here that aught to help a lot with my storage requirements, but they make the archives not randomly accessible. Doing this for data past a certain age probably makes since.
This is probably the most challenging aspect of my plans for better Graphite scaling. Will I get the time? Will a better solution present itself? Will a entirely different method of handling telemetry that will get in the way of my plans of using Prometheus for our operational telemetry?
Graphite Metric Generator
Quick and dirty scaling tests for Graphite. This small tool will generate
random metrics with random data, or just random data for a known list of
metric names. The
gentestmetrics command is part of Buckytools.
Usage of ./gentestmetrics: Copyright 2015 42 Lines, Inc. Original Author: Jack Neely -c int Number of random metrics to generate. (default 1000) -f string Use metric names found in this file. Overrides -c. -i int Seconds between each timestamp/value pair per metric. (default 60) -l int Upper limit to randomly generated integer values. (default 100) -p string Prefix prepended to each metric name. (default "test.") -s int Time stamp to start -- iterations go BACKWARDS from here (default now) -v int Use the given value rather than a random integer. -x int How many timestamp/value pairs for each generated metric name. (default 1)
Metrics are printed to STDOUT. The most interesting usage is generating data for a list of known metrics delimited by newlines in a text file. Say with a 1 minute interval and 5 data points each:
./gentestmetrics -f metrics.txt -i 60 -x 5
The generated metrics are output in chronological order. However, the
timestamp given via
-s dictates the timestamp of the last data point rather
than the first.
Is There Anyone Not Podcasting?
A few friends and I started the Practical Operations Podcast. We talk about the practical side of operations work in small to large scale environments and the not so obvious implications of various tools and how you might use them in real world conditions. Check us out!