Life is busy around the holidays with much family and new family additions. If there is one thing I wish I could remind the world it is that Christmas does last 12 days from Christmas Day until January 6th. So, Merry Christmas and a Happy 2016 to all! Here are some updates in no particular order.

Google Jump Hash Support in Buckytools

It looks like my Graphite cluster will be used for long term storage as we migrate toward Prometheus – which means I get no relief in my ever-scaling cluster. So, I’m adding Google’s Jump Hash to Buckytools. With this hashing algorithm you cannot remove a node in the middle of the hash ring without affecting all nodes afterwards in the ring so full support of Graphite’s replication factor will find its way into Buckytools as well. If I’ve not merged yet, take a look at the jump branch. The plan here is to be directly compatible with the FNV1a-Jump hashing now implemented in carbon-c-relay.

New Graphite Storage Backend

To support my Graphite work I’m moving toward a new storage backend that uses a Ceres-like approach. One or multiple files per metric using a columnar data format so the files grow based on the number of data points stored. Implementing this in Go will give incredible performance improvements and the new file format will give a marked reduction in storage space used. Some code for this is in my new Journal project.

Also, key to this project is being able to regularly compress older data. There are some interesting choices here that aught to help a lot with my storage requirements, but they make the archives not randomly accessible. Doing this for data past a certain age probably makes since.

This is probably the most challenging aspect of my plans for better Graphite scaling. Will I get the time? Will a better solution present itself? Will a entirely different method of handling telemetry that will get in the way of my plans of using Prometheus for our operational telemetry?

Graphite Metric Generator

Quick and dirty scaling tests for Graphite. This small tool will generate random metrics with random data, or just random data for a known list of metric names. The gentestmetrics command is part of Buckytools.

Usage of ./gentestmetrics:
Copyright 2015 42 Lines, Inc.
Original Author: Jack Neely

  -c int
        Number of random metrics to generate. (default 1000)
  -f string
        Use metric names found in this file.  Overrides -c.
  -i int
        Seconds between each timestamp/value pair per metric. (default 60)
  -l int
        Upper limit to randomly generated integer values. (default 100)
  -p string
        Prefix prepended to each metric name. (default "test.")
  -s int
        Time stamp to start -- iterations go BACKWARDS from here (default now)
  -v int
        Use the given value rather than a random integer.
  -x int
        How many timestamp/value pairs for each generated metric name. (default 1)

Metrics are printed to STDOUT. The most interesting usage is generating data for a list of known metrics delimited by newlines in a text file. Say with a 1 minute interval and 5 data points each:

./gentestmetrics -f metrics.txt -i 60 -x 5

The generated metrics are output in chronological order. However, the timestamp given via -s dictates the timestamp of the last data point rather than the first.

Is There Anyone Not Podcasting?

A few friends and I started the Practical Operations Podcast. We talk about the practical side of operations work in small to large scale environments and the not so obvious implications of various tools and how you might use them in real world conditions. Check us out!