Telemetry and monitoring with a Time Series Database (TSDB) is a set of 3 overlapping problem spaces. Those are:
Ingestion and Retrieval of Raw Data Points – Operational Intelligence Aggregation and Roll Ups of Data for Long Term Retention and Trending – Business Intelligence Efficient Storage and Retrieval of High Velocity Event Data as a Statistical Model – Histograms Most solutions that exist are really only aware of 2 of these problems.
My friend Breandan and I presented at the local Linux Users’ Group on Thursday, September 12, 2019. Our presentation is called “Observability Through the Lenses of Metrics and Events.” In a large part, its an effort to build conversations around the differences between Metrics, Log Events, and Tracing and why you would use each.
Of course, the subtext is that we’d like to drum up some new clients for our employer, 42 Lines, Inc.
When I was young and learning to program it was a milestone to write a Pizza Menu and Ordering program. Select your pizza, crust, toppings, and see how much that pizza is going to cost. Text mode or graphical and in a number of languages I created this program over and over.
Today, its seems that the milestone to achieve is to write a Time Series Database (TSDB). You can’t be cool without one under your belt.
I run a centralized Alertmanager service for about 100 individual software development and operations teams. Normally, each team has their own Prometheus VM(s) and sometimes a dedicated Prometheus VM or two will be created for a specific big or busy service. A single pair of Alertmanager instances scales very nicely at this load with 100s of alerts firing.
However, after a specific configuration change, I started having really weird effects. Commonly I would get two identical pages and some other teams experienced this.
At Monitorama PDX 2019 I was honored to be selected to give a 5 minute lighting talk. I figured there woud be no better time to show off some neat things you can do with PromQL.
The talk recordings are up, and here is a direct link to the time offset in the Lighting Talks where I spoke. Enjoy!
Monitorama PDX 2019 - Lightning Talks from Monitorama on Vimeo.
I’ve been experimenting with Thanos over the last few months to add long term storage of Prometheus metric data and a single query endpoint for a large Prometheus fleet. This looks like a really well designed solution to end the 30-ish day retention limits and to solve the oft asked question “What host name is my Prometheus box?”
Thanos requires that each Prometheus VM or machine in the fleet have a unique set of external labels.
This tool generates Prometheus 2.x TSDB blocks containing test data. If you
have ever wanted to test Prometheus or Thanos behavior on a specific
time series foot print, say a year’s worth of data, then this tool is for you.
nos-tal-gia (noun) – A sentimental longing or wistful affection for the past, typically for a period or place with happy personal associations.
I’ve been thinking a lot about retro-computing. I am most definitely a child of the 1980’s – the glory days of the personal computer revolution. However, I was never a “gamer.” In fact, I used to think the term was an insult. So my nostalgia is wrapped up in my curiosity of how computers work and the ability to master a new bit of code.
Rule #4 states that failure will happen, therefore you should plan for the eventual reality. The Linux workstations I build and use (if I have any say about it) use at least 2 hard drives in some mirrored or otherwise redundant fashion. My current patternis to build workstations with a small (120G or there about) SSD drive as the boot drive that contains my OS install and swap space. /home, scratch, and possibly other areas are mounted from a 2 disk mirrored array of spinning rust.
If a job in Linux System Administration / Operations can teach you one
thing its how to keep up with the ever changing landscape that Open Source
is. I’ve been working with Linux for 20+ years, and with that comes some,
hopefully, wisdom of experience. Linux distributions, and Open Source are
divergent in terms of change. The more things change, the more things there
are to change.