Java isn’t my favorite language to work in. However, I realized that to roll out a successful Observability plan that I needed good examples and most of the teams I work with create Spring Boot applications. So off I went to create a simple Java Spring Boot application that demonstrated a structured logging approach, metrics following the 4 Golden Signals pattern, and integration with basic tracing. The goal is to show how to pivot through one tool’s data to another. The problem was working with Prometheus’s Exemplars.

I swear that I’ve been looking at and thinking about how best to use Prometheus Exemplars since sometime in 2019. So I’ve been surprised to find that support for them is still very new. The Java world has only gotten some support for them this past year, 2021. However, the documentation indicates that they are well supported when pairing Prometheus’s client_java library with OpenTelemetry. I wanted to show off being able to pivot from a metric dashboard or an alert to the traces generated by the offending request.

When working with Java Spring Boot style applications I usually suggest folks implement Prometheus metrics by way of Micrometer. However, when I started this journey, support here didn’t look promising. So, I ripped out Micrometer from my example Spring Boot application and created equivalent instrumentation with Prometheus’s client_java library using version 0.12.0 and then the latest version 0.14.1. Exemplar support has been present since 0.11.0 so either of these versions should have been able to emit Exemplars. None did.

The first bit of troubleshooting I did was to make sure that the Prometheus client_java library was talking to me using the OpenMetrics 1.0 format. Although I’m very used to using curl to see and double check my metrics as the application is exporting them, we need to specify which format to use to get the OpenMetrics goodness and Exemplars.

$ curl -v -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8081/metrics

When using this curl command adding -v will show the headers the HTTP server sends back and will confirm if the returned format is OpenMetrics 1.0. The returned format was, indeed, OpenMetrics 1.0 but no Exemplars appeared.

I built Prometheus 2.30.1 on my laptop and made a quick test configuration to scrape my Java Spring Boot example application when it was running locally. Prometheus was definitely collecting metrics from my example application as expected. But no exemplars.

When running Java code with the OpenTelemetry Java Agent the Prometheus client_java library will recognize this and add trace_id and span_id Exemplars to Counter and Histogram metric types. The idea being that you don’t have to re-instrument your code to be able to link metrics to traces. It just works. There’s plenty of options to turn it off, but how do you turn it on? It’s on by default. So, to override this bit of logic my next step to troubleshoot this problem was to manually instrument Exemplars in my code. Per the documentation, this overrides any logic of when and if to add Exemplars to the output as they are manually instrumented. Cool!

So let’s make sure our Histogram object has Exemplars enabled:

prometheusHistogram = io.prometheus.client.Histogram.build()
  .namespace("custommetricsdemo")
  .name("histogram")
  .help("Test Prometheus Client Library latency histogram")
  .withExemplars()
  .register();

Next, be sure to manually observe values for the Exemplars:

// Histogram type: Testing manual / explicit Exemplar support.
prometheusHistogram.observeWithExemplar(sw.getTotalTimeSeconds(), "span_foo", "0xdeadbeef", "trace_bar", "DEADBEEF");

Expecting to see Exemplars I ran this code, setup Prometheus to scrape the metrics again, and used curl with the headers set correctly. Yet again, no Exemplars were in the output. No Exemplars were recorded by Prometheus. Here, I double checked that I had Prometheus running with --enable-feature=exemplar-storage to be absolutely sure that Prometheus would show the Exemplars. But as they were not present in the curl output I wasn’t surprised when this produced no changes.

Here, I was stumped. I had verified versions of the code and libraries. I had triple checked that I was using the API correctly and, of course, it was compiling. Again, Java is not my favorite language to work in. However, I can usually bang enough rocks together to come up with a working Java patch or micro-service. I’m still not completely sure what a Bean is, but that shouldn’t affect my ability to make this example work. Why was this so difficult when the Grafana folks demonstrate this all the time? (Ok, that’s easy. Most of Grafana’s stuff is written in Golang which had the earliest support for new Prometheus features.)

Finally, I reached out to the Prometheus Developers’ Mailing List. Actually, a couple times as I was trying to assemble this for a flashy demo and then later when I actually had the time to figure this out. Fabian Stäber and I ended up exchanging multiple emails on what was going on here. Fabian added this support to the client_java library and is definitely the expert I needed.

In my build.gradle file I had the implementation dependencies stated like the following to include the Prometheus client_java library and the OpenTelemetry library so that I could directly query the current Span object to add additional span attributes as part of the example.

implementation 'io.prometheus:simpleclient:0.14.1'
implementation 'io.prometheus:simpleclient_hotspot:0.14.1'
implementation 'io.prometheus:simpleclient_httpserver:0.14.1'
implementation platform "io.opentelemetry:opentelemetry-bom:${project['otel-agent.version']}"
implementation 'io.opentelemetry:opentelemetry-api'

Apparently, the Spring Boot framework implicitly includes an old version of io.prometheus:simpleclient_common:0.10.0 for any number of reasons. But I didn’t have that specific library stated in my dependencies to override. This older version does not have Exemplar support. (How does Java run with miss-matched library dependencies again?) At Fabian’s suggestion, I added the following line to my build.gradle file:

implementation 'io.prometheus:simpleclient_common:0.14.1'

And like magic, the Exemplars were present in the curl output and in my local Prometheus server!

Being that examples of this working in Java are hard to find due to how new this support is, I wanted to give a full and complete example of a working setup. My Java Spring Boot example application shows how one might wrap their Spring Boot application with the OpenTelemetry Agent and get those sweet Exemplars to start showing up. Examples for manual instrumentation of Exemplars and a Counter that is auto-instrumented are both there. Perhaps others might find this useful!

• https://github.com/jjneely/java-exemplar-example

In fact, if you remove the Gradle implementation dependency line above you can recreate the bug I was banging my head against.

How do we get back to Micrometer? That would be the ultimate goal, and although Micrometer doesn’t have direct support for Exemplars I was hoping that the auto-instrumentation features from the Prometheus libraries would at least provide the OpenTelemetry Span and Trace IDs. Unfortunately, this does not work. Sleuth 3.1.0 was very recently announced which includes underlying support for Exemplars in the Spring Boot world. This issue indicates that Micrometer will have Exemplar support via Sleuth in version 1.9 which shouldn’t be too far away as 1.8.1 is current as of this writing.

Interested in seeing a more complete Observability example with Java Spring Boot applications? Let me know in the comments!