Engineering Software, Linux, and Observability. The website of Jack Neely.    

Helm Chart Prometheus Rules

The best thing about using the Prometheus Operator to manage Prometheus in Kubernetes is the CRDs. Alert rules can be managed directly by the application Helm Charts, FluxCD, or your Cloud Native pipeline of choice. For me, for now, that is Helm Charts. The only problem is that both Helm and the Prometheus Alert Rules use Golang Text Templates. The question is: how does one keep the Helm Chart templates from getting ugly and super complex?

The best way I’ve found is to keep the Prometheus Rules in a configuration file that the Helm Chart template will directly include. For example, I include alerts with a Helm Chart that deploys a Kafka exporter.

{{- if .Values.prometheus.rules.enabled }}
kind: PrometheusRule
  name: {{ include "kafka-exporter.fullname" . }}
  {{- if .Values.prometheus.serviceMonitor.namespace }}
  namespace: {{ .Values.prometheus.serviceMonitor.namespace }}
  {{- end }}
  labels: {{ include "" . }} {{ include "kafka-exporter.chart" . }} {{ .Release.Name }} {{ .Release.Service }}
    {{- if .Values.labels -}}
    {{ .Values.labels | toYaml | nindent 4 -}}
    {{- end }}
    {{- if .Values.prometheus.rules.additionalLabels }}
{{ toYaml .Values.prometheus.rules.additionalLabels | indent 4 -}}
    {{- end }}
  {{- range $path, $_ := .Files.Glob "rules/*" }}
    - name: {{ $.Release.Namespace }}/{{ $.Release.Name }}/{{ $path }}
      {{- $.Files.Get $path | nindent 8 }}
  {{- end }}
{{- end }}

This creates a Rule Group per included file and ensures that each group has a unique name. The included files are found in a rules/ directory that is beside (and not in) the templates/ directory. These files have content that looks like the following.

- alert: KafkaUnderReplicatedPartitions
  expr: sum without (partition) (kafka_topic_partition_under_replicated_partition) > 0
  for: 10m
    severity: critical
    summary: "{{ $externalLabels.cluster }} kafka service has under-replicated partitions."
    description: "{{ $externalLabels.cluster }} kafka service has under-replicated partitions afecting topic {{ $labels.topic }}."
    runbook_url: "<Wiki Link here>"

There is some Helm templating that this setup prevents, but in many cases this may be quite handy.

 Previous  Up  Next

comments powered by Disqus