Helm Chart Prometheus Rules
The best thing about using the Prometheus Operator to manage Prometheus in Kubernetes is the CRDs. Alert rules can be managed directly by the application Helm Charts, FluxCD, or your Cloud Native pipeline of choice. For me, for now, that is Helm Charts. The only problem is that both Helm and the Prometheus Alert Rules use Golang Text Templates. The question is: how does one keep the Helm Chart templates from getting ugly and super complex?
The best way I’ve found is to keep the Prometheus Rules in a configuration file that the Helm Chart template will directly include. For example, I include alerts with a Helm Chart that deploys a Kafka exporter.
{{- if .Values.prometheus.rules.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: {{ include "kafka-exporter.fullname" . }}
{{- if .Values.prometheus.serviceMonitor.namespace }}
namespace: {{ .Values.prometheus.serviceMonitor.namespace }}
{{- end }}
labels:
app.kubernetes.io/name: {{ include "kafka-exporter.name" . }}
helm.sh/chart: {{ include "kafka-exporter.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.labels -}}
{{ .Values.labels | toYaml | nindent 4 -}}
{{- end }}
{{- if .Values.prometheus.rules.additionalLabels }}
{{ toYaml .Values.prometheus.rules.additionalLabels | indent 4 -}}
{{- end }}
spec:
groups:
{{- range $path, $_ := .Files.Glob "rules/*" }}
- name: {{ $.Release.Namespace }}/{{ $.Release.Name }}/{{ $path }}
rules:
{{- $.Files.Get $path | nindent 8 }}
{{- end }}
{{- end }}This creates a Rule Group per included file and ensures that each group has a
unique name. The included files are found in a rules/ directory that is
beside (and not in) the templates/ directory. These files have content that
looks like the following.
- alert: KafkaUnderReplicatedPartitions
expr: sum without (partition) (kafka_topic_partition_under_replicated_partition) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $externalLabels.cluster }} kafka service has under-replicated partitions."
description: "{{ $externalLabels.cluster }} kafka service has under-replicated partitions afecting topic {{ $labels.topic }}."
runbook_url: "<Wiki Link here>"There is some Helm templating that this setup prevents, but in many cases this may be quite handy.