Prometheus is great for alerting, but where do those alerts come from? How is Prometheus configured? Specifically, how is it configured when running on Kubernetes?
Before we set up Prometheus, you need to know about two types of Prometheus rules. They are
- Alerting rules - Alerts are written in PromQL. They evaluate one or more expressions and fire alerts based on the result.
- Recording rules - These are precomputed expressions that can be queried without having to execute the original expression every time.
We will see how to configure alerts for Prometheus instances installed using the prometheus-community/prometheus Helm chart. Since we are using a Helm chart, we will configure rules using the Helm values file.
What is Prometheus.yml?
prometheus.yml is the global configuration file for Prometheus. Back in the old days, before Kubernetes, it was typically located on servers at /etc/config/prometheus.yml.
Where is prometheus.yml on Kubernetes?
It is located at /etc/config/prometheus.yml in the prometheus-server pod. But you probably shouldn’t go ahead and edit that The prometheus-server pod does not have persistent storage. When the pod goes down, your updated configuration will be lost.