Guest blog post by Luvpreet Singh. Luvpreet is a Platform Engineer at GoHighLevel where he is responsible for the monitoring of cloud infrastructure with Prometheus and Grafana.
Millions of developers write PromQL queries and build custom Grafana dashboards for Kubernetes. And everyone uses the same underlying metrics from node-exporter, kubelet, and kube-state-metrics. Unfortunately, there are some little-known pitfalls that many people don't know how to avoid.
"When people tell me they've learned from experience, I tell them the trick is to learn from other people's experience." — Warren Buffett
In this post, I'll troubleshoot a simple-looking Prometheus query for Kubernetes that is supposed to return a pod's memory usage:
container_memory_working_set_bytes{pod="agency-dashboard-api-89b7f557c-xd4l7"}Unfortunately, this simple query is wrong. Let's find out why — and how to avoid making similar mistakes in your queries.
Mistake #1: Duplicate Series
The first step to debugging any PromQL query is to verify that you have the exact number of time series in the results that you expect. In this case, I expect a single result, given that I'm filtering on a single pod.
Instead, I get two results. Initially, I suspected something might be wrong with my monitoring setup — maybe I had multiple jobs scraping the same data. I added the job label but the query still resulted in two time series for the same pod.
Problem #2: Grouping/Sum Mistakes
Since the query was returning two almost identical time series, the genius in me decided to sum the metrics and divide the result by two.
sum(container_memory_working_set_bytes{pod="my-pod-123", job="kubelet"}) by (pod) / 2But when I cross-checked the results with the data from my GKE console, I found that the query produced varying results according to time.
The lesson? Don't de-dupe metrics by averaging them — not until you understand in depth how they're calculated and why there's a duplicate in the first place.
Problem #3: Unexpected Cardinality
Let's go back to our duplicate time series and take a closer look. How are the two time series different from one another?
The answer lies in the container label. This label exists in one time series but is missing from the other. To understand why, we need to know what a pause container is.
When a pod is created in Kubernetes, the number of containers created is always more than the number specified in the pod manifest. The pause container has all the network configurations of the pod. The duplicate time series now makes sense — one time series per container, including the empty pause container.
I then fixed the query and finally got the correct metrics:
container_memory_working_set_bytes{pod="agency-dashboard-api-89b7f557c-xd4l7", job="kubelet", container!=""}The lesson? Always check your labels. Make sure you understand which labels can change, which are hard-coded, and which are optional.
Summary
I hope that seeing a simple but incorrect PromQL query will help you avoid making similar mistakes with your own metrics. PromQL is simple, but that simplicity is deceptive. For every query, make sure you know exactly what you're measuring. It's easy for extra data to slip in and throw off your aggregations.
Good luck and happy monitoring!