Many people think you need CPU limits on Kubernetes but this isn't true. In most cases, Kubernetes CPU limits do more harm than help.
I will explain why CPU limits are harmful with three analogies between CPU starved pods and thirsty explorers lost in a desert. We will call our intrepid explorers Marcus and Teresa.
In our stories, CPU will be water and CPU starvation will be death. Like CPU, water in our story will be a renewable resource. In simple terms, if you have 100% CPU usage at a given minute, that doesn't "use up" the CPU for the next minute. CPU renews itself from moment to moment.
Marcus and Teresa are travelling in the desert. They have a magical water bottle that produces 3 liters a day. Each person needs 1 liter a day to survive.
Story 1 - without limits, without requests: Marcus is greedy so he drinks all the water before Teresa can drink any. Teresa dies of thirst. This is because there were no limits or requests. Marcus was able to drink all the water, cause CPU starvation, and Teresa was CPU throttled.
Story 2 - with limits, with or without requests: Teresa gets very ill one day and needs some extra water. Marcus drinks his one liter and there are two liters left over. Teresa drinks one liter and now there is one liter remaining. Marcus wont let Teresa drink it because her limit is 1 liter per day so she dies of thirst. This is what happens when you have CPU limits. Resources are available but you aren't allowed to use them.
Story 3 - without limits, with requests: Marcus gets very ill and needs extra water one day. He tries to drink the entire bottle but is stopped when only 1 liter remains in the bottle. This is saved for Teresa because she needs 1 liter a day. She drinks her 1 liter. Nothing remains. They both live. This is what happens when you have no CPU limits but you do have requests. All is good.
The above stories are surprisingly precise analogies for why CPU limits are considered harmful.
Don't like analogies? I wrote a more technical explanation of this when explaining the Prometheus CPU throttling alert, CPUThrottlingHigh.
Many people think you need limits to prevent one pod from interfering with another pod. This is not true! You can remove Kubernetes CPU limits and still prevent a CPU hungry pod from causing CPU starvation! The trick is to just define CPU requests.
Here is a snippet from the original Kubernetes documentation:
The important part is highlighted. Pods always get the CPU requested by their CPU request! (They can take advantage of excess CPU too if they have no limit.)
Disable your CPU limits! If you give all your K8s pods accurate CPU requests, then no-one can throttle them because CPU is reserved for them if they need it. This has nothing to do with limits.
Lets summarize the difference between CPU limits and requests in a table:
I'm assuming for simplicity that either all pods have CPU limits or none have them. The same is true for CPU requests. This makes everything simpler to reason about.
I've started a video series to elaborate on the concepts in this post, as well as other misconceptions. Check it out.
Tim Hockin (one of the original Kubernetes maintainers at Google) has recommended the same for years.
Everything in this post is about CPU and not memory. Memory is different because it is non-compressible - once you give memory you can't take it away without killing the process. We've covered the best practices for Kubernetes memory limits here. In short, our bottom line recommendation is:
This post focused on K8s CPU limits, but there are actually other reasons why CPU throttling can happen. This is a good opportunity to plug Robusta.dev, which tells you the root cause of Prometheus alerts on Kubernetes. It analyzes your alerts and adds missing context. This is true for all alerts, including Kubernetes high CPU throttling alerts.
This is all about CPU limits, but if you've read until here you probably use Kubernetes!
Do you know the four ways to track changes to Kubernetes resources?
If you like this post, please star us on GitHub and help us keep producing content like this.