For the Love of God, Stop Using CPU Limits on Kubernetes (Updated)

By Natan Yellin, Robusta.dev co-founder

CPU limits on Kubernetes are an antipattern

Many people think you need CPU limits on Kubernetes but this isn't true. In most cases, Kubernetes CPU limits do more harm than help.

I will explain why CPU limits are harmful with three analogies between CPU starved pods and thirsty explorers lost in a desert. We will call our intrepid explorers Marcus and Teresa.

In our stories, CPU will be water and CPU starvation will be death. Like CPU, water in our story will be a renewable resource. In simple terms, if you have 100% CPU usage at a given minute, that doesn't "use up" the CPU for the next minute. CPU renews itself from moment to moment.

Three colorful analogies about Kubernetes CPU Limits

Marcus and Teresa are travelling in the desert. They have a magical water bottle that produces 3 liters a day. Each person needs 1 liter a day to survive.

Story 1 - without limits, without requests: Marcus is greedy so he drinks all the water before Teresa can drink any. Teresa dies of thirst. This is because there were no limits or requests. Marcus was able to drink all the water, cause CPU starvation, and Teresa was CPU throttled.

Story 2 - with limits, with or without requests: Teresa gets very ill one day and needs some extra water. Marcus drinks his one liter and there are two liters left over. Teresa drinks one liter and now there is one liter remaining. Marcus wont let Teresa drink it because her limit is 1 liter per day so she dies of thirst. This is what happens when you have CPU limits. Resources are available but you aren't allowed to use them.

Story 3 - without limits, with requests: Marcus gets very ill and needs extra water one day. He tries to drink the entire bottle but is stopped when only 1 liter remains in the bottle. This is saved for Teresa because she needs 1 liter a day. She drinks her 1 liter. Nothing remains. They both live. This is what happens when you have no CPU limits but you do have requests. All is good.

The above stories are surprisingly precise analogies for why CPU limits are considered harmful.

Don't like analogies? I wrote a more technical explanation of this when explaining the Prometheus CPU throttling alert, CPUThrottlingHigh.

Preventing CPU throttling and insufficient CPU without limits

Many people think you need limits to prevent one pod from interfering with another pod. This is not true! You can remove Kubernetes CPU limits and still prevent a CPU hungry pod from causing CPU starvation! The trick is to just define CPU requests.

Here is a snippet from the original Kubernetes documentation:

The important part is highlighted. Pods always get the CPU requested by their CPU request! (They can take advantage of excess CPU too if they have no limit.)

Disable your CPU limits! If you give all your K8s pods accurate CPU requests, then no-one can throttle them because CPU is reserved for them if they need it. This has nothing to do with limits.

Kubernetes Requests vs Limits

Lets summarize the difference between CPU limits and requests in a table:

I'm assuming for simplicity that either all pods have CPU limits or none have them. The same is true for CPU requests. This makes everything simpler to reason about.

Best practices for CPU limits and requests on Kubernetes

Lets summarize:

Use CPU requests for everything (if you need help setting them, see KRR)
Make sure they are accurate
Do not use CPU limits.

Tim Hockin (one of the original Kubernetes maintainers at Google) has recommended the same for years.

What about memory limits and requests?

Everything in this post is about CPU and not memory. Memory is different because it is non-compressible - once you give memory you can't take it away without killing the process. We've covered the best practices for Kubernetes memory limits here. In short, our bottom line recommendation is:

Always use memory limits
Always use memory requests
Always set your memory requests equal to your limitsS

But how can I determine what CPU requests I actually need?

We built an open source tool so you can determine CPU and memory requests based on historical Prometheus data.

Kubernetes Resource Recommender - determine CPU and memory requests based on historical data

Check it out on GitHub. For questions, message me on LinkedIn or open a GitHub issue.

Natan Yellin

,

CEO

Natan has been writing software for over 15 years. He regularly posts about all things Kubernetes on LinkedIn.‍

Never miss a blog post

As a DevOps or SRE, you likely monitor dozens of applications and services with the goal of keeping them up and running all the time. To do so effectively, you probably rely on alerts from Prometheus and similar systems to get notifications about important issues. But all too often those alerts can be noisy and instead of trusting those alerts to tell you something important, you find yourself ignoring them. When this happens, you and your team can miss out on critical alerts.

New In September 2024

This month we released composable dashboards in the UI and other highly-requested features. Join us, on October 01st - We're doing a webinar with the author of the most influential engineering-leadership book in the past decade. Don't miss it!