For the love of god, stop using CPU limits on Kubernetes

updated on 14 April 2022

By Natan co-founder

CPU Limits on Kubernetes are an antipattern. Let me explain why with a few stories about thirsty explorers lost in a desert. We will call them Marcus and Teresa.

Marcus and Teresa have a magical water bottle that produces 3 liters a day. Each person needs 1 liter a day to survive.

Story 1: Marcus is greedy so he drinks all the water before Teresa can drink any. Teresa dies of thirst. This is because there were no CPU limits and no CPU requests. Markus caused CPU starvation and Teresa was throttled.

Story 2: Teresa gets very ill one day and needs some extra water. Markus drinks his one liter and there are two liters left over. Teresa drinks one liter and now there is one liter remaining. Markus wont let Teresa drink it because her limit is 1 liter per day so she dies of thirst. This is what happens when you have CPU limits. Resources are available but you aren't allowed to use them.

Story 3: Marcus gets very ill and needs extra water one day. He tries to drink the entire bottle but is stopped when only 1 liter remains in the bottle. This is saved for Teresa because she needs 1 liter a day. She drinks her 1 liter. Nothing remains. They both live. This is what happens when you have no CPU limits but you do have requests. All is good. 

The end.

What most people get wrong

Many people don't understand why CPU requests are sufficient to protect Teresa in story 3. Here is a snippet from the original Kubernetes documentation:

Screen Shot 2022-04-13 at 22.35.35-al4ei

Don't like analogies? I wrote a technical explanation of this when explaining CPUThrottlingHigh,

How CPU Requests really work

Hint: they're not just used for scheduling!

Relevant Runbook Automation

I wrote an automated runbook for Prometheus+Robusta that looks at this when determining how to fix CPUThrottlingHigh. Source code here.

Caveats and closing notes

  1. Everything here applies to CPU not memory. For memory, set limits equal to your requests.
  2. Tim Hockin (one of the original Kubernetes maintainers at Google) has recommended the same for years.

More Kubernetes reading

Four ways to track Kubernetes changes.

change-tracking-with-robusta (1)-7bzi4

Robusta homepage

Read more