TLDR: On some cloud providers, you get half the CPU you expect due to burstable nodes. Without burstable nodes, overhead is improved but still significant.
I recently deployed a pod requesting 1 CPU to a shiny new Kubernetes cluster. The cluster had plenty of empty nodes with 2 CPUs each. Yet my Pod was stuck in Pending and couldn't be scheduled.
How can a node with 2 CPUs not fit a pod requesting 1 CPU?
In this post, we'll understand node overhead on Kubernetes. Then we'll benchmark cloud providers and find the worst offenders.
How much CPU is reserved on Kubernetes Nodes?
Good question. According the docs, every node reserves CPU and Memory for itself. The reserved resources are split into three parts: kube-reserved, system-reserved, and eviction-threshold.
After you subtract reserved resources from total CPU/Memory, what’s left over for pods is known as Node Allocatable.
So the real question about GKE (and other providers) is just how big Node Allocatable is, relative to the node’s total resources?
To find out, I ran kubectl describe on my GKE node. The node's capacity is 2 CPU, but node allocatable is 0.94 CPU! GKE is taking more than half the CPU for itself.