August 21, 2022

Kubernetes is the POSIX of the cloud

By Natan Yellin, Robusta.dev co-founder

What is Kubernetes anyway?

Kubernetes is a cloud operating system!

I know, people usually call Kubernetes a "container orchestrator". But what does that even mean? Why does the world need a container orchestrator to begin with?

To understand Kubernetes we have to go back to the basics and start from scratch.

What do operating systems do?

Lets quote MIT's course on operating systems:

What do applications want from an O/S?

  • Abstract the hardware for convenience and portability
  • Multiplex the hardware among multiple applications
  • Isolate applications to contain bugs
  • Allow sharing among applications

On a single machine, these roles are well understood. They are performed by Windows, Linux, or some other operating system, often a POSIX compatible one.

But what does an operating system look like in the cloud era?

What are the responsibilities of a cloud operating system?

In a microservices world, applications no longer run on single machines. In some abstract sense, the new machine is an entire data center. If you look at a datacenter from afar, what needs does it have? What must be done to operate it?

The datacenter as a whole has needs that are surprisingly similar to the needs of each individual machine. For example, in a datacenter that runs Docker containers, we must schedule those containers to individual machines. On each individual machine the processes in those containers must then be scheduled to individual CPUs.

In other words, datacenters, machines, and operating systems are all part of one large self-interacting system. Just like a cauliflower under a microscope has a pattern that resembles the cauliflower as a whole, datacenters and machines have a fractal resemblance to one another.

What does Kubernetes do?

Now we can speak about Kubernetes and what it does.

Kubernetes handles the same responsibilities on the layer of a datacenter that an operating system handles for an individual machine.

These core responsibilities of Kubernetes include process isolation, hardware abstraction, concurrency management, storage management, security, permission handling, and so on.

Here is the analogy between Kubernetes and an operating system in table form.

Obviously Kubernetes isn't replacing the operating system here, but it is providing similar APIs at a higher abstraction layer - that of the datacenter, not the individual machine.

Kubernetes is the POSIX of the cloud

Kubernetes is POSIX not UNIX, because there isn’t one Kubernetes. It's an API, not an implementation.

You can run Minikube, KIND, K3s, OpenShift, GKE, EKS, or AKS. They all support the same YAML specs, but the implementation details can differ.

Like every operating system, Kubernetes supports different hardware using drivers. You can apply the same LoadBalancer resource to different Kubernetes distributions. On EKS it will be implemented one way, on GKE another. Same with CSI (storage) drivers.

Operating systems also define executable formats. In Kubernetes case, this is Docker containers.

Making Kubernetes Boring

The best thing about Linux is that it's boring.

You can develop and use Linux without being a kernel expert. This is hugely important.

We're not yet there with Kubernetes. Some people say Kubernetes sucks because of its complexity. We say Kubernetes is complex because you want complex things. But can we simplify Kubernetes despite that?

If Kubernetes is POSIX then we're missing both glibc (the C standard library) and the rest of usermode.

Kubernetes adoption is everywhere, but Kubernetes itself needs to become simpler.

PaaS vs Kubernetes

How can we make Kubernetes simpler?

One approach is to abstract away the underlying details and hiding them from you via a high level Platform as a Service or PaaS.

This PaaS approach has been unsuccessful to date. Simpler isn't always better.

Instead, it is time to take an operating system like approach. This means not abstracting away the operating system, but rather providing complementary use mode tools to manage it.

A good example of this is how we simplify multi-cluster Kubernetes management with Robusta.dev. We don't replace the way you already create clusters or try to hide the underlying clusters from you.

No one succesful has made POSIX simpler by removing APIs or forcing you to use their hardware. So that approach wont work for a Kubernetes PaaS either.

The Kubernetes world needs more tools like Robusta.dev which take what is already there, with all it's complexity, and make it easy to get that complexity under control without hiding it. Our motto is that we make monitoring Kubernetes simple. We don't try to make Kubernetes itself simpler by removing features from it.

What a Kubernetes usermode should include

I recently asked on Twitter what the hardest part of Kubernetes is. The anwers were all about monitoring, cluster management, troubleshooting, and observability.

Doesn't Prometheus do all that? Well, sort of.  Prometheus does a great job gathering metrics, but that's only one part of a holistic Kubernetes monitoring solution.

Just like traditional UNIX commands can be combined together into bash scripts, we need ways to combine together all the parts of Kubernetes monitoring and observability into one unified and coherent platform. Otherwise each additional component adds complexity, not reduces it.

Spread the love by sharing on LinkedIn or Twitter, and HN.

Subscribe to receive articles directly in your inbox