Why everyone should track Kubernetes changes and top four ways to do so

updated on 12 April 2022

By Natan YellinRobusta.dev co-founder

Imagine the following: you plug in an electric kettle, a fuse blows, and the power goes out. You obviously suspect the kettle.

Now, imagine a different scenario: you work in an electrical store where shoppers can plug in any device they wish to try out. Suddenly the power goes out. You’d love to know which devices were just plugged in, right?

The above is a metaphor for Kubernetes. In most companies, people are constantly deploying changes and rolling out new versions. Unfortunately, there is no centralized view to see “what devices were just plugged in”.

ArgoCD and Gitops improve the situation, but don’t solve it

Here is why ArgoCD and Gitops aren't enough:

  1. You can deploy applications from multiple git repositories, so there still isn’t a single source of truth
  2. People can bypass gitops and make manual changes with kubectl edit

What you really need is a way to track all Kubernetes changes and correlate them with alerts.

Robusta to the rescue

Our team experienced the above problem as engineers. So when we built and open sourced Robusta, we wanted to solve this for once and for all.

Robusta is an event-triggered automations engine. Using Robusta you can subscribe to changes in a cluster (or multiple clusters) and publish that information to useful locations.

There are four variations to this, depending on where you send that data.

Solution 1: The Robusta UI

This is the easiest to setup. You run one Helm command to install Robusta and it's bundled Prometheus stack. Now you have a single dashboard with all changes and alerts across your clusters.

change-tracking4-min-ztd6h

There is no downside to trying this. If you don't like the UI, you can uninstall Robusta with a single Helm command. If you already have Prometheus installed, you can forward your existing alerts to Robusta by webhook.

The UI runs in our SaaS platform by default, but there is an on-prem version available as well. (Email natan@robusta.dev for details.) Or you can use the alternatives below which are completely open source and run entirely on-prem.

Solution 2: Grafana Dashboards

Let's improve our existing dashboards in Grafana by automatically adding annotations when applications are updated:

grafana-deployment-enrichment1-jnmit

This takes eight lines of YAML to configure. You define a custom playbook in Robusta's values.yaml file, which is used for configuring and installing Robusta with Helm:

customPlaybooks:
- triggers:
  - on_deployment_update: {}
  actions:
  - add_deployment_lines_to_grafana:
      grafana_api_key: '********'
      grafana_dashboard_uid: 09ec8aa1e996d6ffcd6817bbaff4db1b
      grafana_url: http://grafana.namespace.svc

This works by connecting the on_deployment_update trigger to the add_deployment_lines_to_grafana action.

Solution 3: Slack notifications

This is the same as above, but we're sending the result to Slack and not Grafana.

resource_babysitter1-82osm

Here is the YAML configuration for this:

customPlaybooks:
- triggers:
  - on_deployment_update: {}
  actions:
  - resource_babysitter: {}
  sinks:
  - slack

It works by connecting the on_deployment_update trigger to the resource_babysitter action and sending the result to the Slack sink. You could just as easily send the output to MS Teams, Telegram, DataDog, OpsGenie, or any other supported sink.

Solution 4: Reverse GitOps

This one is a little unusual, but we can send the same change data to a git repository.

git_change_audit1-79yme

Usually git repositories are used with GitOps as the source of truth for YAML files. However, git is also a convenient way to store audit data about what actually changed in your cluster. We can use git to audit every change in your cluster. 

Every time someone makes a change, whether it's an ad-hoc change or a planned deployment, it's written to a git repository at a path determined by the cluster name, namespace, and resource type.

Here is how we configure this Robusta automation:

triggers:
- on_kubernetes_any_resource_all_changes: {}
actions:
- git_change_audit:
    cluster_name: some_name
    git_key: '********'
    git_url: git@github.com:arikalon1/robusta-audit.git

Like the other examples, we're hooking up a trigger to an action. The trigger is more broad here, as it is on_kubernetes_any_resource_all_changes. The action is git_change_audit.

Summary

Hopefully this will help you setup change tracking on your Kubernetes cluster. Try Robusta's famous 97.68 second installation and start tracking changes immediately. While you're at it, level-up your Prometheus alerts.

Feeling like a pro? Try writing your own playbook action in Python to send change data to another location.

Sharing and Subscribing

Share on Twitter, LinkedIn, Reddit, or HN

Enjoyed reading this? Subscribe to blog posts and updates by email.

Read more