This is a guest blog post by Audrey Turco, Head of DevOps at JOOR, a B2B fashion wholesaler, with offices in New York and Spain. Audrey shares the five Helm add-ons her team at JOOR loves most.
After years of handling messy, inconsistent yaml files for Kubernetes, I made the switch to Helm five plus years ago, and I haven’t looked back since. Everyone who interacts with Kubernetes should know about Helm by now. It’s a tool that automates the creation, packaging, configuration and deployment of your Kubernetes based applications. By combining your configuration files into a single reusable package, it makes deployments more consistent, repeatable and reliable while removing a lot of human error.
But that's just scratching the surface of what Helm can do, and with the addition of a few simple plugins and tools (things that I use every day), Helm can become a really powerful application for managing our deployment resources.
Here are those tools.
1. Helmfile - for keeping Helm DRY.
As useful as Helm is, it becomes a lot to manage if you have more than one application or environment, each with their own cycle of new versions and values. This is especially painful if there are multiple versions of the same application deployed at once.
A key requirement is the ability to reproduce all the current state on demand, as well as being able to quickly see what that desired state is. You essentially want a table of contents for all your Helm chart releases.
Helmfile gives us that with helmfile.yaml. It supports a multi-environment directory structure and the ability to call commands on all children helmfiles. It lets you use release templates to avoid re-writing similar flows because of differing secrets or values.
Helmfile has become an essential part of any CD efforts I am involved in. Developed by roboll, Helmfile wraps Helm in a declarative spec that allows you to bundle multiple charts together and create a comprehensive deployment artifact.
By separating your environment-specific values from your charts, and performing a diff of your existing deployment and only applying changes, Helmfile helps us stay consistent with GitOps and DRY (Don't Repeat Yourself) principles. Helmfile’s ‘apply’ command allows us to apply resources only when there are changes.
One of the most interesting Helmfile features is the ability to use templatization for Helm chart values (a feature that Helm lacks any meaningful support for). Similar to Helm templates, Helmfile supports values files appended with `.gotmpl` to be parsed as yaml-rendering golang templates, with included support for Sprig templating functions. Used in conjunction with Helm values, this gives you quite a bit of flexibility within values files. Helmfile values files also support a merged inheritance, meaning all common values can be placed in one values file, with only environment specific values necessary per environment.
After years of trial and error, I have found the best methodology is to create one common helm chart that all of your services can consume, with specific resources either enabled or disabled by service and/or environment. This makes it easy to add new functionality to an existing service merely in the values files, without changing your chart. This allows me to update deprecated endpoints in one place, instead of across multiple files or repos. This allows for more dynamic CD, as all of our images can be set with –set docker.image.tag=1234 to update a container reference, as that reference is referenced one time in one place in our charts. Then we can start using the same deployment scripts in multiple repos if needed or desired. We can also use base env files to inject common environmental variables into all of our pods, or to set the number of replicas without rewriting the wheel in every service yaml.
2. ChartMuseum - for storing Helm Charts
If you need a place to store helm charts, ChartMuseum can be handy. It is a repository designed to work with Google Cloud Storage, Amazon s3, Microsoft Azure blob Storage, Alibaba Cloud OSS Storage, Openstack Object Storage, Oracle Cloud Infrastructure Storage, Baidu Cloud BOS Storage, DigitalOcean Spaces, Minio and etcd.
One of the benefits of ChartMuseum is a deployable UI, which can make it easy for developers to see Helm packages.
If you utilize ChartMuseum, you're going to want a simple connection to storage, and there are two popular options: the helm cm-push plugin, which will push your charts to storage with a simple command, and the Helm s3 plugin, which provides s3 support. It allows you to have a private or public Helm chart repository hosted on Amazon s3.
3. Helm Diff - Because everybody needs to run diff
No matter where you are coming from in your dev or engineering role, you know the value of a good diff command and use it frequently. Well, Helm would not be complete without its own. And that's where we get the Helm Diff plugin. While simple, it offers a preview of Helm upgrades as a colored diff. This allows us to avoid unnecessary mistakes.
4. Helm Test/Unittest - For local and in cluster unit testing
With everything we process through Helm, we need a way to keep our Helm charts consistent and robust.
One of the best tools for the job is Helm Unittest. It allows you to write tests in pure yaml, render them locally, and add nothing to your cluster.
In addition to Helm unittest, there is the builtin "helm test". Helm test is great for testing a release with a pod defined in the chart.
I have used this to test:
- Ingress Health
- Database connectivity
- Access to object storage
- KMS Key Access
With the test command, you can specify a pod template, and certain commands to run (like pre-install jobs) that can allow you to quickly download a test file from s3 to your pod, or ensure that the database credentials in use are correct.
These tests can also be used for service e2e testing.
Run these commands with the `helm test` command or the `helmfile test` command if using helmfile wrapper.
5. Helm Secrets - For Deployment Secret Management
It can be challenging when we need to add secrets to our deployments and want to utilize the best security practices. One of the solutions to the above problem is the Helm secrets plugin. It utilizes Mozilla SOPS to encrypt and decrypt secrets. This allows you to keep your application config complete and centralized,keep your encrypted secrets in a private git repository, enable version control and the principle of least privilege on those secrets, all while still following the ideals of enablement. I typically set up my secrets files with developer access to non production secrets files, and heavily restricted access to upper environment secrets. You can have your secret values follow the same merged inheritance order as helm values, but they must be decrypted before editing. Note that secrets are plain yaml and not go templates.
It is worthwhile to add that these plugins can be utilized with Helm or Helmfile, which allows you to pass in any arguments you’d like to Helm itself. Helm offers an easy way to list all available installed plugins, and there are a number of resources that have collated a fairly complete list of tools available.
Closing Notes
While these tools are helpful, ultimately the quality of your Helm charts will determine the success of your CD methodologies and these plugins.
Helm has a fairly comprehensive list of tips and tricks, as well as best practices in their docs. One method I employ is running a Helm lint and Helm diff on PR creation, with the output printed to the pull request itself, for oversight. Utilizing quality gates and version control can also help us keep aligned with best practices for flexible, extensible charts.
In any event, good luck and happy helming!
Audrey Turco is the Head of DevOps at JOOR and has been passionate about Kubernetes for almost a decade. Follow Audrey on LinkedIn for more Kubernetes content.
