Why controlling your Kubernetes distribution can mean controlling your destiny

As the Kubernetes landscape grows and matures, we see a rise in Kubernetes distributions. Similar to how Linux distributions take upstream, open source code, test it, package it and make it easier to consume, Kubernetes vendors are doing similar things with both the Kubernetes orchestrator itself, and software that can enhance the experience of using Kubernetes. This selection from upstream projects can be daunting; the CNCF landscape is huge, with lots of overlapping projects, including Kubernetes installers and managers, ingress options, service mesh options, and dozens of other projects in various stages of maturity.

In addition to making choices about which software to ship, distributions also serve to help meet basic expectations - Linux distributions, for example, typically default to the Bourne Again Shell, while FreeBSD defaults to tcsh, OpenBSD to ksh, and these decisions have an outsized effect on the user experience. When a Kubernetes distribution makes decisions about whether to ship with a particular logging system or ingress controller, this also has an outsized effect on the user experience.

The role of the Operating System

As a Kubernetes adopter, I look at my operating system differently than I did when I ran applications on bare metal. Kubernetes, it could be argued, pushes some of the operating system distribution to the background. For example, I don't want to take a database server package from my operating system distribution, I want to take it from my Kubernetes distribution, or from upstream, or perhaps from a third-party database vendor like Crunchy or Vitess.

Conversely, I want my operating system to include useful packages like a shell and an SSH server, so I can set up Kubernetes and do some basic troubleshooting of the node; I may also rely on particular configurations on the bare metal operating system for things like custom networking or storage hardware.

From the bottom of the stack, up

When building a greenfield cluster, you can potentially save time and effort by relying on the distribution’s “opinion”, oftentimes not limited to the Kubernetes distribution, but including the operating system and add-on software. This can potentially include commercial support, so that you have a vendor’s phone number to call or a Slack channel to ping if you get in over your head. This reduces the learning curve, and can potentially avoid tech debt.

On the other hand, when decisions are made for you, you may not be able to address some of your more niche or future requirements. In some environments, we may need your cluster to connect with a proprietary container storage service, like Netapp or Silk. You may need special networking configurations, like bonded interfaces, or multicast. When you have these dependencies in the OS, will your distribution give you the flexibility to include them in the stack?

Making the choices yourself can also help avoid lock-in; for example, you can pack up and move your ingress objects between two clusters if they both use ingress-nginx, but you can't move an OpenShift route object to a non-OpenShift cluster.

With a heavily opinionated Kubernetes stack, you may duplicate effort and be left with overhead. If your Kubernetes distribution is tied tightly to an included Elasticsearch stack, and your organization prefers Splunk, you may end up having to run both, and figure out on your own how to federate data from one logging system to another.

Our vision for Kubernetes without boundaries

Ultimately, the only right opinion is the one belonging to you and your stakeholders. As your Kubernetes practice grows and matures, you will identify specific needs and develop your own approach to address your unique requirements. You will want a Kubernetes distribution that allows the flexibility to change out layers of your infrastructure and add-on app services. You will also want the ability to have domain-specific experts select and tune layers of your Kubernetes stack, assert the opinions of your organization, and provide those to developers to work with. Our vision at Spectro Cloud is that Kubernetes should serve its users - IT and diverse dev teams - and that means not having to compromise on technology choices. We allow you to select and customize layers, and enforce policy-driven automation for day 2 operations of your new and existing deployments. If you want to see Palette in action, please reach out to us for a demo.