Kubernetes administration, choices and challenges

Starks, Lannisters, and Kubernetes administration

General discussion around challenges and choices confronting Kubernetes administrators

Overview

“It depends”. Even the most seasoned Kubernetes administrators probably pull this answer out to many questions around Kubernetes cluster management. Although it may seem a bit vague, in many cases it’s spot on! As Kubernetes becomes a preferred choice for running a variety of workloads, Kubernetes administrators need to get comfortable using “it depends” to answer many questions such as:

• What environment should I pick to host the cluster?
• Which operating system should I use for the nodes? And what version? How should I perform OS hardening?
• Which version of Kubernetes do I install to begin with? Should I look for a security hardened CIS compliant distro?
• What about runtime security? Do I need it day 1? If so, which one do I choose?
• Which CNI plugin do I use?
• Do I need to integrate a storage solution? If so, which one?
• Do I need a central Logging & Monitoring stack? ELK? Datadog? New shiny tool I saw at KubeCon last month?

There can be many other questions where the answer boils down to… you guessed it… “it depends”. “Great, but WHAT does it depend on?”, you might ask. The simple answer is that it depends primarily on the needs of the workloads that will be running on the cluster and the needs of the organization, much like someone looking to buy a house would make a decision primarily based on current needs and activities of their family. On top of the questions above, of course, there are other concerns like cost, manageability, upgradability etc. A Kubernetes administrator needs to take all these factors into account to make sure their clusters fit their exact needs.

Where do I build it? How do I build it?

Let’s spend a little more time with the house hunting analogy. Assume we have two families, the Starks and the Lannisters. Both arrive in Westeros and are looking for a new house. The Starks have two kids, ages five and nine, whereas the Lannisters are recently married with no children. The Starks look for a home with three bedrooms, close to good schools, and with a big backyard. The Lannisters are fine getting a smaller house with a small backyard, but they would like to be closer to entertainment hot spots. The Starks buy a villa in the Winterfell neighborhood whereas the Lannisters buy a townhouse in King’s Landing. Once they move into their respective homes, both families look for options to secure their homes. The Starks install a comprehensive security system with several sensors and video surveillance. The Lannisters’ town home is in a gated community and they don’t feel the need to do anything additional about security for the time being. Over time, each family customizes their respective home to their changing needs. The Starks discover one of their kids is really interested in music, and they build her a music room. The Lannisters on the other hand, are getting over the honeymoon phase and decide it’s time to get back in shape, so they build a little gym in their garage.

As you can see, the Starks and Lannisters had different needs and requirements to begin with. Over time they shared some general concern around security, which they addressed differently based on their situations and their needs. There were some concerns that were specific to one family but did not apply to the other. Their decisions ensured that the ever changing needs of their families were always accommodated and they were comfortable in their homes. If the Starks were forced to live in King’s Landing they would feel suffocated and the Lannisters would be miserable in Winterfell.

Similarly, Kubernetes cluster administrators should have the freedom to deploy clusters that best suit the current and future needs of the applications that will run on them while taking other factors like cost, manageability, serviceability in mind. For example, GPU support for AI workloads may influence choice of nodes to deploy onto. The choice of OS might be dictated by a dependency on Kernel modules that application workloads need, or there may be a standard OS mandated by the organization. Kubernetes version may depend on the environment — Production/Dev/Integration. While production workloads need to run on stable version of K8S, for dev clusters, developers might ask for a newer version to experiment with new features. You might want to choose between calico, flannel or another CNI based on the richness and granularity of security controls they provide. For load balancers, you might pick a cloud-based offering when deploying to public clouds but choose NginX for private setups. When it comes to integrating add-ons for Monitoring, Logging, Runtime Security, administrators have even more choice to pick a solution to address their needs. Moreover, the application workloads mature over time and the needs might change.

Conclusion

So, what’s the “right” way to deploy Kubernetes? Kubernetes administrators need to juggle the different requirements and needs of their workloads and organizations to find the best (possibly more than one) Kubernetes cluster profiles to offer to developers. In other words, “it depends”.