Kubernetes security best practices: 5 easy ways to cut risk

Securing Kubernetes with a cloud-native approach

Security is a hot topic in the world of Kubernetes. As more production workloads and data move to live in Kubernetes clusters, the more important it is to protect your infrastructure. And it seems like every month there’s a scary new report about security issues, threats and vulnerabilities.

Unfortunately, even if you’re a seasoned IT professional, you can’t necessarily apply your familiar security practices from classical data centers and clouds directly to your Kubernetes infrastructure.

For example, Kubernetes networking is much more complex, so default firewall policies must be adjusted in order to accommodate an elevated number of internal network segments.

The same goes for storage: with constantly appearing and disappearing persistent volumes (PV) and persistent volume claims (PVC), standard rules must be adjusted.

Things work differently in cloud native!

In this article we’ll cover five best practices you _can _follow to improve your Kubernetes security posture, and show you some of the security features we offer in Palette to help you reduce your attack surface.

1. Scan for vulnerabilities

Kubernetes and its ecosystem have many moving parts, and the codebase changes weekly. It’s no surprise that there are vulnerabilities that bad actors could exploit. But not to worry, we have some magic to mitigate these security risks!

Have you heard of kube-bench, kube-hunter and sonobuoy? They’re open-source diagnostic tools that allow you to scan your Kubernetes cluster for known vulnerabilities and misconfigurations, running tests against best practice benchmarks.

We’ve embedded them in our Palette solution, making them easier to consume. You can even schedule regular scans to make sure your clusters are always checked:

Of course there’s a whole world of security tools out in the Kubernetes landscape for you to take advantage of. Through our repository of integrations we make it easy to build these tools into your Cluster Profiles and deploy them to your clusters.

2. Govern access with RBAC

Start by configuring SSO

Whether you’re setting up a Kubernetes cluster for your home lab or for a huge international corporation, chances are that at some point you’d want to create separate accounts and keep admin access separated from normal users.

Palette helps you to quickly connect any SAML or OIDC-based solution such as Okta for SSO:

Thanks to this connection, you’ll immediately be able to log in to Palette under your corporate account and not overuse the “admin” account. Once logged in, you’ll be able to configure granular role based access control (RBAC) and increase the overall security level of your clusters.

Next, use workspaces to control granular access

When your cluster is being managed by Palette, you’re able to configure granular RBAC out of the box. That means you can easily see and modify Roles and Cluster Roles from our neat UI.

But not only that: you’re able to create workspaces that can stretch across multiple clusters. Within these workspaces you’re able to create cluster role bindings.

This gives you an opportunity to achieve something you can’t easily achieve with vanilla K8s: a set of RBAC rules that’s identical across multiple clusters and managed from one place using the Palette UI or Terraform.

When you have several development teams collaborating on multiple applications distributed across various environments (say, dev, stage and production), you can have a quick win for this use case by creating a separate workspace for each application.

Further information about this option can be found in here: https://docs.spectrocloud.com/workspace/workload-features#workspacerolebinding

Always assign minimal rights to each user

When reading community-driven RBAC best practices articles, you’ll have noticed that they recommend assigning only minimal rights to Service Accounts.

We can’t stress enough how important this is. When Service Accounts are too powerful, they make privilege escalations easier for attackers. An intruder who manages to gain access through an old php application would potentially be able to get control of your entire cluster.

Palette allows you to simplify the whole RBAC management process, leaving you time to improve the security of your perimeter. So, please follow this advice once your application works and is stable, it can save you a lot of sweat!

3. Patch and enforce governance with reconciliation loops

We spoke earlier about the vulnerabilities that are inherent to a complex IT system like Kubernetes, and the ecosystem around it. Thankfully, vendors and the community are constantly publishing patches and releasing new versions.

It shouldn’t be a surprise that we recommend you keep K8s up to date, running one of the three latest available K8s versions. And the quicker you apply patches to all the different software elements in your infrastructure, the fewer unpatched known vulnerabilities you’ll have.

But the question then is: how do you keep on top of applying patches, particularly when you have many different clusters running in different environments?

Using a declarative management tool based on Cluster API is the answer. For example, Spectro Cloud Palette’s reconciliation loop mechanism lets you change the ‘desired state’ of your cluster. Modify your Cluster Profile once and Palette applies the change to all the clusters based on this profile. As a bonus, any unexpected deviations from the desired state — in other words, config drift — will be detected and automatically rolled back.

In practice this means that rolling out patches and upgrades to your clusters is quick and straightforward. And because you can introduce approval steps and manage the update process using Terraform, you can maintain full governance.

4. Tackle secrets management

Kubernetes environments don’t exist in vacuum. If you’re reading this article, there’s a fair chance you’re integrating it with the surrounding ecosystem. While you can indeed keep your Kubernetes secrets in etcd, it’s definitely recommended to benefit from this ecosystem and its solutions.

One of such solutions that pretty much has become an industry standard is Vault from HashiCorp. Fortunately, Palette is well integrated with it as well as with a number of other products:

Give it a go, you can quickly read our docs without even creating a login: https://docs.spectrocloud.com/integrations/vault

5. Reduce your attack footprint

Use an immutable OS when possible

The smaller your OS footprint is, the fewer potential options you’re giving to attackers. When possible, try to reduce everything you conveniently can. Let’s say if you’re not planning to install a lot of non-K8s software on your machines, consider using an immutable OS. It helps with security audits as well!

Launch independent clusters instead of co-locating apps

If someone manages to exploit one of your applications, you don’t want them to be able to easily attack other applications to access other areas of your infrastructure. Ideally, you want to isolate applications from each other on separate clusters to limit the ‘blast radius’.

The same goes for your databases: a number of small independent databases will be more difficult to control if they’re fully isolated.

Historically, creating and managing multiple clusters has been challenging and manual, but management automation tools like Palette allow you to create new clusters quickly and painlessly. Take full advantage of this capability!

Benefit from Virtual Clusters:

Do you need to test a new application? Are you being asked to install some unknown Helm Chart as a dependency for some product you’re testing?

Test it within a well-contained perimeter using Palette Virtual Clusters. This modern feature allows you to create virtual clusters very quickly, test your software and remove them completely, therefore considerably reducing any chance of propagation for malicious code.

Bonus: some quick hints for you to follow

• Harden your nodes: if you’re in direct control of your nodes (i.e. not using managed K8s), don’t forget they are still Linux machines, so all the classic hardening rules apply! Close unnecessary ports, change default credentials, and ensure data is encrypted at rest.
• Isolate your nodes: Kubernetes nodes should belong to their own network and not be directly connected to any other networks. This minimizes exposure of the nodes.
• Activate TLS for intra-node communication: this reduces options for attackers to eavesdrop.
• Rotate your certificates regularly. Controlled rotation shouldn’t harm anyone, so stop using a 10-year validity period! A mature enterprise Kubernetes management platform like Palette can help you automate certificate rotation as part of its day 2 operations capabilities.
• Don’t skimp on audit logging. In the case of a security breach, your forensic analysis is going to be as good as your logs. Export them to a solution that’s running outside the cluster so you don’t lose access to them!

Your way forward: the CKS certification

The Kubernetes security best practices we’ve shared in this blog are just the start.

There’s so much more to the wider security context around Kubernetes, and a great way to deepen your knowledge is to pass the Certified Kubernetes Security (CKS) exam. I’m CKS qualified personally and I can testify it’s a very useful certification to hold! To get you started, we have a free lab for you: https://www.spectrocloud.com/blog/practice-for-your-cks-exam-with-palette/. Enjoy!