Kubernetes in air-gapped environments: how to do it right

Across the federal government, air-gapped environments are used to keep sensitive information secure. Think: national security uses cases, military operations, or intelligence missions. 

In fact, many federal programs involving classified data now require the use of air-gapped networks, environments, or computers. 

So, if this sounds like a mission space that you support (or want to begin supporting), it’s essential to know how air-gapped systems work, what’s possible, and how to ensure you get the most out of yours with Kubernetes. 

What does it mean to be air-gapped?

Air-gapped environments distance a platform running an application from the public internet. They are entirely isolated from other computers or networks that are connected to the internet. 

The term “air-gapped” actually comes from the idea that there is literal air, physical separation, between the system and the outside world. So, an individual device, such as a computer, can be air-gapped and so can a network or environment. For example a network of industrial control systems in a factory, or networked medical equipment in a hospital.

By leveraging an air-gapped deployment methodology you can control the flow of software into your environment. To transfer data into an air gapped computer, you need physical access and must bring files over on removable media such as USB drives. 

Because a properly air gapped deployment method creates a physical barrier between you and any potential adversaries, it also makes your infrastructure inherently more secure. While malware could still enter the system for example through an infected USB device or connected mobile phone, remote hacks and data exfiltration over the network are impossible. Best practice is also to lock down hardware even from physical exploits: for example by disabling ports and securing BIOS.

A secondary benefit is that mission-critical applications can continue to run reliably and with continuous interoperability in the absence of the internet or public repositories. 

How does an airgap affect Kubernetes?

A typical Kubernetes cluster needs access to about 20 online repositories to retrieve the base container images and common add-ons. If you’re using an air-gapped deployment method, you must instead point to a single, highly-available repository on a local area network.

This is in fact a good practice anyway. Even if your Kubernetes cluster is designed to have regular internet access, it will still need to be able to fall back to a local software repository if operating in a Denied, Disrupted, Intermittent, and Limited Impact environment, often called a D-DIL environment.

There’s also airgap security to consider – which includes everything from role-based access controls (RBAC), which is essential in high-security environments like those that would require air-gapping, to ensuring the proper patches and upgrades are in place. More on that later. 

What about the edge?

Organizations may air-gap data centers, offices and other traditional, manned computing environments of any scale. 

But many environments at the edge are also air-gapped, either by necessity or design. An IoT device on an oil pipeline in the wilderness, a submarine in the middle of the ocean, a forward outpost in enemy territory, a defense contractor’s factory — these are all environments that either must be disconnected for security reasons, or simply cannot rely on consistent, high-bandwidth WAN connectivity being available. 

When the internet goes away, the Kubernetes environment at the edge location, and all the applications scheduled on it, need to continue functioning. This means it must independently be able to restart any services or failed nodes locally, enforce policy, and generally continue to perform its duties. This all requires a local cache of containers and persistent storage so that the application can store data while disconnected. Your air-gapped networks and systems need the same capability. 

Enter Palette from Spectro Cloud

It is possible to run Kubernetes clusters while air-gapped and still take advantage of modern data management and security practices. There’s no tradeoff necessary. 

Palette from Spectro Cloud allows you to manage the entire lifecycle of any combination of Kubernetes environments regardless of deployment — whether your Kubernetes distribution is on-premises or in the cloud, in production or in development. Palette VerteX is a version designed to meet government and public sector requirements. Installing a local instance of Palette can address all sorts of challenges when it comes to operating in air-gapped environments. 

And this is critical. Because these types of environments are becoming more common and more in-demand than ever. And because bad actors — from nation states to lone wolves and everything in between — are becoming more sophisticated. Additionally, in a world where you can’t always rely on network connectivity, it's good to design for resiliency and for the edge.

Your Kubernetes air-gap checklist

So, you’re ready to give air-gapped Kubernetes a try. But where to start? And with what tools? We’ve built a handy checklist to help you cover all your bases. 

Choose the optimal architecture

A decentralized architecture provides an exceptional foundation for performance, scaling, and resiliency – especially in air-gapped environments. Palette’s architecture was engineered, and later patent-protected, to deliver on the types of unique and essential missions common to federal government users. Perhaps most importantly, policy enforcement happens at cluster, inside the airgap.

Make deployment easy

Start by gathering all the containers, images, manifests and Helm charts you’ll need to build each cluster. Spectro Cloud delivers a single bundle with all the artifacts necessary to build, deploy and manage, including a repository of 50+ curated cloud native computing foundation (CNCF) projects.

Consider day-to-day lifecycle management

Kubernetes clusters need regular care and feeding after deployment, what we call “day 2” operations. This includes reconfiguration, scaling, observability, patches and upgrades, and so on.

How you manage this very much depends on how your airgapped environment is configured. If you have for example an instance of the Palette management plane deployed within the airgapped environment and have access to it, you can use the Palette UI to perform these actions on all the clusters you have under management.

If your workload clusters are deployed within an airgap separated from the management plane, you still have options. For basic day to day operations, each workload cluster is to an extent self managing, using the Palette agent and Cluster API to reconcile the cluster state continually against the desired state documented in your Cluster Profiles. 

If you need to perform a management action, you can log in to the Palette CLI, use the Kubernetes dashboard, or use our new Palette LocalUI to access Palette’s full capabilities directly on the cluster. 

Stay on top of security

One of the primary motivations for running airgapped is to improve security, so we know you care about it. 

Palette has a wide range of security capabilities that enable you to build hardened clusters and verify the security posture of your infrastructure. For example, KubeBench, KubeHunter and Sonobuoy scans are baked in natively, assessing your clusters against CIS benchmarks, penetration testing and conformance to CNCF specifications. These will help you not only keep your clusters secure, but also simplify the validation process needed to get new code into production faster.

We’re also FIPS Verified across the management plane and the full stack of each cluster, including edge clusters.

For edge clusters in particular, we build hardened and immutable images to deploy to the edge device. Immutability ensures that even if tampered with, the device will revert to a known good state after a reboot. 

Leverage local registries

It can be difficult to deploy, scale and manage a highly-available open container initiative (OCI) compatible repository. That’s why Spectro Cloud provides a fully configured Harbor registry to support OCI containers and Helm charts. It also has the ability to scan containers on ingress to ensure the containers are secure.

Across the DoD, Zarf is widely used to package an entire application’s container images, manifests and Helm charts up in a single tarball that’s easy to validate and move from a conventional connected development environment, to a disconnected “high side” environment. Spectro Cloud is the only vendor to support Zarf directly from the platform’s UI.

Learn more

If deploying and managing Kubernetes clusters in air-gapped environments is proving challenging for your team, or if you have questions, please get in touch. We'd be happy to share our experience and help you move forward faster.

In the meantime, check out our new documentation on air-gapped deployments and see how our Palette VerteX can help.