Published 2022-05-16

Why Continuous Integration is nothing without Continuous Security

Director Devops @ Spectro Cloud

Speed kills

DevOps is all about increasing speed, with the ultimate goal of iterating quickly to deploy to production environments. But all the benefits of fast, continuous delivery in an agile environment are for nothing if security is compromised.

Continuous security must always be a critical aspect of any software development process or lifecycle. But when security risks are so widespread, that’s easier said than done. And if security practices only get applied from the testing to the deployment phase of the project lifecycle, you’re starting from a huge disadvantage.

Introducing CICS

In this blog, we’ll explore the concept of CICS (Continuous Integration Continuous Security), and show you how to seamlessly integrate security into your Continuous Integration pipeline to create a secure deployment process with an easy-to-manage workflow, right from the earliest stages of the development lifecycle.

From a CICS perspective, security is the discipline and practice of shielding the entire DevOps environment through strategies, policies, processes, and technology. In a Kubernetes environment, CICS focuses on managing the complete lifecycle of clusters, new or existing, across multiple environments, with top-to-bottom control mandates to focus on areas requiring intense security follow-ups.

Let us enumerate a few of the core areas with security concerns:

Let us enumerate a few of the core areas with security concerns:

  1. Application Source Code
  2. Operating System
  3. Container Images
  4. Kubernetes resources
  5. Cloud Service hosting the cluster
  6. And lastly, the Kubernetes cluster itself
  • At the build stage of the lifecycle, the key focus is the application source code to be deployed.
  • At point of release, security should focus on the elements of the build: OS images, container images, and the associated Kubernetes resources.
  • At deployment, security must target the underlying environment (such as a public cloud or edge server) and the K8s cluster that will host the container workload.

Figure1: Kubernetes Service Deployment - Security Spectrum Figure1: Kubernetes Service Deployment - Security Spectrum

Building your toolbox

For each of the six focus areas we’ve described, there are multiple security activities you can conduct (for instance, AV scanning or lint checks), and many possible tools you can choose to make each one happen. How do we find the right tool for our requirements?

To find the right tool for you, it’s important to look at a few criteria:

  1. CI integration. For a Continuous Security model, we would need to choose tools that would integrate well with the CI system. For this the tool needs to have capability to run as a CLI tool or be invoked with an API call.
  2. Standard outputs. If the tools produce a standard output format [yaml, json] it becomes easier to bring all the info together in a security dashboard. The CI jobs can be configured to run on-demand, to help developers to fix security issues and verify the same with the build jobs.
  3. Rich vulnerability database support. This is vital to detect issues. The list of vulnerable databases the tool supports can come in handy during the selection process.
  4. Integrations with alert and ticketing systems. Though not required for the CI integration, it is recommended to have an alert mechanism when new issues are reported. An email alert from the tool when a security scan reveals a new vulnerability would be useful to proactively identify and fix issues.

Tool examples for different areas of security

This is not the first time we’re giving you a list of security tools to go with your Kubernetes environment. If you haven’t already, we invite you to check out this blog and the webinar that goes with it. But in the table below we map a few key tools to the core areas and activities of the CICS pipeline that we discussed above. You can see how the mapping plays out.

ToolsCore AreaSecurity TypeTool Choice Criteria
FossaSource CodeSource Code SecurityJira Integration
Overall Security Report
CLI Support
golangcilintSource CodeLint ChecksRich set of Linters
FossaSource CodeOpen Source
Compliance
Jira Integration
Audit Reports
CLI Support
Trivy
Clair
KubernetesContainer ImagesCLI based
Json output
KubebenchKubernetesBenchmarkCLI based
Json output
SonobuoyKubernetesConformanceCLI based
Json output
Kubescape
Kubesec
KubernetesRisk AnalysisCLI based
Json output
KubeHunterKubernetesPenetrationCLI based
Json output
VulsOSOS SecurityCLI based
OVAL, Errata,
NVD db support
Linux benchOSOS BenchmarkCLI Based
Json output
ClamAVOSAnti VirusCLI Based

Code snippets

Here at Spectro Cloud we are a helpful bunch, and we are all about promoting good security practices and making it easy for other devs and ops practitioners to adopt them. So, take these snippets with our blessings!

Here are a few examples of security scans in different stages of deployment. The code snippets given below are simple bash functions that can be included in any CI flow. The idea is to have the output of all the scans consolidated in a common location and use the same for creating a dashboard. The text file output can be used for manually consolidating the vulnerabilities while the json output can be used programmatically to create a report.

Build stage

# Prerequisites - golangcilint installed on the build instance / container run_lint() { echo "Running Lint check" golangci-lint run ./... --timeout 10m --tests=false > lint.txt golangci-lint run ./... --out-format json --timeout 10m --tests=false > lint.json # Upload the scan results to Google bucket gsutil cp -r lint.txt gs://<bucketname>/compliance/<date>/ gsutil cp -r lint.json gs://<bucketname>/compliance/<date>/ }

Release stage

# Prerequisites - trivy installed on the build instance / container run_container_scan() { echo 'Running container scan' images="image1 image2 image3 image4" for each_image in ${images} do trivy --download-db-only trivy image ${each_image} >> image-cve.txt trivy image -q -f json -o image-cve.json ${each_image} done # Upload the scan results to Google bucket gsutil cp -r image-cve.txt gs://<bucketname>/compliance/<date>/ gsutil cp -r image-cve.json gs://<bucketname>/compliance/<date>/ }

Deploy stage

# Prerequisites - kubectl, kube-bench installed on the build instance / container run_k8s_benchmark() { echo 'Running Benchmark’' kubectl create -f kube-bench-json.yaml cp_state=$(kubectl get job kube-bench-master -o jsonpath={.status.succeeded}) node_state=$(kubectl get job kube-bench-node -o jsonpath={.status.succeeded}) while [ ! ${cp_state} ] || [ ! ${node_state} ] do cp_state=$(kubectl get job kube-bench-master -o jsonpath={.status.succeeded}) node_state=$(kubectl get job kube-bench-node -o jsonpath={.status.succeeded}) done cp_pod=$(kubectl get pods | grep kube-bench-master | awk '{print $1'}) kubectl logs ${cp_pod} > benchmark-cp.json node_pod=$(kubectl get pods | grep kube-bench-node | awk '{print $1'}) kubectl logs ${node_pod} > benchmark-node.json kubectl delete -f kube-bench-json.yaml # Upload the scan results to Google bucket gsutil cp -r benchmark-cp.txt gs://<bucketname>/compliance/<date>/ gsutil cp -r benchmark-node.json gs://<bucketname>/compliance/<date>/ }

Of course, we encourage you to experiment and adapt to fit the tools to your environment and preferred workflow. Do get in touch if you have any suggestions or questions!

Wrapping up

This blog gives you an overview of how to begin Continuous Security within a Continuous Integration environment, through examining some of the key security control points at build, release and deploy stages, and the tools you can use to run checks on each of your builds.

This is just the starting point; you can continue your journey with Continuous Integration by exploring more tools adaptable to the different development environments. We will be covering the notification and dashboard aspects of CICS in a follow-up blog to wrap this up. So stay tuned!

Related Articles

  • Blog Posts

    How to keep your Kubernetes secrets… secret

    Read our article
  • Blog Posts

    Kubernetes for dummies tutorial

    Read our article
  • Blog Posts

    Enterprise challenges for containers and Kubernetes

    Read our article
  • Blog Posts

    Why we Invested in Spectro Cloud

    Read our article
Be the first to receive the latest on
K8s, Palette, our upcoming webinar, events, and much more!

We are using the information you provide to us to send you our montly newsletter. You may unsubscribe at any time.
For more information, please see our Privacy Policy.

Spectro Cloud uniquely enables organizations to manage Kubernetes in production, at scale. Our Palette management platform gives effortless control of the full Kubernetes lifecycle, across clouds, data centers, bare metal and edge environments.
Connect with us
Connect with us

© 2022 Spectro Cloud®. All rights reserved.