Why Continuous Integration is nothing without Continuous Security

Speed kills

DevOps is all about increasing speed, with the ultimate goal of iterating quickly to deploy to production environments. But all the benefits of fast, continuous delivery in an agile environment are for nothing if security is compromised.

Continuous security must always be a critical aspect of any software development process or lifecycle. But when security risks are so widespread, that’s easier said than done. And if security practices only get applied from the testing to the deployment phase of the project lifecycle, you’re starting from a huge disadvantage.

Introducing CICS

In this blog, we’ll explore the concept of CICS (Continuous Integration Continuous Security), and show you how to seamlessly integrate security into your Continuous Integration pipeline to create a secure deployment process with an easy-to-manage workflow, right from the earliest stages of the development lifecycle.

From a CICS perspective, security is the discipline and practice of shielding the entire DevOps environment through strategies, policies, processes, and technology. In a Kubernetes environment, CICS focuses on managing the complete lifecycle of clusters, new or existing, across multiple environments, with top-to-bottom control mandates to focus on areas requiring intense security follow-ups.

Let us enumerate a few of the core areas with security concerns:

Let us enumerate a few of the core areas with security concerns:

1. Application Source Code
2. Operating System
3. Container Images
4. Kubernetes resources
5. Cloud Service hosting the cluster
6. And lastly, the Kubernetes cluster itself

• At the build stage of the lifecycle, the key focus is the application source code to be deployed.
• At point of release, security should focus on the elements of the build: OS images, container images, and the associated Kubernetes resources.
• At deployment, security must target the underlying environment (such as a public cloud or edge server) and the K8s cluster that will host the container workload.

Building your toolbox

For each of the six focus areas we’ve described, there are multiple security activities you can conduct (for instance, AV scanning or lint checks), and many possible tools you can choose to make each one happen. How do we find the right tool for our requirements?

To find the right tool for you, it’s important to look at a few criteria:

1. CI integration. For a Continuous Security model, we would need to choose tools that would integrate well with the CI system. For this the tool needs to have capability to run as a CLI tool or be invoked with an API call.
2. Standard outputs. If the tools produce a standard output format [yaml, json] it becomes easier to bring all the info together in a security dashboard. The CI jobs can be configured to run on-demand, to help developers to fix security issues and verify the same with the build jobs.
3. Rich vulnerability database support. This is vital to detect issues. The list of vulnerable databases the tool supports can come in handy during the selection process.
4. Integrations with alert and ticketing systems. Though not required for the CI integration, it is recommended to have an alert mechanism when new issues are reported. An email alert from the tool when a security scan reveals a new vulnerability would be useful to proactively identify and fix issues.

Tool examples for different areas of security

This is not the first time we’re giving you a list of security tools to go with your Kubernetes environment. If you haven’t already, we invite you to check out this blog and the webinar that goes with it. But in the table below we map a few key tools to the core areas and activities of the CICS pipeline that we discussed above. You can see how the mapping plays out.

Code snippets

Here at Spectro Cloud we are a helpful bunch, and we are all about promoting good security practices and making it easy for other devs and ops practitioners to adopt them. So, take these snippets with our blessings!

Here are a few examples of security scans in different stages of deployment. The code snippets given below are simple bash functions that can be included in any CI flow. The idea is to have the output of all the scans consolidated in a common location and use the same for creating a dashboard. The text file output can be used for manually consolidating the vulnerabilities while the json output can be used programmatically to create a report.

Build stage

 
# Prerequisites - golangcilint installed on the build instance / container
run_lint() {
  echo "Running Lint check"
  golangci-lint run	./...  --timeout 10m  --tests=false > lint.txt
  golangci-lint run	./...  --out-format json --timeout 10m  --tests=false >  lint.json

  # Upload the scan results  to Google bucket
  gsutil cp -r lint.txt    gs:///compliance//
  gsutil cp -r lint.json gs:///compliance//
}
	

Release stage

 
# Prerequisites - trivy installed on the build instance / container
run_container_scan() {
  echo 'Running container scan'
  images="image1 image2 image3 image4"
  for each_image in ${images}
    do
      trivy --download-db-only
      trivy image ${each_image} >> image-cve.txt
      trivy image -q -f json -o image-cve.json ${each_image}
    done

  # Upload the scan results  to Google bucket
  gsutil cp -r image-cve.txt    gs:///compliance//
  gsutil cp -r image-cve.json gs:///compliance//
}
	

Deploy stage

 
# Prerequisites - kubectl, kube-bench installed on the build instance / container
run_k8s_benchmark() {
  echo 'Running Benchmark’'
  kubectl create -f kube-bench-json.yaml
  cp_state=$(kubectl get job kube-bench-master -o jsonpath={.status.succeeded})
  node_state=$(kubectl get job kube-bench-node -o jsonpath={.status.succeeded})
  while [ ! ${cp_state} ] || [ ! ${node_state} ]
    do
      cp_state=$(kubectl get job kube-bench-master -o jsonpath={.status.succeeded})
      node_state=$(kubectl get job kube-bench-node -o jsonpath={.status.succeeded})
    done

  cp_pod=$(kubectl get pods | grep kube-bench-master | awk '{print $1'})
  kubectl  logs ${cp_pod} > benchmark-cp.json

  node_pod=$(kubectl get pods | grep kube-bench-node | awk '{print $1'})
  kubectl logs ${node_pod} > benchmark-node.json
  kubectl delete -f kube-bench-json.yaml

  # Upload the scan results  to Google bucket
  gsutil cp -r benchmark-cp.txt    gs:///compliance//
  gsutil cp -r benchmark-node.json gs:///compliance//
}
	

Of course, we encourage you to experiment and adapt to fit the tools to your environment and preferred workflow. Do get in touch if you have any suggestions or questions!

Wrapping up

This blog gives you an overview of how to begin Continuous Security within a Continuous Integration environment, through examining some of the key security control points at build, release and deploy stages, and the tools you can use to run checks on each of your builds.

This is just the starting point; you can continue your journey with Continuous Integration by exploring more tools adaptable to the different development environments. We will be covering the notification and dashboard aspects of CICS in a follow-up blog to wrap this up. So stay tuned!