DevOps | Continuous Integration, Continuous Delivery and Continuous Deployment | Container Scanning with Anchore Engine

DevOps | Continuous Integration, Continuous Delivery and Continuous Deployment…

9th June 20199th June 2019

Docker

Docker provides a user space runtime environment allowing applications and services to be executed with a smaller footprint of the operating system via containers. With containers, it becomes easier for teams across different units, such as Development, UAT and Operations to work seamlessly across applications. Docker containers are lightweight and are easily scalable.

Dockerfile

This simple Dockerfile encapsulates a java Spring boot micro service.

FROM azul/zulu-openjdk:11.0.2 

EXPOSE 8080

RUN mkdir -p /app/

ADD build/libs/springBootApp*.jar /app/springBootApp.jar

ENTRYPOINT exec java -jar /app/springBootApp.jar

Nexus Private Registry

Having our images pushed to Dockerhub.io is satisfactory for publicly available images, however we want to use our own private registry. Private Docker registries are supported in Nexus(3). We can ensure only one version of each image is supported. We have avoided the use of ‘latest’ when working with Docker images and applied the same semantic versioning strategy we apply for other artefacts (https://www.linkedin.com/pulse/devops-part-2-continuous-integration-delivery-deployment-surjit-bains/).

Create a Repository in Nexus

Disabling redeploy for this repository, so that each version of a Docker Image is immutable. Create another Docker Repository (docker-stage), with redeploy enabled to stage our Docker Images and apply updates. We have defined DNS and reverse proxies image.polarpoint.io and stage-image.polarpoint.io that go to the repository and port for our two new Docker repositories.

Building Docker images

Building docker images is started as follows:

docker build -t static-image.polarpoint.io/springBootApp:0.13.0-SNAPSHOT .
Sending build context to Docker daemon  768.3MB
Step 1/5 : FROM azul/zulu-openjdk:11.0.2
11.0.2: Pulling from azul/zulu-openjdk
6cf436f81810: Already exists
987088a85b96: Already exists
b4624b3efe06: Already exists
d42beb8ded59: Already exists
d15285a66a34: Pull complete
9734df7bad40: Pull complete
Digest: sha256:3e6fac72ac772ffcc78044dc2088805b6324213cc7624c5ae5b4fa7b1fcd5e47
Status: Downloaded newer image for azul/zulu-openjdk:11.0.2
 ---> 2426b4e13c8c
Step 2/5 : EXPOSE 8080
 ---> Running in ef64a8b668ff
Removing intermediate container ef64a8b668ff
 ---> d30a49157890
Step 3/5 : RUN mkdir -p /app/
 ---> Running in c641ac156f73
Removing intermediate container c641ac156f73
 ---> 9b094a4c5a8e
Step 4/5 : ADD build/libs/springBootApp-*.jar /app/springBootApp.jar
 ---> 07c12d8ff1c6
Step 5/5 : ENTRYPOINT exec java -jar /app/springBootApp.jar
 ---> Running in 09ab672c55fb
Merge branch 'development' of https://xxx into development
Removing intermediate container 09ab672c55fb
 ---> 21d04fa1f1f5
Successfully built 21d04fa1f1f5
Successfully tagged static-image.polarpoint.io/springBootApp:0.13.0-SNAPSHOT

Helm

Helm is the package manager for Kubernetes. Applications are packaged in Charts that are made up of collections of Kubernetes components that have been templated out including the definition and configuration of resources to be deployed to a Kubernetes cluster. Helm is made up of client and server side elements. Helm refers to the client-side command. Tiller is the Helm server side that runs in Kubernetes and handles the Helm packages.

Installing Anchore Engine

Anchore is an open source container compliance platform that ensures security and stability of production container deployments. Installing it with default values in one line:

helm install --name polar-io-comp-anchore stable/anchore-engine

After a few minutes, Anchore Engine is installed into our cluster and has updated it’s CVE vulnerability data:

helm ls
NAME            	REVISION	UPDATED                 	STATUS  	CHART                	NAMESPACE

pol-io-comp-anchore	1       	Sun Mar 17 18:51:58 2019	DEPLOYED	anchore-engine-0.11.0	jenkins

we can list the services running in our cluster the anchore-engine-api is listening on port 32199:

kubectl get services 
NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          
pol-io-comp-anchore-anchore-engine-api           NodePort    xxxx            <none>        8228:32199/TCP  
pol-io-comp-anchore-anchore-engine-catalog       ClusterIP   xxxx            <none>        8082/TCP        
pol-io-comp-anchore-anchore-engine-policy        ClusterIP   xxxx            <none>        8087/TCP        
pol-io-comp-anchore-anchore-engine-simplequeue   ClusterIP   xxxx            <none>        8083/TCP        
pol-io-comp-anchore-postgresql                   ClusterIP   xxxx            <none>        5432/TCP  1

Configuring the Anchore Plugin in Jenkins

Install the Anchore plugin into Jenkins and configure it to use the Anchore api service ClusterIP and port:

Adding additional stages to our Jenkins pipeline

With Anchore engine installed we add a number of extra stages in our Jenkins pipeline.

Build Docker image using the staging Docker registry:

...

def imageLine = "${stagingDockerRegistry}/${dockerName}"
sh """
         docker login -u ${USER} -p ${PASSWORD} ${stagingDockerRegistry}
   """
appContainer = docker.build("${imageLine}")

...

Push the created image to our staging Docker registry:

...
def imageLine = "${stagingDockerRegistry}/${dockerName}"
echo "Build: about to call docker publish to staging repository ${imageLine}"
appContainer.push()
...

Scan the newly created image using our Anchore engine running in the cluster:

...
def imageLine = "${stagingDockerRegistry}/${dockerName}"
writeFile file: 'anchore_images', text: imageLine
anchore name: 'anchore_images'
...

This generates a report within Jenkins that summaries common exposure and vulnerabilities lists:

As we have passed the security gate we pull the Docker image from our staging repository, retag and push to our stable private registry:

...
    sh """
              docker login -u ${USER} -p ${PASSWORD} ${stagingDockerRegistry}
              docker pull  ${stagingImageLine}
              docker tag  ${stagingImageLine}  ${imageLine}

              docker login -u ${PRIV_USER} -p ${PRIV_PASSWORD} ${dockerRegistry}
              docker push ${imageLine}

       """
...
}

Nightly Scanning

In addition to running scans against our images as they are being built, we want to scan all the -FINAL stored in our private Docker registry each night. This allows us to create a strategy for patching Docker images that have been deployed to production. Ensuring we can identify new vulnerabilities in our production deployed images.

Adding a Jenkins pipeline to run every night and push it to Anchore engine to be scanned

    try {
          sh """
          curl https://image.polarpoint.io:443/v2/_catalog  | jq '.repositories[]'  | sort  | xargs -I _ curl -s -k -X GET https://image.polarpoint.io:443/v2/_/tags/list | jq -M '.["name"] + ":" + .["tags"][]' | grep 'FINAL' >>anchore_images
          """
            anchore name: 'anchore_images'
        }

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1531  100  1531    0     0   2161      0 --:--:-- --:--:-- --:--:--  2159

...
"terraform:0.6.0-FINAL"
"visualiser:0.8.0-FINAL"
"visualiser:0.9.0-FINAL"
"springBootApp:0.15.0-RELEASE"
...
02:03:16  2019-03-31T02:03:16.229 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/terraform:0.1.0-RELEASE
02:03:16  2019-03-31T02:03:16.334 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/terraform:0.2.0-RELEASE
02:03:16  2019-03-31T02:03:16.443 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/terraform:0.3.0-RELEASE
02:03:16  2019-03-31T02:03:16.565 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/terraform:0.4.0-RELEASE
02:03:16  2019-03-31T02:03:16.691 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/terraform:0.5.0-RELEASE
02:03:16  2019-03-31T02:03:16.799 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/terraform:0.6.0-RELEASE
02:03:17  2019-03-31T02:03:17.541 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.1.0-RELEASE
02:03:17  2019-03-31T02:03:17.679 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.2.0-RELEASE
02:03:17  2019-03-31T02:03:17.805 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.3.0-RELEASE
02:03:17  2019-03-31T02:03:17.920 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.4.0-RELEASE
02:03:18  2019-03-31T02:03:18.035 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.5.0-RELEASE
02:03:18  2019-03-31T02:03:18.149 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.6.0-RELEASE
02:03:18  2019-03-31T02:03:18.273 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.7.0-RELEASE
02:03:18  2019-03-31T02:03:18.389 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.8.0-RELEASE
02:03:18  2019-03-31T02:03:18.506 INFO   AnchoreWorker   Querying vulnerability listing for image.mycnets.com/visualiser:0.9.0-RELEASE
02:03:18  Archiving artifacts