
DevOps | Continuous Integration, Continuous Delivery and Continuous Deployment…
Docker
Docker provides a user space runtime environment allowing applications and services to be executed with a smaller footprint of the operating system via containers. With containers, it becomes easier for teams across different units, such as Development, UAT and Operations to work seamlessly across applications. Docker containers are lightweight and are easily scalable.
Dockerfile
This simple Dockerfile encapsulates a java Spring boot micro service.
FROM azul/zulu-openjdk:11.0.2 EXPOSE 8080 RUN mkdir -p /app/ ADD build/libs/springBootApp*.jar /app/springBootApp.jar ENTRYPOINT exec java -jar /app/springBootApp.jar
Nexus Private Registry
Having our images pushed to Dockerhub.io is satisfactory for publicly available images, however we want to use our own private registry. Private Docker registries are supported in Nexus(3). We can ensure only one version of each image is supported. We have avoided the use of ‘latest’ when working with Docker images and applied the same semantic versioning strategy we apply for other artefacts (https://www.linkedin.com/pulse/devops-part-2-continuous-integration-delivery-deployment-surjit-bains/).
Create a Repository in Nexus
Disabling redeploy for this repository, so that each version of a Docker Image is immutable. Create another Docker Repository (docker-stage), with redeploy enabled to stage our Docker Images and apply updates. We have defined DNS and reverse proxies image.polarpoint.io and stage-image.polarpoint.io that go to the repository and port for our two new Docker repositories.
Building Docker images
Building docker images is started as follows:
docker build -t static-image.polarpoint.io/springBootApp:0.13.0-SNAPSHOT . Sending build context to Docker daemon 768.3MB Step 1/5 : FROM azul/zulu-openjdk:11.0.2 11.0.2: Pulling from azul/zulu-openjdk 6cf436f81810: Already exists 987088a85b96: Already exists b4624b3efe06: Already exists d42beb8ded59: Already exists d15285a66a34: Pull complete 9734df7bad40: Pull complete Digest: sha256:3e6fac72ac772ffcc78044dc2088805b6324213cc7624c5ae5b4fa7b1fcd5e47 Status: Downloaded newer image for azul/zulu-openjdk:11.0.2 ---> 2426b4e13c8c Step 2/5 : EXPOSE 8080 ---> Running in ef64a8b668ff Removing intermediate container ef64a8b668ff ---> d30a49157890 Step 3/5 : RUN mkdir -p /app/ ---> Running in c641ac156f73 Removing intermediate container c641ac156f73 ---> 9b094a4c5a8e Step 4/5 : ADD build/libs/springBootApp-*.jar /app/springBootApp.jar ---> 07c12d8ff1c6 Step 5/5 : ENTRYPOINT exec java -jar /app/springBootApp.jar ---> Running in 09ab672c55fb Merge branch 'development' of https://xxx into development Removing intermediate container 09ab672c55fb ---> 21d04fa1f1f5 Successfully built 21d04fa1f1f5 Successfully tagged static-image.polarpoint.io/springBootApp:0.13.0-SNAPSHOT
Helm
Helm is the package manager for Kubernetes. Applications are packaged in Charts that are made up of collections of Kubernetes components that have been templated out including the definition and configuration of resources to be deployed to a Kubernetes cluster. Helm is made up of client and server side elements. Helm refers to the client-side command. Tiller is the Helm server side that runs in Kubernetes and handles the Helm packages.
Installing Anchore Engine
Anchore is an open source container compliance platform that ensures security and stability of production container deployments. Installing it with default values in one line:
helm install --name polar-io-comp-anchore stable/anchore-engine
After a few minutes, Anchore Engine is installed into our cluster and has updated it’s CVE vulnerability data:
helm ls NAME REVISION UPDATED STATUS CHART NAMESPACE pol-io-comp-anchore 1 Sun Mar 17 18:51:58 2019 DEPLOYED anchore-engine-0.11.0 jenkins
we can list the services running in our cluster the anchore-engine-api is listening on port 32199:
kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) pol-io-comp-anchore-anchore-engine-api NodePort xxxx <none> 8228:32199/TCP pol-io-comp-anchore-anchore-engine-catalog ClusterIP xxxx <none> 8082/TCP pol-io-comp-anchore-anchore-engine-policy ClusterIP xxxx <none> 8087/TCP pol-io-comp-anchore-anchore-engine-simplequeue ClusterIP xxxx <none> 8083/TCP pol-io-comp-anchore-postgresql ClusterIP xxxx <none> 5432/TCP 1
Configuring the Anchore Plugin in Jenkins
Install the Anchore plugin into Jenkins and configure it to use the Anchore api service ClusterIP and port:
Adding additional stages to our Jenkins pipeline
With Anchore engine installed we add a number of extra stages in our Jenkins pipeline.
Build Docker image using the staging Docker registry:
... def imageLine = "${stagingDockerRegistry}/${dockerName}" sh """ docker login -u ${USER} -p ${PASSWORD} ${stagingDockerRegistry} """ appContainer = docker.build("${imageLine}") ...
Push the created image to our staging Docker registry:
... def imageLine = "${stagingDockerRegistry}/${dockerName}" echo "Build: about to call docker publish to staging repository ${imageLine}" appContainer.push() ...
Scan the newly created image using our Anchore engine running in the cluster:
... def imageLine = "${stagingDockerRegistry}/${dockerName}" writeFile file: 'anchore_images', text: imageLine anchore name: 'anchore_images' ...
This generates a report within Jenkins that summaries common exposure and vulnerabilities lists:
As we have passed the security gate we pull the Docker image from our staging repository, retag and push to our stable private registry:
... sh """ docker login -u ${USER} -p ${PASSWORD} ${stagingDockerRegistry} docker pull ${stagingImageLine} docker tag ${stagingImageLine} ${imageLine} docker login -u ${PRIV_USER} -p ${PRIV_PASSWORD} ${dockerRegistry} docker push ${imageLine} """ ... }
Nightly Scanning
In addition to running scans against our images as they are being built, we want to scan all the -FINAL stored in our private Docker registry each night. This allows us to create a strategy for patching Docker images that have been deployed to production. Ensuring we can identify new vulnerabilities in our production deployed images.
Adding a Jenkins pipeline to run every night and push it to Anchore engine to be scanned
try { sh """ curl https://image.polarpoint.io:443/v2/_catalog | jq '.repositories[]' | sort | xargs -I _ curl -s -k -X GET https://image.polarpoint.io:443/v2/_/tags/list | jq -M '.["name"] + ":" + .["tags"][]' | grep 'FINAL' >>anchore_images """ anchore name: 'anchore_images' }
}
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1531 100 1531 0 0 2161 0 --:--:-- --:--:-- --:--:-- 2159 ... "terraform:0.6.0-FINAL" "visualiser:0.8.0-FINAL" "visualiser:0.9.0-FINAL" "springBootApp:0.15.0-RELEASE" ... 02:03:16 2019-03-31T02:03:16.229 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/terraform:0.1.0-RELEASE 02:03:16 2019-03-31T02:03:16.334 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/terraform:0.2.0-RELEASE 02:03:16 2019-03-31T02:03:16.443 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/terraform:0.3.0-RELEASE 02:03:16 2019-03-31T02:03:16.565 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/terraform:0.4.0-RELEASE 02:03:16 2019-03-31T02:03:16.691 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/terraform:0.5.0-RELEASE 02:03:16 2019-03-31T02:03:16.799 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/terraform:0.6.0-RELEASE 02:03:17 2019-03-31T02:03:17.541 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.1.0-RELEASE 02:03:17 2019-03-31T02:03:17.679 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.2.0-RELEASE 02:03:17 2019-03-31T02:03:17.805 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.3.0-RELEASE 02:03:17 2019-03-31T02:03:17.920 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.4.0-RELEASE 02:03:18 2019-03-31T02:03:18.035 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.5.0-RELEASE 02:03:18 2019-03-31T02:03:18.149 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.6.0-RELEASE 02:03:18 2019-03-31T02:03:18.273 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.7.0-RELEASE 02:03:18 2019-03-31T02:03:18.389 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.8.0-RELEASE 02:03:18 2019-03-31T02:03:18.506 INFO AnchoreWorker Querying vulnerability listing for image.mycnets.com/visualiser:0.9.0-RELEASE 02:03:18 Archiving artifacts
References
Nexus:
https://www.sonatype.com/nexus-repository-oss
Helm:
Anchore container scanner plugin:
https://wiki.jenkins.io/display/JENKINS/Anchore+Container+Image+Scanner+Plugin
Anchore engine