Feature branches CI with Openshift and Jenkins pipeline

A few months ago, I setup feature branches CI with Openshift using Jenkins pipeline and Openshift client tools.

Our company is on an intranet, so this is after experience with CI platforms like Atlassian Bamboo and Bitbucket pipelines, as well as orchestration platforms like Docker Swarm and plain Kubernetes.

In my opinion, this is the best setup for private clouds, as it is mostly code as configuration and it’s all open-source. So after half a year of tweaking it, I wanted to share my Jenkinsfile and development workflow.

Another important point of this setup is that it allows me to deploy previous builds, which was an issue I have had since migrating away from Atlassian Bamboo, which had a very handy deployment feature despite its numerous flaws.

Jenkinsfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
//Jenkinsfile parameters
groupName = my-group
projectName = my-project
ftpServer = dev-ftp-1
deploySrv = dev-test-1
registryAddress = dev-reg-1

node('agents') {
//Build parameters for builds and/or deployments, add more if necessary
properties([
choice(name: 'DoBuild', choices: 'true\nfalse', description: 'Build or deploy a previous image'),
choice(name: 'WhichBuild', choices: (env.BUILD_NUMBER.toInteger()..1).join(\n), description: 'If deploying, which previous build'),
choice(name: 'WhereToDeploy', choices: 'feature-branch\ndev-server\nopenshift-int\nopenshift-ppd\npre-prod-tar\nNONE', description: 'Where to deploy, NONE for no deploy')
])

//extract build information ahead of time
buildNumber = env.BUILD_NUMBER
buildName = env.BRANCH_NAME + "/" + buildNumber
buildToDeploy = params.DoBuild == "true" ? buildNumber : params.WhichBuild
whereToDeploy = params.whereToDeploy
buildType = params.DoBuild == "true" ? "build" : "deploy"
branchName = env.BRANCH_NAME.replace("/", "-").replace("_", "-").toLowerCase() //replace problematic Openshift characters
imageName = "${registryAddress}/${groupName}/${projectName}:${branchName}"
buildDescription = (buildType == "build" ? "Build" : "Deployment from build ${buildNumber}") + " to ${whereToDeploy} started"

echo buildDescription
//Optionally implement curl-based build started notification here using buildDescription, e.g Slack

try {
stage('clean') {
//Clean tasks, for example:
deleteDir()
}

stage('checkout') {
//Checkout/clone/pull tasks, for example:
checkout scm
}

if(buildType == "build") {
stage('build') {
//Build tasks, for example:
sh "gradle bootRepackage"
}

stage('test') {
//Test tasks, for example:
try {
sh "gradle test"
} finally {
junit 'build/test-results/test/*.xml' //fail build if JUnit plugin reports exceptions
}
}

stage('docker') {
//Build and push the image
withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: 'docker-registry', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
sh "sudo docker login -p $PASSWORD -u $USERNAME registry.address"
}
docker.withRegistry('https://registry.address', 'docker-registry') {
sh "docker build -t ${imageName}-${buildNumber} ."
sh "docker push ${imageName}-${buildNumber}"
}
}
}
}

stage('deploy') {
//If not deploying, finish
if(whereToDeploy == "NONE")
return

if(whereToDeploy == "feature-branch" || whereToDeploy == "openshift-int" || whereToDeploy == "openshift-ppd") {
//If deploying feature branch or int/preprod, first determine deployment name
if(branchName == 'feature-development') deployName = "${projectName}-dev"
if(whereToDeploy == 'openshift-int') deployName = "${projectName}-int"
if(whereToDeploy == 'openshift-ppd') deployName = "${projectName}-ppd"
else deployName = "${projectName}-${branchName}"

//Then, login to Openshift, prepare the YAML files and execute oc commands
withCredentials([string(credentialsId: 'registry-token', variable: 'TOKEN')]) {
sh "oc login ${registry-address} --token==$TOKEN"
}

sh "sed -i 's/BRANCHNAME/${deployName}/g' deploy/deployment.yml"
sh "sed -i 's/BRANCHNAME/${deployName}/g' deploy/service.yml"
sh "sed -i 's/BRANCHNAME/${deployName}/g' deploy/route.yml"
sh "sed -i 's/DEPLOYNAME/${deployName}/g' deploy/deployment.yml"
sh "sed -i 's/DEPLOYNAME/${deployName}/g' deploy/service.yml"
sh "sed -i 's/DEPLOYNAME/${deployName}/g' deploy/route.yml"

sh "oc project ${groupName}"
sh "oc apply -f deploy/deployment.yml"
sh "oc apply -f deploy/service.yml"
sh "oc apply -f deploy/route.yml"

//Finally, login and push the image to the registry
docker.withRegistry('https://registry.address', 'docker-registry') {
sh "docker pull ${imageName}-${buildToDeploy}"
sh "docker tag ${imageName}-${buildToDeploy} ${imageName}"
sh "docker push ${imageName}" //Openshift will use the image tagged :latest
}
}
else if(whereToDeploy == 'dev-server') {
//If deploying to a simple server, use SSH to pull and run the image, then wait for healthCheck to resolve
withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: 'docker-registry', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: 'server-login', usernameVariable: 'SRVUSERNAME', passwordVariable: 'SRVPASSWORD']]) {
sh """
sshpass -p${SRVPASSWORD} ssh -o StrictKeyChecking=no -o UserKnownHostsFile=/dev/null ${SRVUSERNAME}@${deploySrv} '\
docker login p ${PASSWORD} -u ${USERNAME} ${registryAddress}; \
docker system prune -f; docker kill ${projectName}; docker rm ${projectName}; \
docker pull ${imageName}-${buildToDeploy}; \
docker run --detach --name ${projectName} --hostname=\$HOSTNAME ${imageName}-${buildToDeploy}';
bash -c 'until [[ "\$curl -s --output /dev/null --head --fail -w ''%{http_code}'' \
http://${whereToDeploy}/healthCheck )" = 200 ]] ; do echo '.'; sleep 5; done; \
echo 'Server is up';
"""
}
}
}
else if(whereToDeploy == 'pre-prod-tar') {
//If deploying a TAR for production, pull, tag, save and upload to an FTP server
sh """
docker login p ${PASSWORD} -u ${USERNAME} ${registryAddress}; \
docker pull ${imageName}-${buildToDeploy}; \
docker tag ${imageName}-${buildToDeploy} ${groupName}:${projectName}-${buildToDeploy}; \
docker save ${groupName}:${projectName}-${buildToDeploy} ${groupName}:${projectName}-${buildToDeploy}.tar; \
curl -T ${groupName}:${projectName}-${buildToDeploy}.tar ftp://${ftpServer};
"""
}
}

currentBuild.result = 'SUCCESS'
}
catch(e) {
throw e;
}
finally {
resultString = currentBuild.result == 'SUCCESS' ? "successful" : "failed"
resultMessage = "${projectName} build ${buildName} ${resultString}"

//Optionally implement curl-based build finished notification here using resultMessage
}
}

Openshift: deployment/service/route.yml, Configmap

The other crucial part of this setup is the collection of files that describe the deployment. They are basically typical deployment/service/route Openshift configuration files. As an indication, these are my files:

deployment.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
apiVersion: v1
kind: DeploymentConfig
metadata:
labels:
app: DEPLOYNAME
name: DEPLOYNAME
namespace: my-project
spec:
replicas: 1
selector:
app: DEPLOYNAME
template:
metadata:
labels:
app: DEPLOYNAME
spec:
containers:
- env:
- name: PROFILE
value: dev
image: >- dev-reg-1/my-group/my-project:BRANCHNAME
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 20
livenessProbe:
httpGet:
path: /
port: 8080
name: my-project
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 0.2
memory: 1Gi
volumeMounts:
- mountPath: /etc/my-app.conf
name: my-app-config-prop
subPath: application.properties
volumes:
- configMap:
defaultMode: 420
name: my-app-config
name: my-app-config-prop
triggers:
- imageChangeParams:
automatic: true
containerNames:
- gds
from:
kind: ImageStreamTag
name: 'my-project:BRANCHNAME'
type: ImageChange

service.yaml:

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: Service
metadata:
name: DEPLOYNAME
namespace: my-group
spec:
selector:
app: DEPLOYNAME
ports:
- protocol: TCP
port: 8083

route.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: Route
metadata:
name: DEPLOYNAME
namespace: my-project
spec:
host: DEPLOYNAME.app.osft
to:
kind: Service
name: DEPLOYNAME
weight: 100
wildcardPolicy: None

Other files can be adjoined to these files, like ConfigMaps, secrets, nodeport services, etc… I have described some of these in previous articles.

Workflow

The workflow enabled by this setup should be straightforward by now:

  • Git pushes trigger a build.
  • By default, builds deploy to feature branches.
  • By using builds with parameters, there is an option to deploy to integration or pre-production deployments, as well as saving images to tarballs and uploading to an FTP server for separate production deployment.
  • Debugging with Openshift can hardly be automated unless you want to go with one pod everywhere (among other constraints). That’s why there is a choice to deploy to traditional dev servers for easier debugging.
  • Many improvements can be made, such as tagging builds or using various plugins. Jenkins has endless possibilities after all. But these are mostly not relevant to this post.

There is a lot of tweaking to be done before this setup can be functional, but when it runs it’s flawless.

You can also see these files on the Github repo. If you have any question or remark, leave a comment, or send me an email.