Automatic Jenkins udpate using Shellscripts and Jenkins

What’s this article about?

Whoever has operated Jenkins manually knows this situation - you log into your Jenkins UI and are greeted by a nice notification that upgrades are available for your version. Since you’re operating Jenkins as part of your Infrastructure and it is configured to work mostly automated you look at the date at which the notification was raised and see that you’ve already missed at least two versions containing important security fixes. To counter that you could log into your Jenkins instance once or twice a day - or automate that task too. Depending on your setup you can use the auto-update method provided by Jenkins itself but usually servlets are - for obvious reasons - not allowed to overwrite themselves.

Since Jenkins is a critical component in my own deployments (it’s deploying configurations, firmware files on embedded devices in my own home- and lab automation systems as well as on external systems, it builds libraries for testing and release purposes, rebuilds applications as their dependencies change and redeploys them, builds my lecture notes as well as an unfinished book from LaTeX sources and releases them online, builds this webpage as well as some other webpages and deploys them, executes some periodic system maintenance jobs, tightly interacts with monitoring to re-bootstrap systems from zero with zero manual interaction in case of catastrophic failure, etc.) I decided that I really want to keep Jenkins up to date in an automatic fashion. Note that Jenkins is usually not trusted in my system but it’s also a component that’s included in the security relevant chain that also does signature verification on nearly every job that it’s executing when processing data retrieved from external systems such as GitHub.

The solution presented in this article is simple and consists of two simple shellscripts. One of them is invoked by a simple pipeline script on a periodic basis, the other is performing an update check and replaces the web application archive inside the servlet container if required. There are some drawbacks of this solution that will also be presented later on. Note that even though this sounds hackish it’s a solution that’s similar to what most auto-update systems already do.

The shellscripts

First why use two shellscripts and why is sudo required? On the setup I’m working on write access to web applications is limited to an webappsadmin user for obvious reasons. The web application archives can also be read by the tomcat servlet container user. The basic idea is to provide a script that can be launched passwordless with sudo and run with webappsadmin privileges by any other user on the system. This script will then check the current available Jenkins version, compare that to the currently installed version and if required fetch and simply deploy the new archive.

The jenkinsup script performing the actual update

The actual update script performs some simple steps:

• Fetch the latest version string from the jenkins webpage. This will be done by scraping the directory listing at http://mirrors.jenkins.io/war/latest/ and be contained inside a getLatestVersion function. The URI will be configured using a JENKINSVERSIONURI environment variable, the result will be stored in the LATESTVERSION variable. Note that this process will use a temporary file and it is assumed that this file is secure so no modifications are possible by untrusted components. This is done by running with webappsadmin privileges. Since my systems are running on FreeBSD I’ll use the omnipresent fetch instead of optional wget that I’d use on Linux.
• Then the latest version string will be compared with the recorded version inside a statefile that I’m storing at /var/db/jenkinsversion.dat. In case the file is not existing the upgrade will be enforced to bootstrap the script correctly. In any other case on any unequal version number the upgrade will be performed.
• I’ve implemented a logecho function that works somewhat like echo but redirects output into a logfile in case the JENKINSLOGFILE variable has been set.
• In case an update should be performed the web application archive is downloaded using fetch and then copied into the web application directory of Tomcat - since it’s the only servlet that I’m running I’m currently copying it to ROOT.war - but of course any other target can be configured. Simply atomatically exchanging the web application archive in the servlet container starts a re-deploy cycle on the application container (a pretty nice feature of most Java Servlet containers). Note that this leads to a short but noticeable service interruption. If one wants to avoid that one should do this on two redundant systems and use a load balancer or routing tricks while upgrading the system - or use an servlet container that’s capable of exchanging servlets without interruption.

#!/bin/sh

JENKINSVERSIONFILE=/var/db/jenkinsversion.dat
JENKINSTEMPTARGET=/tmp/jenkins_latest.war
JENKINSTARGET=/usr/local/apache-tomcat-9.0/webapps/ROOT.war

JENKINSVERSIONURI="http://mirrors.jenkins.io/war/latest/"
JENKINSDOWNLOADURI="http://mirrors.jenkins.io/war/latest/jenkins.war"

JENKINSLOGFILE=/var/log/jenkinsupdate.log
LOGVERBOSE=1

set -e

getLatestVersion() {
        fetch -o jenkinsversion.tmp "${JENKINSVERSIONURI}"
        LATESTVERSION=`cat jenkinsversion.tmp | grep 'jenkins.war</a>' | awk -F 'right">' '{ print $2; }' | awk -F ' </td>' '{ print $1; }'`
        rm jenkinsversion.tmp
}

logecho() {
        if [ "${JENKINSLOGFILE}" == "" ]; then
                echo ${1}
        else
                echo ${1} >> ${JENKINSLOGFILE}
        fi
}

# Get latest version and verify if it has changed (/var/db/jenkinsver.dat)

getLatestVersion

UPDATE=0
if [ ! -e ${JENKINSVERSIONFILE} ]; then
        UPDATE=1
else
        KNOWNVERSION=`cat ${JENKINSVERSIONFILE}`
        if [ "${KNOWNVERSION}" == "${LATESTVERSION}" ]; then
                if [ ${LOGVERBOSE} -gt 0 ]; then
                        logecho "Version ${KNOWNVERSION} already known, not updating"
                fi
        else
                UPDATE=1
        fi
fi

if [ ${UPDATE} -eq 1 ]; then
        logecho "Trying to update to ${LATESTVERSION}"

        fetch -o ${JENKINSTEMPTARGET} "${JENKINSDOWNLOADURI}"
        echo "${LATESTVERSION}" > ${JENKINSVERSIONFILE}
        logecho "Fetched successfully, Deploying"

        cp ${JENKINSTEMPTARGET} ${JENKINSTARGET}
        logecho "Done"
        logecho " "
fi

As usual the script has to be made executable and since we’re also running it using sudo later on it should be read only. I’ve stored the script at /root/tools/jenkins/jenkinsup. In this case one requires

chmod 555 /root/tools/jenkins/jenkinsup

Running using cron

This script is already sufficient for upgrading Jenkins on a periodic basis. One could simply add that script to /etc/crontab to check daily for updates:

15	05	*	*	*	webappsadmin	/root/tools/jenkins/jenkinsup

Running using Jenkins

It might also be nice to run the update script using Jenkins itself because of different trigger methods (cron, AMQP/MQTT messages, REST requests, etc.) and the ability to execute the job only when all nodes are idle. In this case I’m currently using a separate script /root/tools/jenkins/runjenkinsup that’s simply executing the script using sudo:

#!/bin/sh

sudo -u webapps nohup /root/tools/jenkins/jenkinsup &

As usual the script has to be owned by the administrative user, made executable and read only.

To work correctly the tomcat user - called wwww on my deployment - will be required to passwordless perform that sudo call into the webapps context. To allow that one has to modify /usr/local/etc/sudoers which is usually done using the visudo command. Then one has to append the line

www ALL=(webappsadmin) NOPASSWD: /root/tools/jenkins/jenkinsup

This allows the www user to execute the /root/tools/jenkins/jenkinsup script without any password prompt as webappsadmin and thus introduces the required privilege escalation channel to bypass security compartmentation between the administrative and operative users.

Now one just has to configure a Jenkins Job with the required triggers. For example one could realize the same as the above shown cronjob by setting a time sheduled build with the specification

H 5 * * *

The pipeline script is really simple:

pipeline {
    agent {
        label 'master'
    }
    stages {
        stage('Execute upgrade script') {
            steps {
                sh '/root/tools/jenkins/runjenkinsup'
            }
        }
    }
}

This article is tagged:

• Basics
• System administration
• Home automation
• Tutorial