Automatic Jenkins udpate using Shellscripts and Jenkins

13 Oct 2020 - tsp
Last update 13 Oct 2020

__TL;DR__: Auto update Jenkins in Tomcat servlet container by fetching version and web application archive using shellscripts, bypassing user compartmentation between administrative and operative users using sudo and running the jobs using cron or Jenkins.

What’s this article about?

Whoever has operated Jenkins manually knows this situation - you log into your Jenkins UI and are greeted by a nice notification that upgrades are available for your version. Since you’re operating Jenkins as part of your Infrastructure and it is configured to work mostly automated you look at the date at which the notification was raised and see that you’ve already missed at least two versions containing important security fixes. To counter that you could log into your Jenkins instance once or twice a day - or automate that task too. Depending on your setup you can use the auto-update method provided by Jenkins itself but usually servlets are - for obvious reasons - not allowed to overwrite themselves.

Since Jenkins is a critical component in my own deployments (it’s deploying configurations, firmware files on embedded devices in my own home- and lab automation systems as well as on external systems, it builds libraries for testing and release purposes, rebuilds applications as their dependencies change and redeploys them, builds my lecture notes as well as an unfinished book from LaTeX sources and releases them online, builds this webpage as well as some other webpages and deploys them, executes some periodic system maintenance jobs, tightly interacts with monitoring to re-bootstrap systems from zero with zero manual interaction in case of catastrophic failure, etc.) I decided that I really want to keep Jenkins up to date in an automatic fashion. Note that Jenkins is usually not trusted in my system but it’s also a component that’s included in the security relevant chain that also does signature verification on nearly every job that it’s executing when processing data retrieved from external systems such as GitHub.

The solution presented in this article is simple and consists of two simple shellscripts. One of them is invoked by a simple pipeline script on a periodic basis, the other is performing an update check and replaces the web application archive inside the servlet container if required. There are some drawbacks of this solution that will also be presented later on. Note that even though this sounds hackish it’s a solution that’s similar to what most auto-update systems already do.

The shellscripts

First why use two shellscripts and why is sudo required? On the setup I’m working on write access to web applications is limited to an webappsadmin user for obvious reasons. The web application archives can also be read by the tomcat servlet container user. The basic idea is to provide a script that can be launched passwordless with sudo and run with webappsadmin privileges by any other user on the system. This script will then check the current available Jenkins version, compare that to the currently installed version and if required fetch and simply deploy the new archive.

The jenkinsup script performing the actual update

The actual update script performs some simple steps:

#!/bin/sh

JENKINSVERSIONFILE=/var/db/jenkinsversion.dat
JENKINSTEMPTARGET=/tmp/jenkins_latest.war
JENKINSTARGET=/usr/local/apache-tomcat-9.0/webapps/ROOT.war

JENKINSVERSIONURI="http://mirrors.jenkins.io/war/latest/"
JENKINSDOWNLOADURI="http://mirrors.jenkins.io/war/latest/jenkins.war"

JENKINSLOGFILE=/var/log/jenkinsupdate.log
LOGVERBOSE=1

set -e

getLatestVersion() {
        fetch -o jenkinsversion.tmp "${JENKINSVERSIONURI}"
        LATESTVERSION=`cat jenkinsversion.tmp | grep 'jenkins.war</a>' | awk -F 'right">' '{ print $2; }' | awk -F ' </td>' '{ print $1; }'`
        rm jenkinsversion.tmp
}

logecho() {
        if [ "${JENKINSLOGFILE}" == "" ]; then
                echo ${1}
        else
                echo ${1} >> ${JENKINSLOGFILE}
        fi
}

# Get latest version and verify if it has changed (/var/db/jenkinsver.dat)

getLatestVersion

UPDATE=0
if [ ! -e ${JENKINSVERSIONFILE} ]; then
        UPDATE=1
else
        KNOWNVERSION=`cat ${JENKINSVERSIONFILE}`
        if [ "${KNOWNVERSION}" == "${LATESTVERSION}" ]; then
                if [ ${LOGVERBOSE} -gt 0 ]; then
                        logecho "Version ${KNOWNVERSION} already known, not updating"
                fi
        else
                UPDATE=1
        fi
fi

if [ ${UPDATE} -eq 1 ]; then
        logecho "Trying to update to ${LATESTVERSION}"

        fetch -o ${JENKINSTEMPTARGET} "${JENKINSDOWNLOADURI}"
        echo "${LATESTVERSION}" > ${JENKINSVERSIONFILE}
        logecho "Fetched successfully, Deploying"

        cp ${JENKINSTEMPTARGET} ${JENKINSTARGET}
        logecho "Done"
        logecho " "
fi

As usual the script has to be made executable and since we’re also running it using sudo later on it should be read only. I’ve stored the script at /root/tools/jenkins/jenkinsup. In this case one requires

chmod 555 /root/tools/jenkins/jenkinsup

Running using cron

This script is already sufficient for upgrading Jenkins on a periodic basis. One could simply add that script to /etc/crontab to check daily for updates:

15	05	*	*	*	webappsadmin	/root/tools/jenkins/jenkinsup

Running using Jenkins

It might also be nice to run the update script using Jenkins itself because of different trigger methods (cron, AMQP/MQTT messages, REST requests, etc.) and the ability to execute the job only when all nodes are idle. In this case I’m currently using a separate script /root/tools/jenkins/runjenkinsup that’s simply executing the script using sudo:

#!/bin/sh

sudo -u webapps nohup /root/tools/jenkins/jenkinsup &

As usual the script has to be owned by the administrative user, made executable and read only.

To work correctly the tomcat user - called wwww on my deployment - will be required to passwordless perform that sudo call into the webapps context. To allow that one has to modify /usr/local/etc/sudoers which is usually done using the visudo command. Then one has to append the line

www ALL=(webappsadmin) NOPASSWD: /root/tools/jenkins/jenkinsup

This allows the www user to execute the /root/tools/jenkins/jenkinsup script without any password prompt as webappsadmin and thus introduces the required privilege escalation channel to bypass security compartmentation between the administrative and operative users.

Now one just has to configure a Jenkins Job with the required triggers. For example one could realize the same as the above shown cronjob by setting a time sheduled build with the specification

H 5 * * *

The pipeline script is really simple:

pipeline {
    agent {
        label 'master'
    }
    stages {
        stage('Execute upgrade script') {
            steps {
                sh '/root/tools/jenkins/runjenkinsup'
            }
        }
    }
}

This article is tagged:


Data protection policy

Dipl.-Ing. Thomas Spielauer, Wien (webcomplains389t48957@tspi.at)

This webpage is also available via TOR at http://jugujbrirx3irwyx.onion/

Valid HTML 4.01 Strict Powered by FreeBSD IPv6 support