Skip to content

Creating Jenkins Jobs

Jump to bottom
Ben Heasly edited this page Feb 14, 2017 · 19 revisions

We have a Jenkins test server set up for testing and validating our code. The idea is that Jenkins can pull our code from GitHub, run tests on it, and report the results back to GitHub. You can also view detailed results at the Jenkins server's web interface.

Setting up the Jenkins server is described at Installing Jenkins. It should be up and running at this address: 50.112.42.141.

This page is about how to add a new job Jenkins server, for running tests on a new toolbox or project.

Creating a Jenkins Job

Prerequisites

You will need a Jenkins account. An admin will have to make one for you, perhaps David, Nicolas, or Ben Heasly. See Jenkins User Accounts

You also need to make sure that Jenkins can access your repository at GitHub. For the Brainard Lab, the easiest way to do this is to make sure that the GitHub user named render-toolbox is an Owner in your GitHub organization or Admin in the GitHub repository. render-toolbox is a "robot" user that we use just for this purpose, so we have already set up the GitHub-Jenkins communication for it.

We could set up other GitHub users, too. It would take a bit of fussing with the configuration. See [GitHub Integration](Installing Jenkins#GitHub Integration)

The Fast Way

The easiest way to create a new job is to find an existing job that's similar to what you want, copy it, and modify it.

To make the copy, log in to Jenkins, then:

  • New Item
    • enter a name for the new job
    • choose Freestyle project
    • for Copy from, enter the name of an existing job
    • choose OK

This will take you to a page with loads of things to configure. Most of it you won't need to touch. The things to change for your new job are probably:

  • General
    • Description
    • Project url for GitHub (this is the GitHub clone url, ending in .git)
  • Source Code Management
    • Repository URL (same as Project url, above)
    • Additional Behaviours -> Name of local subdirectory where you want repo to go
  • Build
    • Execute shell Command (see Generating a Job Script, below)
  • choose Save

The rest you can probably leave alone. Of course, you can customize the job however you like. See below for more about configuring jobs from scratch.

The Full Way

Jenkins is a popular and complex tool, with lots of ways to configure it! This is daunting. We don't use most of its features. You could spend all day clicking on things and wondering what they do.

Here are the steps to configure a Jenkins job from scratch, similar to several jobs that we already have working.

Log in to Jenkins, then:

  • New Item
    • enter a name for the new job
    • choose Freestyle project
    • choose OK

This will take you to a page with loads of things to configure. Here's what has worked for us so far:

  • General
    • add a Description -- a sentence about the job, like "Nightly validations for IBIOColorDetect."
    • check GitHub project
      • enter the Project url (the GitHub clone url, ending with .git)
  • Source Code Management
    • choose "Git"
      • enter the Repository URL (same as Project url above)
    • under Additional Behaviours choose Add -> Check out to a sub-directory
      • enter the Local subdirectory for repo, like "IBIOColorDetect"
  • Build Triggers (choose one)
    • check GitHub hook trigger for GITScm polling (run tests for each GitPush)
    • check Build periodically
      • enter a Schedule, like "@daily @midnight" (click the help button for loads of other options)
  • Build Environment
    • check Delete workspace before build starts
  • Build
    • choose Add build step -> Set build status to "pending" on GitHub commit
    • choose Add build step -> Execute shell
      • paste in a command (see Generating a Job Script, below)
  • Post-build Actions
    • choose Add post-build action -> Set GitHub commit status (universal)
      • under What -> Status Result, choose One of default messages and statuses
    • choose Add post-build action -> Publish TAP Results
      • under Publish TAP Results -> Test results, enter "**/*.tap"
    • choose Add post-build action -> Delete workspace when build is done
  • choose Save

Testing a Job

Once you have created you job, you can test it out and view the results.

Manual Trigger

The easiest way to test a job is to trigger it manually.

From the Jenkins main page:

  • choose the name of a job, like "IBIOColorDetect"
  • choose Build Now

This will add a new build to the Build Table below. The job should start running in a few seconds. Note the number of the job, like "#8".

Viewing the Console Output

You view details of a completed or running job. The richest detail is probably the console output from the job run.

From the Jenkins main page:

  • choose the name of a job, like "IBIOColorDetect"
  • from the Build History table, choose a job number, like "#8"
  • choose Console Output

This will be a big execution log. For completed jobs, you can scroll around and search. For running, jobs, Jenkins will stream the latest output into the window so you can follow its execution.

GitHub Trigger

If your job is configured to run in response to GitHub pushes, you may want to test out the Jenkins-GitHub integration. One way to do this would be to go to GitHub and edit the README file.

If you are logged in to GitHub, you can do this right in the browser:

  • go to your GitHub repository, like IBIOColorDetect
  • choose README.md in the list of files
  • choose the "pen" icon to edit this file
  • find a typo and fix it
  • choose Commit changes

This should trigger a Jenkins build within a few seconds. On the Jenkins main page, you should see your new job listed in the Build Executor Status table.

Generating a Job Script

Part of creating a Jenkins job is filling in the command to run for Build -> Execute shell. In general, this could be any command that you want Jenkins to run. For our purposes, we would like Jenkins to start up Matlab and invoke the automated tests for our toolbox or project.

Additionally, we would like Matlab to run the tests inside a Docker container. This allows each testing session to be isolated from previous or concurrent testing sessions. It also allows you to run tests locally using the same Docker environment that Jenkins uses. This should make Jenkins less of a special case and should reduce spurious test results that only seem to happen on Jenkins.

It takes several steps to set up a job like this:

  • decide the test command to run and its toolbox dependencies
  • locate Matlab on the host and make it available inside the Docker container
  • do other Docker container setup as needed, like locating Java
  • make the version of code that Jenkins is testing available inside the Docker container
  • run the job
  • make Matlab exit with a status code that indicates test success or failure

Yuck! That would be a bit much to write by hand!

MatlabJobSupport can help by generating a shell script that does all of the above for you. The idea is that you set up the tests you want to run by writing a short Matlab script. Then MatlabJobSupport prints out a shell script that you can paste into your Jenkins job.

An good example is the script mjsExampleIsetbioValidations.m. This generates a Jenkins command that looks like this:

#!/bin/sh
## Begin script generated by mjsWriteDockerRunScript.m

# embed the Matlab job as JSON
JOB_JSON="{\"cleanupCommand\": \"\",\"diskGB\": null,\"jobCommand\": \"ieValidateFullAllAssert\",\"memoryGB\": null,\"name\": \"validateIsetbio\",\"setupCommand\": \"\",\"toolboxCommand\": \"tbUse(''isetbio'')\"}"

# find where Matlab is installed, dynamically
MATLAB_LINK="$(which matlab)"
MATLAB_EXECUTABLE="$(readlink -f "$MATLAB_LINK")"
MATLAB_BIN_DIR="$(dirname "$MATLAB_EXECUTABLE")"
MATLAB_DIR="$(dirname "$MATLAB_BIN_DIR")"

# use the Java that comes with with Matlab
JAVA_HOME="/usr/local/MATLAB/from-host/sys/java/jre/glnxa64/jre"

# refresh the Docker image
docker pull "ninjaben/mjs-base:latest"

# invoke "docker run" with lots of options 
# using conventions established in ninjaben/mjs-base
docker run --rm --net=host \
  -v "$MATLAB_DIR":/usr/local/MATLAB/from-host \
  -v "$WORKSPACE":/mjs/projects \
  -v "$WORKSPACE":/opt/toolboxes \
  -e "INPUT_DIR=/var/mjs" \
  -e "OUTPUT_DIR=/var/mjs" \
  -e "WORKING_DIR=/var/mjs" \
  -e "JAVA_HOME=$JAVA_HOME" \
  ninjaben/mjs-base:latest \
  -r "mjsRunJobAndExit('$JOB_JSON');"

## End script generated by mjsWriteDockerRunScript.m

We have a few more examples like this, for generating Jenkins job scripts.

Matlab Integration

Matlab and Jenkins are different animals. But we can get Matlab-based tests to work well with Jenkins.

Mathworks has several blog posts on this topic:

From all of these posts, I took away two things:

  1. Our Matlab tests need to set the Matlab exit() status to be consistent with the test results:
  • all tests pass -> exit(0)
  • any test fails -> exit(-1), or any non-zero exit code
  1. We can get detailed test results by asking Matlab to write a file that follows the Test Anything Protocol, aka TAP. Jenkins can display these nicely with a table for each test run and a summary of recent test runs.

Setting the exit() Status

MatlabJobSupport includes a utility mjsRunJobAndExit(). This executes a Job, then exits Matlab with an appropriate exist status: 0, if the job finished normally, -1 if the job threw an exception.

As above, the shell scripts generated by MatlabJobSupport invoke mjsRunJobAndExit(). So that part is taken care of.

Writing TAP Results

Not all tests code will generate TAP results. But if we use Matlab's testing framework, we can get Matlab to generate the TAP results for us.

For an example of getting Matlab to write test results to a TAP file, see tbAssertTestsPass() from the ToolboxToolbox. Since ToolboxToolbox is included with MatlabJobSupport, this function will be available to all our test jobs.

In addition, tbAssertTestsPass() will work well with mjsRunJobAndExit(). If all tests pass, the assertion will succeed, the test job will finish normally, and Matlab will exit with status 0. If any test fails, the assertion will fail, the test job will throw an exception, and Matlab will exit with status -1.

Viewing TAP Results

When we installed Jenkins, we included the Jenkins TAP plugin. This allows Jenkins to read the TAP test results produced by Matlab and display them nicely to us in the browser.

Here's how to view a summary of recent test runs. From the Jenkins main page:

  • choose the name of a job, like "ToolboxToolbox"
  • there should be a nice line graph on the right hand side, with the title TAP Tests

Here's how to view a table of results from a single test run. From the Jenkins main page:

  • choose the name of a job, like "ToolboxToolbox"
  • from the Build History table, choose a job number, like "#"
  • choose TAP Test Results

Status Badges

Jenkins can generate status "badges" that indicate the success or failure of tests. These are nice things to display on other web sites. For example, we can put them in our GitHub README files to show the build status on the repo home page.

Jenkins provides an image link for the status badge of each project. It's up to Jenkins to update the image with the current build status. Other sites just display the image like any other.

We included the Embeddable Build Status Plugin when we installed Jenkins. So all you have to do is log in and copy the link for your project.

From the Jenkins main page:

  • choose the name of a job, like "ToolboxToolbox"
  • on the left, choose Embeddable Build Status
  • choose any "unprotected" URL

A good URL to choose is the one called "Markdown (with view)", "unprotected". You can paste this syntax directly into your GitHub README.

Emails

Jenkins can send emails about jobs. We enabled this when we installed Jenkins. You also need to set up the email config for each job. As always, there are many things to configure and get confused by.

The following config should work if you want to send emails whenever a job fails, to a fixed list of recipients and/or to the developers listed in the git logs.

From the Jenkins main page:

  • choose the name of a job, like "ToolboxToolbox"
  • on the left, choose Configure

This will take you to a page with loads of things to configure. Here's what has worked for us so far:

  • Post-build Actions
    • choose Add post-build action -> Editable Email Notification
      • under Project Recipient List, you can enter a list of email addresses separated by commas. This will work if you want to sent emails to a fixed list of recipients. Otherwise, just leave $DEFAULT_RECIPIENTS as it is.
      • under Default Content, enter a message. Include the variable $DEFAULT_CONTENT in the message to include useful info about the job.
      • choose Advanced Settings...
      • under Triggers -> Add Trigger, choose Failure - Any
        • under Send to -> Add, choose Recipient List (will be the list you typed above)
      • under Triggers -> Add Trigger, choose Failure - Any
        • under Send to -> Add, choose Developers (will be detected from git logs)
  • choose Save

Each time the job fails, Jenkins should send an email to each of the addresses that you entered into the Recipient List and/or the developers found in the Git logs.

Clone this wiki locally