Skip to content

Installing Jenkins

Jump to bottom
Gaylord Holder edited this page Apr 3, 2018 · 27 revisions

One good use for MatlabJobSupport will be to run automated tests and validations on a remote Jenkins server. This page is about how to install Jenkins and integrate with GitHub and Matlab.

See Creating Jenkins Jobs for more about setting up tests for specific projects.

EC2 Instance

We need a server to run Jenkins. In order for Jenkins to run MatlabJobSupport jobs, the server also needs to have Matlab and Docker installed.

In the Brainard Lab AWS account, we have machine image all set up for this. It's the AMI called RTB Jobs 9.

For these instructions, I will refer to an instance that I just now set up and I hope to leave running. Here are its stats:

  • AMI RTB Jobs 9
  • instance type m4.large
  • IAM role ecsInstanceRole
  • 50GB "GP2" SSD root volume
  • Name = Jenkins
  • security sroups 'default', all-ssh, and web
  • SSH key pair named render-toolbox
  • elastic IP address 50.112.42.141

I can SSH to this instance with:

ssh -i render-tooblox.pem [email protected]

Jenkins Setup

To set up Jenkins on this instance, I am following the Jenkins wiki. I used the sections "Installation" and "Using Linux iptables for port 80 -> 8080", just as written.

Installation

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

Using Linux iptables for port 80 -> 8080

#Requests from outside
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
#Requests from localhost
sudo iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080

Browser Test

Then I was able to access Jenkins at its elastic IP address. Here.

Initial Password

Jenkins starts with a special initial password. I was able to discover this from the SSH terminal using either of these commands:

# either
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

# or
cat /var/log/jenkins/jenkins.log

Jenkins Startup

After entering the initial password, I followed the Jenkins startup wizard. I chose the default set of plugins and created an admin user with a strong password.

Docker Group

For MatlabJobSupport, we want Jenkins to be able to invoke Docker commands. So we need to put the jenkins user into the docker group. This requires a restart of Jenkins and Docker (but not the whole system).

sudo gpasswd -a jenkins docker
sudo service docker restart
sudo service jenkins restart

Jenkins Plugins

Finally, I installed some more Jenkins plugins that will come in handy.

  • choose Manage Jenkins -> Manage Plugins -> Available
  • check Embeddable Build Status Plugin
  • check TAP Plugin
  • check Role-based Authorization Strategy
  • choose Install without Restart

GitHub Integration

Now that Jenkins is set up, I am connecting it to GitHub. I'm taking advice from a blog called The Right Code, and the Jenkins GitHub plugin

Create SSH Key Pair for jenkins User

We need an SSH key pair for the jenkins user so that Jenkins can authenticate with GitHub.

sudo passwd jenkins
# enter a password
su jenkins
# enter that password
ssh-keygen -t rsa -b 4096 -C "[email protected]"
# enter that password
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
# enter that password
exit

Put the SSH public key on GitHub

Now that we have the key pair, we can register the public key with GitHub.

I am using our "render.toolbox" GitHub account. But there is nothing special about that account. The same should work with any account that has access to our GitHub organizations. The only requirement is that the user must have write permissions for the repositories (or organizations) that we want Jenkins to test.

su jenkins
# enter that password
cat ~/.ssh/id_rsa.pub
# copy it
exit
# GitHub log in as [email protected]
# paste in new ssh key

Now log in to GitHub as "[email protected]"

  • choose Settings -> SSH and GPG Keys
  • choose New SSH key
  • Title "Jenkins"
  • paste in the key from above
  • choose Add SSH key

Create a GitHub Access Token for Jenkins

From GitHub, create a Person Access Token for Jenkins to use. This will allow Jenkins to get notified about GitHub commits and and to set build statuses that we can see at GitHub.

Log in to GitHub as "[email protected]"

  • choose Settings -> Personal access tokens
  • choose Generate new token
  • Token description "Jenkins"
  • Select scopes:
    • repo
    • admin:org
    • admin:repo_hook
    • admin:org_hook
    • notifications
  • choose Generate token
  • copy the new token to the clipboard

Put Access Token into Jenkins

Now put the token generated above into Jenkins

  • choose Manage Jenkins -> Configure System -> GitHub -> Add GitHub Server
  • choose Credentials -> Add -> Jenkins
  • choose Kind -> Secret Text
  • Secret, paste in the token copied above
  • ID "GitHub"
  • choose Add
  • in Credentials drop-down, select "GitHub"
  • choose Test Connection
  • choose Save

Public Access

We would like the public to be able to view things like test results and [status badges](Creating Jenkins Jobs#Status Badges). This will help with debugging and communication.

I found that I had to do a little configuration to make these things readable by the public.

First I choose a security strategy that is configurable:

  • choose Manage Jenkins -> Configure Global Security
  • choose Access Control -> Authorization -> Role-Based Strategy
  • choose Save

Then I created an "anonymous" Role that is allowed to view results and badges:

  • choose Manage Jenkins -> Manage and Assign Roles -> Manage Roles
  • under Global Roles, enter in Role to add "anonymous"
  • choose Add
  • under Global Roles -> anonymous, check several permissions:
    • Overall: Read
    • Credentials: View
    • Job: Read, ViewStatus, Workspace
    • View: Read
  • choose Save

The I assigned this "anonymous" role to public aka "Anonymous" User

  • Manage Jenkins -> Manage and Assign Roles -> Assign Roles
  • under Global Roles -> Anonymous, check "anonymous"
  • choose Save

Jenkins User Accounts

We would like registered users to be able to create jobs, trigger jobs, etc. Here's how a Jenkins admin can create a new Jenkins user account.

First create the User:

  • Manage Jenkins -> Manage Users -> Create User
    • fill out the form
    • choose Create User

Then assign a Role that tells Jenkins what the User is allowed to do:

  • Manage Jenkins -> Manage and Assign Roles -> Assign Roles
    • under Global roles -> User/group to add, enter the user name of the new user
    • choose Add
    • in the Global roles table, check the "admin" box next to the name of the new user
    • choose Save

Jenkins Emails

Jenkins can send us emails!

Sendmail SMTP Setup

For this to work, we need an email service. brainard-jenkins uses the sendmail package to provide a localhost, send-only email server.

Install the packages

  • sendmail-cf
  • heirloom-mailx

Edit the file /etc/mail/sendmail.mc At the end of the file you should have a set of configuration options that look something like:

MASQUERADE_AS(`brainard-jenkins.psych.upenn.edu')dnl
dnl #
dnl # masquerade not just the headers, but the envelope as well
dnl #
FEATURE(masquerade_envelope)dnl
dnl #
dnl # masquerade not just @mydomainalias.com, but @*.mydomainalias.com as well
dnl #
FEATURE(masquerade_entire_domain)dnl
dnl #
MASQUERADE_DOMAIN(localhost)dnl
MASQUERADE_DOMAIN(localhost.localdomain)dnl
MASQUERADE_DOMAIN(ec2-50-112-42-141.us-west-2.compute.amazonaws.com)dnl
MASQUERADE_DOMAIN(amazonaws.com)dnl
dnl MASQUERADE_DOMAIN(mydomain.lan)dnl
MAILER(smtp)dnl
MAILER(procmail)dnl

Adapt for your instance.

Enable the service so it starts on boot.

Start the sendmail service

service sendmail start


Test mail is working:

mailx -s 'testing' [email protected] < /dev/null

Use the ``mailq`` command and tail the /var/log/maillog files to verify delivery.
This sort of email often gets treated as SPAM.

## Jenkins SMTP Setup

Here's what I did, which seems to be working:
 - Manage Jenkins -> Configure System
 - under Extended E-mail Notification:
   - Make sure ``SMTP server`` is blank
   - choose Advanced
   - verify ``Use SMTP Authentication`` is _unchecked_
   - choose Save

## Jenkins Git Setup
Jenkins can also ready the Git commit logs to find the emails of people who contributed to our projects.  This allows Jenkins to email all contributors, even ones who don't have accounts on the Jenkins server.  It's cool.

To enable this:
 - Manage Jenkins -> Configure System
 - under Git plugin, check Create new accounts base on author/committer's email
 - choose Save

# Jenkins Updates
Sometimes Jenkins will show a big red notification asking you to update to a newer version.  This turned out to be pretty easy to do, following advice on the Jenkins Wiki about [Debian Package Upgrade](https://wiki.jenkins-ci.org/display/JENKINS/Automated+Upgrade)

SSH to the Jenkins server like this (or similar):

ssh -i render-tooblox.pem [email protected]


Then just update the `jenkins` package and install:

sudo apt-get update sudo apt-get install jenkins


This "just worked".  Nice.
Clone this wiki locally