Jenkins Guide

The ci-management or releng/builder repos in an LF project consolidates the Jenkins jobs from project-specific VMs to a single Jenkins server. Each Git repo in every project has a view for their jobs on the main Jenkins server. The system utilizes Jenkins Job Builder for the creation and management of the Jenkins jobs.

Jenkins job project views.

Quick Start

This section provides details on how to create jobs for new projects with minimal steps. All users in need to create or contribute to new job types should read and understand this guide.

As a new project you will be mainly interested in getting your jobs to appear in the Jenkins server silo archiving it by creating a <project>.yaml in the releng/builder or ci-management project’s jjb directory.

Example for releng/builder projects:

git clone --recursive https://git.opendaylight.org/gerrit/releng/builder
cd builder
mkdir jjb/<new-project>

Example for ci-management projects:

git clone --recursive https://gerrit.onap.org/gerrit/ci-management
cd ci-management
mkdir jjb/<new-project>

Where <new-project> should be the same name as your project’s Git repo in Gerrit. If your project name is “aaa” then create a new jjb/aaa directory.

Note

In similar matter, if your project name is “aaa/bbb” then create a new jjb/aaa-bbb directory by replacing all “/” with “-“.

Note

builder/jjb/global-jjb or ci-management/jjb/global-jjb are submodules of releng/builder or ci-management repositories which require a git submodule update --init or using –recursive with git clone to get them fetched.

Next we will create <new-project>.yaml as follows:

---
- project:
    name: <new-project>
    project-name: <new-project>
    project: <new-project>
    mvn-settings: <new-project>-settings
    jobs:
      - gerrit-maven-clm
      - gerrit-maven-merge
      - gerrit-maven-release
      - gerrit-maven-verify
      - gerrit-maven-verify-dependencies

    stream: master

- project:
    name: <new-project>-sonar
    jobs:
      - gerrit-maven-sonar
    build-node: centos7-builder-4c-4g
    project: <new-project>
    project-name: <new-project>
    branch: master
    mvn-settings: <new-project>-settings

Replace all instances of <new-project> with the name of your project as explained before.

The template above shows how to add each job from global-jjb. We recommend defining a local job-group for the project or defining each job needed in a list.

Add the following jobs for minimal setup on a Maven based project:

- gerrit-maven-clm
- gerrit-maven-merge
- gerrit-maven-release
- gerrit-maven-verify
- gerrit-maven-sonar

Optionally, you can add other jobs as well:

- gerrit-maven-verify-dependencies

Global-jjb defines groups of jobs recommended for ci, maven, python, node, rtd and more future languages as global-jjb is always under constant improvement. If you would like to explore more about these options available please refer to the Global JJB Templates section.

The changes to these files get published in Gerrit and reviewed by the releng/builder or ci-management teams for the LF project. After approvals, these Gerrits get merged and the jobs published in Jenkins.

git add jjb/<new-project>
git commit -sm "Add <new-project> jobs to Jenkins"
git review

This will push the jobs to Gerrit and your jobs will appear in Jenkins once the releng/builder or ci-management teams has reviewed and merged your patch.

Build Minions

Log Server

While Jenkins stores the console logs on the Jenkins instance, this is short term and typically depending on the job type purged after a week. We highly recommend preferring the log server over the Jenkins system when viewing console logs as it reduces load on Jenkins and we compress logs stored on the log server so downloads are faster.

We store Log server archives for 6 months.

At the end of a build the job ships logs to a Nexus logs repo and can be conveniently accessed via the https://logs.example.org URL. The Job Build Description will contain the specific log server URL for a build log. Jobs triggered via Gerrit Trigger will have the URL to the logs left as a post build comment.

Example Jenkins Build Description:

Build logs: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/distribution-check-carbon/167

Example Gerrit Comment:

jenkins-releng                                                  03-05 16:24
Patch Set 6: Verified+1
Build Successful
https://jenkins.opendaylight.org/releng/job/builder-tox-verify-master/1066/ : SUCCESS
Logs: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-tox-verify-master/1066

The log path pattern for the logs server is LOG_SERVER_URL/SILO/JENKINS_HOSTNAME/JOB_NAME/BUILD_NUMBER typically if you know the JOB_NAME and BUILD_NUMBER you can replace the paths before it to convert the URL between Jenkins and the Log server.

We compress and store individual log files in gzip (.gz) format on the Nexus log repository. You can access these files through the URL.

  • Jenkins Production: https://logs.example.org/production
  • Jenkins Sandbox: https://logs.example.org/sandbox

Log Cleanup Schedule

The log servers are setup with cron jobs that purge logs during regular scheduled intervals.

  • Jenkins Production: Delete logs everyday at 08:00 UTC which are older than 180 days.
  • Jenkins Sandbox: Delete logs and jobs every week on Saturday at 08:00 UTC.

Jenkins Job Builder

Jenkins Job Builder takes simple descriptions of Jenkins jobs in YAML format and uses them to configure Jenkins.

JJB Overview

Jenkins Job Builder translates YAML code to job configuration suitable for consumption by Jenkins. When testing new Jenkins Jobs in the Jenkins Sandbox, you will need to use the jenkins-jobs executable to translate a set of jobs into their XML descriptions and upload them to the Jenkins Sandbox server.

Install JJB

You can install the latest version of JJB and its dependencies with pip using Python Virtual Environments or lock a specific version of JJB in jjb/requirements.txt, as a workaround for known issues. The documentation is available in pip-assisted install.

Virtual Environments

For pip-assisted, we recommend using Python Virtual Environments to manage JJB and it’s Python dependencies.

The documentation to install virtual environments with virtualenvwrapper. On Linux systems with pip run:

pip install --user virtualenvwrapper

A virtual environment is a directory that you install Python programs into and update the shell’s $PATH, which allows the version installed in the virtual environment to take precedence over any system-wide versions available.

Create a new virtual environment for JJB.

virtualenv jjb

With in your virtual environment active, you can install JJB which is visible when the virtual environment that is active.

To activate your virtual environment.

source ./jjb/bin/activate
# or
workon jjb

To deactivate your virtual environment.

deactivate

Install JJB using pip

To install JJB and its dependencies, make sure you have created and activated a virtual environment for JJB.

  1. Set a virtualenv

    virtualenv jjb
    source jjb/bin/activate
    
  2. Install JJB

    pip install jenkins-job-builder==2.0.5
    

    Note

    If a requirements.txt exists in the repository with the recommended JJB version then, use the requirements file to install JJB by calling.

    # From the root of the ci-management or builder directory
    pip install -r jjb/requirements.txt
    

    To change the version of JJB specified by jjb/requirements.txt to install from the latest commit to the master branch of JJB’s Git repository:

    cat jjb/requirements.txt
    -e git+https://git.openstack.org/openstack-infra/jenkins-job-builder#egg=jenkins-job-builder
    
  3. Check JJB installation:

    jenkins-jobs --version
    

Global JJB Templates

Global-JJB is a library project containing reusable Jenkins Job Builder templates. The intention is to save time for projects from having to define their own job templates. Documentation is available via global-jjb documentation specific sections of interest linked here:

Note

For infra admins, the CI Job Templates contain useful jobs for managing Jenkins and VM Images. We recommend to deploy these jobs to all new infra projects.

Packer Images

The ci-management repo contains a directory called packer which contains scripts for building images used by Jenkins to spawn builders. There are 2 files necessary for constructing a new image:

  1. packer/templates/BUILDER.json
  2. packer/provision/BUILDER.yaml

Replace BUILDER with the name of your desired builder image type.

The templates file contains packer configuration information for building the image. The provision file is a script for running commands inside the packer-builder to construct the image. We recommend using the Ansible provisioner as that is the standard used by LF packer builds.

While developing a new builder image type, we can use the lfdocs-jenkins-sandbox to build and deploy the image for testing. Configure a Jenkins Job the new image type using the global-jjb gerrit-packer-merge job template.

Example job definition:

- project:
  name: packer-robot-jobs
  jobs:
    - gerrit-packer-merge

  project: releng/builder
  project-name: builder
  branch: master
  archive-artifacts: '**/*.log'

  build-node: centos7-builder-2c-1g

  platforms: centos-7
  templates: robot

The gerrit-packer-merge job creates jobs in the format PROJECT_NAME-packer-merge-PLATFORM-TEMPLATE. Where PROJECT_NAME is the project-name field, PLATFORM is the platforms field, and TEMPLATES is the templates field in the yaml above. In this example the resultant job is builder-packer-merge-centos-7-robot.

Follow the instructions in the Pushing a patch to Gerrit section to push this job to the Sandbox.

Once the job is on the Jenkins Sandbox, run the job and it will attempt to deploy the new image and make it available. Once the job completes look for a line in the logs that look like:

==> vexxhost: Creating the image: ZZCI - CentOS 7 - robot - 20180301-1004

This line provides the name of the new image we built.

Jenkins Production & Jenkins Sandbox

The Jenkins server is the home for all project’s Jenkins jobs. Most of the job configuration gets managed through code using JJB through the ci-management or releng/builder repos.

To access the Jenkins Production URL for any project use: https://jenkins.PROJECT_DOMAIN

Similarily, the project’s corresponding Jenkins Sandbox URL would be: https://jenkins.PROJECT_DOMAIN/sandbox

Any users with an LFID can access the Jenkins Production site, but for Jenkins Sandbox please request an access. To do so, refer to the section Get access to the Jenkins Sandbox

Project contributors do not edit the Jenkins jobs directly on the Jenkins production server. Instead, we encourage them to use the Jenkins Sandbox.

The Jenkins Sandbox has similar configuration to the production instance. Jenkins Sandbos does not publish artifacts in Nexus or Nexus3 or vote in Gerrit which makes it a safe environment to test the jobs. Users can edit and trigger the jobs directly to test the behaviour.

The Jenkins Sandbox can contain dummy configuration files and dummy credentials in case it helps take the test further and not fail on the first steps due to the configuration not being present. Any attempt to actually use the configuration files in order to make any server comunications will fail. To add dummy configuration files, please create a new ticket to Helpdesk.

In such case, merge jobs, push, CLM, Docker or Sonar jobs get tested to some extent due to this limitation. Once the job template gets merged and becomes available in Jenkins Production, we can confirm the jobs are actually making server comunications as expected with Nexus-IQ, Sonar, Gerrit or Nexus.

The Sandbox has limited amount of Virtual Machine nodes instances to test compared to production Jenkins.

Jenkins Sandbox

Sandbox Overview

Facts to keep in mind before working on the Sandbox:

  • Jobs are automatically deleted every Saturday at 08:00 UTC
  • Committers can login and configure Jenkins jobs in the Sandbox directly
  • Sandbox jobs cannot perform any upload/deploy tasks
  • There are no project configuration files and project credentials loaded into the system
  • Sandbox jobs cannot vote on Gerrit
  • Jenkins nodes have OpenStack configuration similarly to the production instance

Get access to the Jenkins Sandbox

The Sandbox provides a testing/experimentation environment used before pushing job templates to the production instance.

To access the Sandbox use: jenkins.<project-domain>/sandbox

The access to the Sandbox uses the same LFID used in the production Jenkins instance, but in this case a new LF Helpdesk ticket (for the related project) needs creation to request the sanbox access.

The LF helpdesk team can add users to the appropriate group to grant permissions to access the Sandbox via https://identity.linuxfoundation.org/. The group that controls this access is <project>-jenkins-sandbox-access For example: https://identity.linuxfoundation.org/content/<project>-jenkins-sandbox-access

The requester will receive an invitation to join this group. Once accepted, the user can now access the Sandbox same way as the production Jenkins.

Push jobs to Jenkins Sandbox

Push jobs to the Jenkins Sandbox using one of these methods:

  1. Via Gerrit Comment
  2. Via JJB CLI

Method 1 is easier as it does not require installing anything on your local system. This method requires pushing the patch to Gerrit on each test. We recommend this method for quick one off edits or if you are testing another contributor’s patch.

Method 2 is more convenient for those who work on JJB templates more than once or twice.

Push jobs via Gerrit comment

This is the easiest and fastest way to start using the Sandbox. This is the recommended default way to use the Sandbox since this does not require the user to install JJB or configure it at all.

This is the recommended way to push jobs to the Sandbox system and does not require installation of Jenkins Job Builder locally.

To push jobs to the Sandbox with jjb-deploy, add a comment on the Gerrit patch from ci-management:

jjb-deploy <job name>

The resultant job’s configuration reflects the same code the patch’s code base in the Gerrit. The job pushed into the Sandbox will reflect the changes made in the patch.

Note

You can use * wildcard for job names. This is not a good practice. We highly recommended to use specific Jenkins job names instead.

Push jobs via JJB CLI

JJB CLI needs configuration first.

Note

Use this configuration if you prefer to use the JJB tool locally on your system.

After getting access to the Sanbox group, configure the following.

Create a jenkins.ini with the following contents modifying the relevant data:

Example:

;<jenkins.ini contents>

[job_builder]
ignore_cache=True
keep_descriptions=False
recursive=True

[jenkins]
user-id=<Provide your Jenkins Sandbox user-id (LFID)>
password= <Refer below steps to get API token>
url=https://jenkins.<project-domain>/sandbox
ignore_cache=True

How to retrieve API token? Login to the Jenkins Sandbox using your LFID, go to the user page by clicking on your username. Click Configure and then click Show API Token.

To start using the Sandbox, we must do a clone of ci-management or releng/builder (in case of ODL) repo for the project. For example:

git clone ssh://<LFID>@gerrit.<project-domain>:29418/ci-management

Make sure you sync global-jjb also using:

git submodule update --init

Install JJB (Jenkins Job Builder).

Execute the following commands to install JJB on your machine:

cd ci-management (or cd builder)
sudo pip install virtualenvwrapper
mkvirtualenv jjb
pip install jenkins-job-builder
jenkins-jobs --version
jenkins-jobs test --recursive jjb/

Note

More information on Python Virtual Environments

To work on existing jobs or create new jobs, navigate to the /jjb directory where you will find all job templates for the project. Follow the below commands to test, push or delete jobs in your Sandbox environment.

Verify JJB

After you edit or create new job templates, test the job in the Sandbox environment before you submit this job to production CI environment.

jenkins-jobs --conf jenkins.ini test jjb/ <job-name>

For Example:

jenkins-jobs --conf jenkins.ini test jjb/ ci-management-jjb-merge

If the job you would like to test is a template with variables in its name, it must be manually expanded before use. For example, the commonly used template {project-name}-jjb-merge might expand to ci-management-jjb-merge.

A successful test will output the XML description of the Jenkins job described by the specified JJB job name.

Execute the following command to pipe-out to a directory:

jenkins-jobs --conf jenkins.ini test jjb/ <job-name> -o target

The output directory will contain files with the XML configurations.

Push a Job

Ensure you have configured your jenkins.ini and verified it by outputting valid XML descriptions of Jenkins jobs. Upon successful verification, execute the following command to push the job to the Sandbox:

jenkins-jobs --conf jenkins.ini update jjb/ <job-name>

For Example:

jenkins-jobs --conf jenkins.ini update jjb/ ci-management-jjb-merge

Delete a Job

Execute the following command to Delete a job from Sandbox:

jenkins-jobs --conf jenkins.ini delete jjb/ <job-name>

For Example:

jenkins-jobs --conf jenkins.ini delete jjb/ ci-management-jjb-merge

You can also delete the job from the UI options in Jenkins Sandbox.

Edit Job via Web UI

In the Sandbox, you can directly edit the job configuration by selecting the job name and clicking on the Configure button. Click the Apply or Save (to save and exit the configuration) buttons to save the job.

This is useful in the case where you might want to test quick tweaks to a job before modifying the YAML.

Edit the job in your terminal and follow the described steps in Verify JJB and Push Job <push-job> to push any changes and have them ready to push to Gerrit.

Important

When pushing to the Sandbox with jenkins-jobs, do not forget the <job-name> parameter. Otherwise, JJB will push all job templates into the Sandbox and will flood the system.

If that happens, use `ctrl+c` to cancel the upload.

A successful run of the desired job will look like this:

INFO:jenkins_jobs.builder:Number of jobs generated:  1

Execute jobs in the Sandbox

Once you push the Jenkins job configuration to the Sandbox environment, run the job from the Sandbox WebUI. Follow the below process to trigger the build:

  1. Login into the Jenkins Sandbox WebUI
  2. Click on the job which you want to trigger
  3. Click “Build with parameters”
  4. Click Build
  5. Verify the Build Executor Status bar to check on progress.

You can click on the build number to view the job details and console output.