Job Groups
Job groups are a great tool to configure categories of jobs together at the
same time. Below the example are some starting point job-groups but we
recommend creating your own to ensure that the jobs configured reflect the
project’s needs.
An example project:
- job-group:
name: odl-maven-jobs
jobs:
- gerrit-maven-clm
- gerrit-maven-merge
- gerrit-maven-release
- gerrit-maven-verify
- gerrit-maven-verify-dependencies:
build-timeout: 180
mvn-version: mvn35
- project:
name: aaa
jobs:
- odl-maven-jobs
In this example we are using the job-group to assign a list of common jobs to
the aaa project. The job-group also hardcodes mvn-version to mvn35 and
build-timeout to 180 for all projects using this job-group.
A benefit of this method is for example disabling entire category of jobs by
modifying the job-group, insert disable-job: true parameter
against the jobs to disable.
Below is a list of Python job groups:
---
- job-group:
name: "{project-name}-python-jobs"
# This job group contains all the recommended jobs that should be deployed
# for a Gerrit-based Python project to verify commits using tox.
jobs:
- gerrit-python-xc-clm
- gerrit-tox-verify
- gerrit-tox-merge
- job-group:
name: "{project-name}-github-python-jobs"
# This job group contains all the recommended jobs that should be deployed
# for a Github-based Python project to verify commits using tox.
jobs:
- github-python-xc-clm
- github-tox-verify
- github-tox-merge
- job-group:
name: "{project-name}-gerrit-pypi-jobs"
# This job group contains the recommended jobs that should be deployed for
# a Gerrit-based Python project to test, build and deploy a package.
jobs:
- gerrit-pypi-merge
- gerrit-pypi-verify
- job-group:
name: "{project-name}-github-pypi-jobs"
# This job group contains the recommended jobs that should be deployed for
# a Github-based Python project to test, build and deploy a package.
jobs:
- github-pypi-merge
- github-pypi-verify
- job-group:
name: "{project-name}-gerrit-pypi-release-jobs"
# This job group contains the recommended jobs that should be deployed for
# a Gerrit-based Python project to promote a package from staging to pypi.
jobs:
- gerrit-pypi-release-merge
- gerrit-pypi-release-verify
- job-group:
name: "{project-name}-github-pypi-release-jobs"
# This job group contains the recommended jobs that should be deployed for
# a Github-based Python project to promote a package from staging to pypi.
jobs:
- github-pypi-release-merge
- github-pypi-release-verify
Job Templates
Python XC CLM
CLM scans for Python based repos. This job will call the Nexus IQ CLI
directly to run the scans.
A new credential named “nexus-iq-xc-clm” needs to exist in the Jenkins
credentials. The credential should contain the username and password
to access Nexus IQ Server.
| Template Names: |
- {project-name}-python-clm-{stream}
- gerrit-python-xc-clm
- github-python-xc-clm
|
|---|
| Comment Trigger: |
|---|
| | run-clm
|
| Required parameters: |
|---|
| |
| build-node: | The node to run build on. |
|---|
| jenkins-ssh-credential: |
|---|
| | Credential to use for SSH. (Generally should
get configured in defaults.yaml) |
|
| Optional parameters: |
|---|
| |
| build-days-to-keep: |
|---|
| | Days to keep build logs in Jenkins. (default: 7) |
| nexus-iq-cli-version: |
|---|
| | Nexus IQ CLI package version to download and use.
(default: 1.44.0-01) |
| nexus-iq-namespace: |
|---|
| | Insert a namespace to project AppID for projects that
share a Nexus IQ system to avoid project name collision. We recommend
inserting a trailing - dash if using this parameter.
For example ‘odl-‘. (default: ‘’) |
| build-timeout: | Timeout in minutes before aborting build. (default: 60) |
|---|
| git-url: | URL clone project from. (default: $GIT_URL/$PROJECT) |
|---|
| java-version: | Version of Java to use for the build. (default: openjdk8) |
|---|
| pre-build-script: |
|---|
| | Shell script to execute before the CLM builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment) |
| stream: | Keyword used to represent a release code-name.
Often the same as the branch. (default: master) |
|---|
| submodule-recursive: |
|---|
| | Whether to checkout submodules recursively.
(default: true) |
| submodule-timeout: |
|---|
| | Timeout (in minutes) for checkout operation.
(default: 10) |
| submodule-disable: |
|---|
| | Disable submodule checkout operation.
(default: false) |
| gerrit_clm_triggers: |
|---|
| | Override Gerrit Triggers. |
| gerrit_trigger_file_paths: |
|---|
| | Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for “file-path” details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit |
|
Python Sonar with Tox
Sonar scans for Python based repos. This job invokes tox to run tests
and gather coverage statistics from the test results, then invokes
Maven to publish the results to either a Sonar server or SonarCloud.
To get the Sonar coverage results, file tox.ini must exist and contain
coverage commands to run.
The coverage commands define the code that gets executed by the test
suites. Checking coverage does not guarantee that the tests execute
properly, but it identifies code that is not executed by any test.
This job reuses the Sonar builders used for Java/Maven projects which
run maven twice. The first invocation does nothing for Python
projects, so the job uses the goal ‘validate’ by default. The second
invocation publishes results using the goal ‘sonar:sonar’ by default.
For example:
[testenv:py27]
commands =
coverage run --module pytest --junitxml xunit-results.xml
coverage xml --omit=".tox/py27/*","tests/*"
coverage report --omit=".tox/py27/*","tests/*"
For more details refer to coverage and sonar documentation:
https://coverage.readthedocs.io/
https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
| Template Names: |
- {project-name}-tox-sonar
- gerrit-tox-sonar
- github-tox-sonar
|
|---|
| Comment Trigger: |
|---|
| | run-sonar
|
| Required parameters: |
|---|
| |
| build-node: | The node to run build on. |
|---|
| jenkins-ssh-credential: |
|---|
| | Credential to use for SSH. (Generally should
get configured in defaults.yaml) |
| mvn-settings: | The name of the settings file with credentials for the project. |
|---|
|
| Optional parameters: |
|---|
| |
| branch: | Git branch, should be master (default: master) |
|---|
| build-days-to-keep: |
|---|
| | Days to keep build logs in Jenkins. (default: 7) |
| build-timeout: | Timeout in minutes before aborting build. (default: 60) |
|---|
| cron: | Cron schedule when to trigger the job. This parameter also
supports multiline input via YAML pipe | character in cases where
one may want to provide more than 1 cron timer. (default: H 11 * * *
to run once a day) |
|---|
| disable-job: | Whether to disable the job (default: false) |
|---|
| git-url: | URL clone project from. (default: $GIT_URL/$PROJECT) |
|---|
| github-url: | URL for Github. (default: https://github.com) |
|---|
| java-version: | Version of Java to use for the build. (default: openjdk8) |
|---|
| mvn-global-settings: |
|---|
| | The name of the Maven global settings to use |
| mvn-goals: | The Maven goal to run first. (default: validate) |
|---|
| mvn-version: | Version of maven to use. (default: mvn35) |
|---|
| parallel: | Boolean indicator for tox to run tests in parallel or series.
(default: false, in series) |
|---|
| pre-build-script: |
|---|
| | Shell script to execute before the Sonar builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment) |
| python-version: | Python version to invoke pip install of tox-pyenv
(default: python2) |
|---|
| sonarcloud: | Boolean indicator to use SonarCloud true|false.
(default: false) |
|---|
| sonarcloud-project-key: |
|---|
| | SonarCloud project key. (default: ‘’) |
| sonarcloud-project-organization: |
|---|
| | SonarCloud project organization.
(default: ‘’) |
| sonarcloud-api-token: |
|---|
| | SonarCloud API Token. (default: ‘’) |
| sonar-mvn-goal: | The Maven goal to run the Sonar plugin. (default: sonar:sonar) |
|---|
| stream: | Keyword used to represent a release code-name.
Often the same as the branch. (default: master) |
|---|
| submodule-recursive: |
|---|
| | Whether to checkout submodules recursively.
(default: true) |
| submodule-timeout: |
|---|
| | Timeout (in minutes) for checkout operation.
(default: 10) |
| submodule-disable: |
|---|
| | Disable submodule checkout operation.
(default: false) |
| tox-dir: | Directory containing the project’s tox.ini relative to
the workspace. The default uses tox.ini at the project root.
(default: ‘.’) |
|---|
| tox-envs: | Tox environments to run. If blank run everything described
in tox.ini. (default: ‘’) |
|---|
| gerrit_sonar_triggers: |
|---|
| | Override Gerrit Triggers. |
| gerrit_trigger_file_paths: |
|---|
| | Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for “file-path” details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit |
|
Tox Verify
Tox runner to verify a project on creation of a patch set. This job
is pyenv aware so if the image contains an installation of pyenv at
/opt/pyenv it will pick it up and run Python tests with the
appropriate Python versions. This job will set the following pyenv
variables before running.
export PYENV_ROOT="/opt/pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
| Template Names: |
- {project-name}-tox-verify-{stream}
- gerrit-tox-verify
- github-tox-verify
|
|---|
| Comment Trigger: |
|---|
| | recheck|reverify
|
| Required Parameters: |
|---|
| |
| build-node: | The node to run build on. |
|---|
| jenkins-ssh-credential: |
|---|
| | Credential to use for SSH. (Generally set
in defaults.yaml) |
|
| Optional Parameters: |
|---|
| |
| branch: | The branch to build against. (default: master) |
|---|
| build-days-to-keep: |
|---|
| | Days to keep build logs in Jenkins. (default: 7) |
| build-timeout: | Timeout in minutes before aborting build. (default: 10) |
|---|
| git-url: | URL clone project from. (default: $GIT_URL/$PROJECT) |
|---|
| pre-build-script: |
|---|
| | Shell script to execute before the Tox builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment) |
| parallel: | Boolean indicator for tox to run tests in parallel or series.
(default: false, in series) |
|---|
| python-version: | Python version to invoke pip install of tox-pyenv
(default: python2) |
|---|
| stream: | Keyword representing a release code-name.
Often the same as the branch. (default: master) |
|---|
| submodule-recursive: |
|---|
| | Whether to checkout submodules recursively.
(default: true) |
| submodule-timeout: |
|---|
| | Timeout (in minutes) for checkout operation.
(default: 10) |
| submodule-disable: |
|---|
| | Disable submodule checkout operation.
(default: false) |
| tox-dir: | Directory containing the project’s tox.ini relative to
the workspace. The default uses tox.ini at the project root.
(default: ‘.’) |
|---|
| tox-envs: | Tox environments to run. If blank run everything described
in tox.ini. (default: ‘’) |
|---|
| gerrit_trigger_file_paths: |
|---|
| | Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for “file-path” details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit |
|
Tox Merge
Tox runner to verify a project after merge of a patch set. This job
is pyenv aware so if the image contains an installation of pyenv at
/opt/pyenv it will pick it up and run Python tests with the
appropriate Python versions. This job will set the following pyenv
variables before running.
export PYENV_ROOT="/opt/pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
| Template Names: |
- {project-name}-tox-merge-{stream}
- gerrit-tox-merge
- github-tox-merge
|
|---|
| Comment Trigger: |
|---|
| | remerge
|
| Required Parameters: |
|---|
| |
| build-node: | The node to run build on. |
|---|
| jenkins-ssh-credential: |
|---|
| | Credential to use for SSH. (Generally set
in defaults.yaml) |
|
| Optional Parameters: |
|---|
| |
| branch: | The branch to build against. (default: master) |
|---|
| build-days-to-keep: |
|---|
| | Days to keep build logs in Jenkins. (default: 7) |
| build-timeout: | Timeout in minutes before aborting build. (default: 10) |
|---|
| git-url: | URL clone project from. (default: $GIT_URL/$PROJECT) |
|---|
| pre-build-script: |
|---|
| | Shell script to execute before the CLM builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment) |
| python-version: | Python version to invoke pip install of tox-pyenv
(default: python2) |
|---|
| stream: | Keyword representing a release code-name.
Often the same as the branch. (default: master) |
|---|
| submodule-recursive: |
|---|
| | Whether to checkout submodules recursively.
(default: true) |
| submodule-timeout: |
|---|
| | Timeout (in minutes) for checkout operation.
(default: 10) |
| submodule-disable: |
|---|
| | Disable submodule checkout operation.
(default: false) |
| tox-dir: | Directory containing the project’s tox.ini relative to
the workspace. The default uses tox.ini at the project root.
(default: ‘.’) |
|---|
| tox-envs: | Tox environments to run. If blank run everything described
in tox.ini. (default: ‘’) |
|---|
| gerrit_trigger_file_paths: |
|---|
| | Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for “file-path” details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit |
|
PyPI Verify
Verifies a Python library project on creation of a patch set. Runs tox
then builds a source distribution and (optionally) a binary
distribution. The project repository must have a setup.py file with
configuration for packaging the component.
The tox runner is pyenv aware so if the image contains an installation
of pyenv at /opt/pyenv it will pick it up and run Python tests with
the appropriate Python versions. The tox runner sets the following
pyenv variables before running.
export PYENV_ROOT="/opt/pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
| Template Names: |
- {project-name}-pypi-verify-{stream}
- gerrit-pypi-verify
- github-pypi-verify
|
|---|
| Comment Trigger: |
|---|
| | recheck
|
| Required Parameters: |
|---|
| |
| build-node: | The node to run the build on. |
|---|
| jenkins-ssh-credential: |
|---|
| | Credential to use for SSH. (Generally set
in defaults.yaml) |
| project: | Git repository name |
|---|
| project-name: | Jenkins job name prefix |
|---|
|
| Optional Parameters: |
|---|
| |
| branch: | The branch to build against. (default: master) |
|---|
| build-days-to-keep: |
|---|
| | Days to keep build logs in Jenkins. (default: 7) |
| build-timeout: | Timeout in minutes before aborting build. (default: 15) |
|---|
| disable-job: | Whether to disable the job (default: false) |
|---|
| dist-binary: | Whether to build a binary wheel distribution. (default: true) |
|---|
| git-url: | URL clone project from. (default: $GIT_URL/$PROJECT) |
|---|
| parallel: | Boolean indicator for tox to run tests in parallel or series.
(default: false, in series) |
|---|
| pre-build-script: |
|---|
| | Shell script to execute before the tox builder. For
example, install system prerequisites. (default: a shell comment) |
| python-version: | Python version to invoke pip install of tox-pyenv
(default: python3) |
|---|
| stream: | Keyword representing a release code-name.
Often the same as the branch. (default: master) |
|---|
| submodule-recursive: |
|---|
| | Whether to checkout submodules recursively.
(default: true) |
| submodule-timeout: |
|---|
| | Timeout (in minutes) for checkout operation.
(default: 10) |
| submodule-disable: |
|---|
| | Disable submodule checkout operation.
(default: false) |
| tox-dir: | Directory containing the project’s tox.ini relative to
the workspace. The default uses tox.ini at the project root.
(default: ‘.’) |
|---|
| tox-envs: | Tox environments to run. If blank run everything described
in tox.ini. (default: ‘’) |
|---|
| gerrit_trigger_file_paths: |
|---|
| | Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for “file-path” details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit |
|
PyPI Merge
Creates and uploads distribution files on merge of a patch set. Runs
tox, builds a source distribution and (optionally) a binary
distribution, and uploads the distribution(s) to a PyPI repository.
This job should use a staging repository like testpypi.python.org,
which sets up use of release jobs to promote the distributions later.
This job can also use a public release area like the global PyPI
repository. The project git repository must have a setup.py file
with configuration for packaging the component.
The tox runner is pyenv aware so if the image contains an installation
of pyenv at /opt/pyenv it will pick it up and run Python tests with
the appropriate Python versions. The tox runner sets the following
pyenv variables before running.
export PYENV_ROOT="/opt/pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
Requires a .pypirc configuration file in the Jenkins builder home directory,
an example appears next that uses API tokens. Note that in the [pypi] entry
the repository key-value pair is optional, it defaults to pypi.org.
[distutils] # this tells distutils what package indexes you can push to
index-servers = pypi-test pypi
[pypi-test]
repository: https://test.pypi.org/legacy/
username: __token__
password: pypi-test-api-token-goes-here
[pypi]
username: __token__
password: pypi-api-token-goes-here
| Template Names: |
- {project-name}-pypi-merge-{stream}
- gerrit-pypi-merge
- github-pypi-merge
|
|---|
| Comment Trigger: |
|---|
| | remerge
|
| Required Parameters: |
|---|
| |
| build-node: | The node to run the build on. |
|---|
| jenkins-ssh-credential: |
|---|
| | Credential to use for SSH. (Generally set
in defaults.yaml) |
| project: | Git repository name |
|---|
| project-name: | Jenkins job name prefix |
|---|
|
| Optional Parameters: |
|---|
| |
| branch: | The branch to build against. (default: master) |
|---|
| build-days-to-keep: |
|---|
| | Days to keep build logs in Jenkins. (default: 7) |
| build-timeout: | Timeout in minutes before aborting build. (default: 15) |
|---|
| cron: | Cron schedule when to trigger the job. Supports daily builds.
This parameter also supports multiline input via YAML pipe | character in
cases where one may want to provide more than 1 cron timer. (default: empty) |
|---|
| disable-job: | Whether to disable the job (default: false) |
|---|
| dist-binary: | Whether to build a binary wheel distribution. (default: true) |
|---|
| git-url: | URL clone project from. (default: $GIT_URL/$PROJECT) |
|---|
| parallel: | Boolean indicator for tox to run tests in parallel or series.
(default: false, in series) |
|---|
| pre-build-script: |
|---|
| | Shell script to execute before the tox builder. For
example, install system prerequisites. (default: a shell comment) |
| pypi-repo: | Key for the PyPI target repository in the .pypirc file,
ideally a server like test.pypy.org. (default: pypi-test) |
|---|
| python-version: | Python version to invoke pip install of tox-pyenv
(default: python3) |
|---|
| stream: | Keyword representing a release code-name.
Often the same as the branch. (default: master) |
|---|
| submodule-recursive: |
|---|
| | Whether to checkout submodules recursively.
(default: true) |
| submodule-timeout: |
|---|
| | Timeout (in minutes) for checkout operation.
(default: 10) |
| submodule-disable: |
|---|
| | Disable submodule checkout operation.
(default: false) |
| tox-dir: | Directory containing the project’s tox.ini relative to
the workspace. The default uses tox.ini at the project root.
(default: ‘.’) |
|---|
| tox-envs: | Tox environments to run. If blank run everything described
in tox.ini. (default: ‘’) |
|---|
| gerrit_trigger_file_paths: |
|---|
| | Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for “file-path” details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit |
|
