Release Notes

v0.48.1-16

Prelude

This release supports breaking changes in the upstream OpenStack Jenkins plugin. Version 2.47+ of the OpenStack Cloud plugin for Jenkins adds two new SlaveOptions params, node_properties and config_drive. Failure to send these params results in a failed request.

A project may need more than one type of release job this patch changes the trigger to match on release file name for pypi and packagecloud release jobs.

New Features

  • Add support for automation of promoting packagecloud packages.
  • Add support for these new parameters, node_properties and config_drive. Add a new function test_version, which will do a numerical (float) comparison between installed plugin versions and user-supplied test version numbers.
  • Add the lf-kubernetes-create and lf-kubernetes delete macros which allow ad-hoc kubernetes cluster creation with a selectable disk size for the docker volume.
  • Release jobs now support non-ff merges.

Known Issues

  • Release jobs still trigger when a release file is deleted.

Upgrade Notes

  • If you are using the INSTANCE_MIN_CAPMAX paramater in your cloud configs, you will need to change it to INSTANCE_MIN when using v2.47+ of the plugin.

Deprecation Notes

  • Deprecated the config option INSTANCE_MIN_CAPMAX, replaced with INSTANCE_MIN when using v2.47+ of the OpenStack plugin.

Bug Fixes

  • Fix grep OS_CLOUD terminates jenkins-verify-images.sh if not found.
  • Removed dependency on ‘user’ venv created by python-tools-install.sh from the sandbox cleanup job. It will call lf-acitivate-venv() instead. Removed references to jjb-install.sh & jjb-cleanup.sh.
  • Removed dependency on ‘user’ venv created by python-tools-install.sh from the builder-jjb-verify job. It will call lf-acitivate-venv() instead.
  • Make shell script builder step customizable in lf-docker-get-container-tag. This will allow ONAP to call their own script. Set to default to “../shell/docker-get-container-tag.sh” which is what teams were already using.

Other Notes

  • Disable generation of pip package lists during builds: packages_end.txt.gz, packages_start.txt.gz & packages_diff.txt.gz. They are no longer valid.

v0.48.1

Bug Fixes

  • Change multi-cloud image validation to look for an OS_CLOUD variable in the cloud.cfg instead of overloading CLOUD_CREDENTIAL_ID as that variable does not usually point to an ID that is in the openstack clouds.yaml file.

v0.48.0

New Features

  • By default, the lf-venv-create() function only installs/upgrades ‘pip’ and and any packages that need to be tied to a specific version like. Currently the only version specific package is: ‘jenkins-job-builder==2.8.0’. The current version of lf-venv-create() supports being called twice but that functionality has NOT yet been thoroughly tested. RELENG-2508 has been created to validate & optimize that functionality.
  • Created new function lf-activate-venv(). This function creates a venv in /tmp and prepends the bin directory to the PATH. The ‘pip install’ command now specifies: ‘–upgrade-strategy eager’. lf-activate-venv() supports an optional –python flag to specify which python to use to create the venv, the default is python3.

    Two new functions: lf-git-validate-jira-urls() and lf-jjb-check-ascii(). They will be used to replace the git-validate-jira-urls.sh & jjb-check-unicode.sh scripts at some point. For now, they are not being used.

Bug Fixes

  • Update the gerrit trigger regexes for the docker job templates to use the shorter and more readable versions of the regexes used in other verify and merge job templates. Clarify documentation for comment triggers.
  • Multi-cloud image validation was not properly working as we force set the OS_CLOUD environment variable before validation of images. This has been rectified to dynamicaly modify the OS_CLOUD variable based upon the cloud.cfg that will be used to define the cloud in Jenkins.
  • We now validate that a cloud.cfg file exists for any defined OpenStack cloud.
  • Removes undocumented, and now unneeded, openstack-cloud variable from the jenkins-cfg-verify job definition
  • Change compare-type to REG_EXP in macros lf_python_clm_xc and lf_pypi_common for config parameter gerrit_trigger_file_paths with the regular expression .* Previously was ANT, which didn’t match anything with pattern .* so recheck/reverify comment triggers did not work. Clarify documentation of comment triggers for python jobs.
  • Add .pypirc config file provider to builders in release job. Extend PyPI documentation with recommended directory layout.
  • Change the comment-added-contains-event strings for the release-verify and release-merge templates to the same regexes used in other verify and merge templates, instead of custom versions that didn’t seem to work; verify now uses ‘^Patch Sets+d+:s+(recheck|reverify)s*$’ merge now uses ‘^Patch Sets+d+:s+remerges*$’
  • Change pip’s upgrade-strategy to “eager” for python-tools-install.sh. Pip changed its default upgrade-strategy to “only-if-required”, which is not correctly upgrading requests (and potentially other packages in the future) to meet other packages’ requirements. This results in errors in the build log. By using the upgrade-strategy “eager”, pip is able to properly install what is needed.

Other Notes

  • The lf-venv-add(), lf-venv-create() & lf-venv-activate functions have been removed. No-one is accessing it yet.

v0.47.0

New Features

  • Refactor PyPI release-verify and release-merge templates to download distribution files from a PyPI staging index and upload the files to a PyPI release index. Remove all the builders that were previously used. Call pip upgrade to install latest version of setuptools and twine. Split the PyPI job groups because the verify & merge templates do not accept the same arguments as release-verify & release-merge templates. Remove stream, branch usage; just one PyPI release job per project now. Extend the PyPI release yaml file schema for log_dir and other values. All this makes the PyPI release ver/mrg templates highly similar to the release-job ver/mrg templates. Revise documentation appropriately. Move the PyPI release features to the lf-release-jobs.(rst,yaml) files. Extend the release-job.sh script to have pypi release functions, and drop the pypi-tag-release.sh script.
  • Added two new functions: lf-venv-create(), lf-venv-add() Changed name of lf-activate() -> lf-venv-activate() Updated functionality of lf-venv-activate (support for paths & python versions

Bug Fixes

  • Added ‘|| return 1’ to appropriate commands. Functions do not support ‘-e’ and a function will normally continue after a command fails.

Other Notes

  • Improved Comments

v0.46.0

New Features

  • Read-the-docs job will run but skip its verification bits unless the repo has .readthedocs.yaml file in the root of its repository. This allows projects to commit changes to their docs/ dir without having to configure the read the docs builds.

Upgrade Notes

  • Extend lf_tox_sonar with sonarcloud, sonarcloud-project-key, sonarcloud-project-organization, sonarcloud-api-token, tox-dir and tox-envs properties; also with lf-infra-tox-parameters macro. Use new sonarcloud property as guard for conditional builder step, if true use lf-infra-maven-sonarcloud, else use lf-infra-maven-sonar.

Bug Fixes

  • Add missing trigger “remerge” to the PyPI release merge template. Move trigger definitions into PyPI templates, instead of defining four separate trigger definition blocks and using each exactly once. Document the required comment text for the triggers.
  • Add macro with lf-infra-wrappers block to set jenkins-ssh-credential parameter to the value in parameter jenkins-ssh-release-credential, which makes the PyPI release merge templates parallel to the release-job merge template. Both need privileges to push a tag on the Jenkins minion. Document the revised configuration parameter. Silence yamllint issues.
  • Add missing underscore in two echo commands in release-job.sh. Move comment for shellcheck disable to new line in lf-env.sh. Extend documentation of the container self-release process. Exclude lf-rtdv3.rst from WriteGoodBear coala processing.
  • Add a verification step to maven releases to make sure the version being defined in the releases file matches the actual version produced by the maven-stage job that created the release candidate. This is to prevent releases being pushed in Nexus with a version different from what the developer intended in the releases file.
  • Add missing {branch} parameter to branch-pattern in gerrit trigger blocks in PyPI release-verify and release-merge templates. Jobs were starting on all defined branches, not limited to target. Change pypi-tag-release script to continue if tag exists, not stop.
  • Update to lf-env.sh. Change lf-set-maven-options(): MAVEN_OPTIONS to maven_options to better support shellcheck.
  • Add step to verify Nexus staging repository is closed correctly or fail the staging job, if the repository is still open. This allows projects to address the validation failures earlier during maven-stage phase, instead of waiting to catch them while running the self-release job.

v0.45.0

New Features

  • Add lf-env.sh ‘library’ script in ~jenkins. Sourcing this ‘library’ script from a bash script provides access to a number of functions. The script is installed by the ‘init’ script at boot time so that is is accessible from any Jenkins build script. Hopefully over time other functions will be added to this library.

Upgrade Notes

  • Add sonar-project-file parameter to ci sonar jobs. By enabling the caller to override the default project file name with an empty string, we enable the ability to provide project settings directly in the sonar-properties field. This removes the requirement for a “sonar-project.properties” file in the repo.

Bug Fixes

  • Fix the auto update image script to compare the image type before updating an the image in the source repository. This fixes the bug that updates images although they are the same flavour but a diffirent type.
  • Add DRY_RUN parameter to the PyPI verify and merge template common macro because it’s required by the pypi_upload.sh script. Adjust verbosity of shell scripts - quiet down pip, add repository name to echo output.
  • Correct doc .pypirc test URL to https://test.pypi.org/legacy/ Change doc .pypirc to use API tokens instead of username/password. Use pypi-test as the default repository name in the PyPI merge template (instead of “staging”) to match established ONAP practice.

v0.44.1

New Features

  • Archive ‘sudo’ logs. The log will be located in the ‘sudo’ sub-directory of the archive. The actual name of the log-file depends on the OS of the builder.

Bug Fixes

  • Change Packer version back to 1.4.2. Packer 1.4.3 frequently has issues setting metadata after a build completes. Packer 1.4.4 is not expected until October, so this change is needed now to fix the broken builds.
  • In the PyPI merge template, add cron parameter to support daily build and push to a staging repo, like the maven merge template. In PyPI release templates, change name of gerrit and github trigger file patterns parameter. This avoids accidental overriding by jobs that limit their actions to subdirectories. The release file patterns are hardcoded in a shell script. Remove params from RST doc. In all PyPI templates, add disabled option and disable-job parameter to be consistent with other python templates.

v0.44.0

New Features

  • Add an additonal Sonar job that allows the caller to provide a builder that runs prior to the Sonar scan.
  • Add template to update OpenStack cloud images.
  • This job finds and updates OpenStack cloud images on the ci-management source repository.
  • The job is triggered in two ways:
    1. When a packer merge job completes, the new image name created is passed down to the job.
    2. Manually trigger the job to update all images.
  • When the job is triggered through an upstream packer merge job, this only generates a change request for the new image built.
  • When the job is triggered manually, this job finds the latest images on OpenStack cloud and compares them with the images currently used in the source ci-management source repository. If the compared images have newer time stamps are all updated through a change request.
  • This job requires a Jenkins configuration merge and verify job setup and working on Jenkins.
  • New templates to build and push Python source and binary distributions to a PyPI server. Includes: {project-name}-pypi-verify-{stream}, gerrit-pypi-verify, github-pypi-verify, {project-name}-pypi-merge-{stream}, gerrit-pypi-merge, github-pypi-merge, {project-name}-pypi-release-verify-{stream}, gerrit-pypi-release-verify, github-pypi-release-verify, {project-name}-pypi-release-merge-{stream}, gerrit-pypi-release-merge, github-pypi-release-merge,

Upgrade Notes

  • Packer merge jobs have a new build parameter when checked also updates the cloud image.
  • lf-infra-packer-build macro now requires 1 new variables to be passed.
    1. update-cloud-image: Set to true when images need to be updated on Jenkins.

Bug Fixes

  • Changed the trigger to run sonar from stage-release to run-sonar. This makes it more concistent with the other parts.
  • Run WhiteSource scan jobs weekly on Sunday.
  • Pip install pyenv from python2 should force more-itertools to 5.0.0 In a fresh python2.7 venv “pip install pyenv” correctly pulls down more-itertools [required: Any, installed: 5.0.0] If for some reason a higher version is already installed this will downgrade more-itertools to a py2 compatible version
  • Allow java-opts to be defined in WhiteSource scans. This avoids java heap failures.

v0.43.1

Upgrade Notes

  • CONTAINER_PULL_REGISTRY and CONTAINER_PUSH_REGISTRY need to be defined in Jenkins global environment varaibles.

Bug Fixes

  • Pin python-cinderclient to 4.3.0.

    A new version of python-cinderclient 5.0.0 is released which breaks the openstack jobs.

  • openstack –os-cloud vex limits show –absolute Error: No module named v1.contrib
  • Debug info shows: File “/home/jenkins/.local/lib/python2.7/site-packages/openstackclient/volume/client.py”, line 40, in make_client from cinderclient.v1.contrib import list_extensions ImportError: No module named v1.contrib
  • Update echo commands in release-job.sh for easy identification.
  • Update release-container-schema to use CONTAINER_PULL_REGISTRY and CONTAINER_PUSH_REGISTRY from Jenkins global variables. These will be used to pull and push operations for self release containers. Can be overwritten in the releases files.
  • Fix warnings from tox (shellcheck).
  • Remove quotes from optional var WSS_UNIFIED_AGENT_OPTIONS. When empty, the quotes will cause WS Unified Agent failures.
  • Upgrade WS Unified Agent CLI to latest version 19.8.1

v0.43.0

New Features

  • Add python tox merge job templates for gerrit and github. These templates are triggered by merge events to test the Python code on the branch that received the merge. Renamed tox-verify macro to tox-common.

Bug Fixes

  • Import GPG signing key in release jobs before verifying Gerrit tag details.
  • INFO validate job checks that repositories matches $PROJECT Catches projects that replace / with - in their INFO file Also ensures that repositories only has one entry. We are not supporting multiple projects with a single INFO.yaml file.
  • Remove unused the MAVEN_CENTRAL_URL variable. The self-release job is designed to work with any Nexus repository info published in staging-repo.txt.gz, which makes the MAVEN_CENTRAL_URL redundant, hence remove the unused variable.
  • Revise tox-install.sh script to invoke python using environment variable PYTHON instead of hardcoding “python”, and remove the pip –quiet flag. Extend the RTD template to pass python-version, default python2.
  • Use existing builder lf-infra-maven sonar, drop incomplete builder lf-tox-maven-sonar, to gain desired behavior of pushing code analysis results to Sonar. Use trivial goal ‘validate’ by default. The script maven-sonar.sh calls maven twice, first with build goals and then with Sonar goals. The incomplete builder did not supply the build goals.
  • Release schema verification needs to happen first before we attempt to assign values to the variables. Validate version only after the schema validation has passed and the variables are assigned.
  • Organize variable setup into functions. Maven release files expects different variables than container release files.
  • Rename “version” variable in container release files to “container_release_tag” which is a better user friendly name given the fact that container versions are rather called tags. Internally, we still process it as “version” to allow reuse of the tag function.

v0.42.1

New Features

  • Add DRY_RUN build param to do a test run the job with publishing artifacts.

Upgrade Notes

  • Update lftools version to v0.26.2.

Bug Fixes

  • Verify both repos before attempting release. We have run into a case where the repo on ODL nexus was good, and the repo on Sonatype nexus was missing. Cover this case by running the verify loop over each repo before attempting release.

v0.42.0

New Features

  • Add support for distribution_type “container”
  • Add function maven_release_file and container_release_file and the logic to choose the correct one. No functional change to maven_release_file.
  • Add docker login step when docker releases are being processed.
  • container_release_file downloads log_dir/console.log.gz and parses it to get a list of container name and version. Verifies pulls container and grabs the image_id then performs the merge then tags and pushes the container.
  • Add lf-sonar-common job-template to lf-ci-jobs.yaml and add lf-infra-sonar to macros.yaml. The purpose of a new job template is to adopt using jenkins sonar plug-in along with the sonar-project.properties file versus pom.xml. Lastly the new job template will ensure that anything using lf-infra-tox-sonar is unaffected.
  • Add support for “Build with Parameters” for projects that do not want to use a release file for maven builds.

Upgrade Notes

  • release-verify and merge will need to run on a docker build-node for example centos7-docker-8c-8g Lftools will need to be updated to 0.26.0 so that -v is supported for lftools nexus release
  • Update lftools version to v0.26.1.

Bug Fixes

  • Fix missing extension in ID for release-schema.yaml.
  • Make “distribution_type” mandatory in future release files.
  • Rename “RELEASE_FILE” parameter to “USE_RELEASE_FILE” in release-jobs. This will match the actual varaible default value better and will not collide with the local “release_file” in the script.
  • Fix “USE_RELEASE_FILE” if statement. We are now using a bool instead of a string. Changing the if statements to evaluate bools.
  • Add pre-build-script parameter to python clm, tox and sonar templates. Gives flexibility to install prerequisite libraries, rearrange the source tree, etc.
  • Restructure shell/release-job.sh into functions.

v0.41.0

Upgrade Notes

  • Update lftools version to v0.26.0.
  • Update packer version to 1.4.3. This new packer version fixed an issues in docker image, where its unable to install the packages into docker containers due to checking of wrong container OS. Fix in 1.4.3: builder/docker: Check container os, not host os, when creating container dir default [GH-7939]

Bug Fixes

  • Project job sections that define gerrit_trigger_file_paths are overriding ones set in Default parameters of lf_release_common Hard Code gerrit_trigger_file_paths to fix this.

v0.40.4

Upgrade Notes

  • Update lftools version to v0.25.4.

Bug Fixes

  • Git fetch needs the dashed version git fetch “$PATCH_DIR/${PROJECT////-}.bundle”. Fix if statement.
  • Release creds are only required for promoting the repo, which uses diff ACL as compared to normal user. Therefore dont use the release creds for the verify jobs and the scm sections in both the job templates.
  • 1. The release merge job is a one way operation. Given this, Release jobs should only exist in the format {project-name}-release-verify and {project-name}-release-merge As these jobs trigger from a change to any branch/** These jobs Must Exit 0 if release job has already tagged a repo. Inevitably a release file will be pulled in from master to branch or a remerge will be requested This will again trigger the release jobs. since in this case the repo is already tagged, the job should not report a failure. This is solved by having the verfiy and merge exit 0 when the repo is already tagged 2. Rather than use project as defined in the release file use ${PROJECT////-} This changes PROJECT=”optf/osdf/foo/bar” to optf-osdf-foo-bar so that we can fetch the log files. by changing /’s in the project names to -‘s

v0.40.3

Known Issues

  • Update release job template to tigger on any branch name, and not just ‘master’. ODL projects branches are version ‘4.0.x’ which requires passing the branch name to the template.

Bug Fixes

  • Fix the release job script to handle any trailing ‘/’ set on log_dir and also handle unbound variables correctly.
  • Update lftools version to v0.25.3.

v0.40.1

Upgrade Notes

  • Projects using lf-release-job will need to add the project’s signing public key in their Jenkins Settings Files.

Bug Fixes

  • Use {GERRIT_PROJECT} when calling Gerrit in release-merge job.
  • Project pattern was incorrectly set to ** must be {project}
  • The self-release jobs does not handle multiple repositories listed in staging-repo.txt file. This fixes the issue by deriving the NEXUS_URL and the STAGING_REPO from each entry in the file. This approach also eliminates the need for having multiple release.yaml files for every staging-repo.
  • Allow lf_release_verify and lf_release_merge to verify tag signature.

v0.40.0

New Features

  • Add packer image $NAME to the description setting so that the image name is displayed above the job logs URL. This saves a some time from looking for the image name in the jobs logs.
  • Allows projects to promote their own builds. Requires setup of accounts and permissions in Gerrit, Jenkins and Nexus. Please refer to the lf-release-jobs documentation for details.
  • Remove orphaned ports from the openstack cloud environment. These orphaned ports are residue of CSIT jobs that needs to be purged, as a part of the openstack cron job. A large number of stale jobs could cause IP address allocation failures.
  • Enable JAVA_HOME to point to openjdk12 install path for CentOS 7.

Upgrade Notes

  • Consolidated lf-infra-jjbini macros with JJB 2.0. This requires renaming any Jenkins managed files “jjbini-sandbox” to “jjbini” to switch to the format supported in JJB > 2.0.
  • Projects using lf-release-jobs need to make sure they have the global variable NEXUSPROXY added in Jenkins production and Jenkins sandbox servers. The value of this variable should be the URL to the project’s Nexus server. Previous commit 118b7cbf171aca498d1a0a3a485bad990ad2e7b6 missed this variable.
  • Projects using lf-release-jobs need to make sure they have the global variable NEXUSPROXY added in Jenkins production and Jenkins sandbox servers. The value of this variable should be the URL to the project’s Nexus server.
  • This change will require to update lf-release-job calls. Update from using “{project-name}-releases-merge-{stream}”, “{project-name}-releases-verify-{stream}” to “{project-name}-release-merge-{stream}”, “{project-name}-release-verify-{stream}”. No upgrade need to be done if using “{project-name}-gerrit-release-jobs” group.

Bug Fixes

  • There is no way on finding out the $JOB_NAME pushed to sandbox with the jjb-deploy command in the logs. The change outputs the $JOB_NAME to the logs, which is useful for debugging purposes.
  • Add release-schema used to validate the releases yaml file as part of lf-release-jobs.
  • Tarball the $JAVADOC_DIR as a workaround for javadoc verfiy jobs to avoid uploading a large number of small files. Uploading a large number of small files does not work well with Nexus unpack plugin which fails on 504 gateway timeout.
  • Allow maven goals to be configured in sonatype-clm.sh. Set to “clean install” by default.
  • Allow maven goals to be configured in maven-sonar.sh. Set to “clean install” by default.
  • Add support for JAVA_OPTIONS in sonar job. Some of the maven build options in the sonar job require to set the JAVA_OPTIONS to specific value for the build to pass This option will help to pass the JAVA_OPTIONS from the template.
  • Perform “lftools schema verify” command to validate the release files against schema/release-schema.yaml Obtain optional maven central URL inside the loop that scans release files.
  • Allow only semantic release versions like “v${SEMVER}” or “${SEMVER}”. Fail the script if the version is not valid. Do not append any additional characters to the release version during tag and push steps.
  • Delete stacks with the --force option to ensure that any delete failures does not stop the openstack-cron jobs from continuing.
  • Download raw version of release-schema.yaml to compare against release files using lftools.
  • Avoid the usage of project specific variables. Do not use ODLNEXUSPROXY var, but instead use a generalized variable.
  • Avoid the usage of project specific variables. Do not use ODLNEXUSPROXY var, but instead use a generalized variable.
  • Using “releases” and “release” in different places is becoming confusing. Standardize to “release” to match lftools command and the majority of the exisiting wording.

    Use “releases” for the list of tech team releases and trggers since it is intuitive there. For example “releases/1.1.1.yaml”

  • Move info-schema to schema/info-schema.yaml to keep schemas consistency.
  • Download only needed files for lf-info-yaml-verify rather than cloning the entire repo.
  • Allow lf-maven-stage jobs to be triggered using either “stage-release” or “stage-maven-release”.
  • Allow lf-maven-docker-stage jobs to be triggered using either “stage-release” or “stage-docker-release”.
  • Upgrade to the WhiteSource Unified Agent version 19.7.1.

v0.39.1

Bug Fixes

  • Update lftools version to v0.25.2.

v0.39.0

Critical Issues

  • lftools v0.24.0 introduced a major issue which caused the Jenkins cloud configuration merge job for OpenStack clouds to fail. This has been corrected in lftools v0.25.1

Bug Fixes

  • Add missing git-url variable in {project-name}-releases-merge-{stream} job.
  • Add the {stream} name in releases-verify and releases-merge jobs.
  • Some ONAP components like DCAEGEN2 do not host a version.properties file in the root of their repos. We need to be able to provide a location and/or different name for the version.properties file for jobs using the lf-maven-versions-plugin builder step.

v0.38.4

New Features

  • Group {project-name}-releases-verify and {project-name}-releases-merge into {project-name}-gerrit-release-jobs.

    Add test jobs for lf-release-jobs.

Bug Fixes

  • Change parameter names used to specify container tag method, with a new default to use the fixed string ‘latest’ as a tag. Merge two shell scripts into one instead of using Jenkins conditional steps. Extend to accept a custom directory for the container-tag.yaml if the yaml-file method is used to set the docker tag information; this is an optional variable which is set to empty by default, and falls back to DOCKER_ROOT.
  • Add missing scm block in gerrit-releases-merge job definition. Add missing submodule-disable variable for jobs using lf-infra-gerrit-scm. Update documentation for gerrit-releases-merge and gerrit-releases-verify to remove submodule options as optional parameters.
  • Update lftools version to v0.25.0
  • Add missing $ to variable tag_file so the yq query can pull the tag from the container-tag.yaml file.
  • Add -l to /bin/bash shebang line at top of docker-get-container-tag.sh to make it a login shell, which automatically includes /home/jenkins/.local/bin on the path, because that is where pip installs the yq command.

v0.38.3

Bug Fixes

  • Add trigger on cron to docker merge macro to support regular rebuilds. This makes the merge macro match the behavior of most other jobs.
  • Add yamllint verification to INFO.yaml files.

Other Notes

  • Update lftools version to v0.24.0.

v0.38.2

Bug Fixes

  • Add missing config for triggering on file paths to docker macros and templates, namely gerrit_trigger_file_paths and github_included_regions, to make the verify and merge macros and templates match the behavior of other jobs.

v0.38.1

Bug Fixes

  • Install yq to be used to read yaml files. In specific, it will be needed to read container-tag.yaml.
  • When calling builder step macros “lf-docker-get-container-tag”, “lf-docker-build” and “lf-docker-push”, make sure the needed variables are also passed explicitly to avoid these variables appear as undefined.
  • Allow DOCKER_ARGS to be empty in docker-build.sh. This is not a required parameter, it can be empty.

    Remove container reference in docker-get-git-describe.sh. The CONTAINER_PUSH_REGISTRY already gets added in the docker-push script. No need to add it again.

    Rename image_name to image_build_tag in docker-get-yaml-tag to match docker-get-git-describe. Add missing “DOCKER_NAME” in the DOCKER_IMAGE.

    docker-get-git-describe.sh and docker-get-yaml-tag.sh should only export the tag variable. Let docker-build.sh process DOCKER_NAME

Other Notes

  • Add yq install as part of python-tools-install.sh Allow future scripts to use yq package.

v0.38.0

New Features

  • gerrit-tox-verify now has a new parameter gerrit-skip-vote (bool) to control whether Jenkins should skip voting depending on the build outcome. It defaults to false since it is the default used by the Jenkins Gerrit Trigger Plugin.
  • gerrit-docker-verify runs for new commits and runs a build of the affected Docker images.
  • gerrit-docker-merge runs for merged commits, runs a build of the affected Docker images and pushes the images to a specified Docker registry.
  • New lf-release-job-merge and lf-release-job-verify templates allow projects to have self-serve releases. Project will create a tagname.yaml file in the releases/ directory of their git repo. example:

    $ cat releases/4.0.0.yaml
    ---
    distribution_type: 'maven'
    version: '4.0.0'
    project: 'odlparent'
    log_dir: 'odlparent-maven-release-master/11/'
    #below is optional
    maven_central_url: 'oss.sonatype.org'
    
  • lf-infra-gerrit-scm and lf-infra-github-scm now require a submodule-disable parameter (bool) to control whether submodules are ignored or not during git fetch operations.
  • All job-templates now provide an optional submodule-disable parameter for git fetch operations, defaulting to false.

Upgrade Notes

  • Any project using the lf-infra-gerrit-scm and lf-infra-github-scm macros in global-jjb should need to add a submodule-disable value. It is recommended to default this value to false since it is the default used by the Jenkins Git Plugin.
  • Update gerrit comment trigger to use a more standard regex and avoid triggering jobs, when these keywords are intended to be used as code review comments between users. Also improve the regexs to make them more succinct and readable.

Bug Fixes

  • Add missing config for triggering on file paths to maven stage macros and templates, namely gerrit_trigger_file_paths nad github_included_regions, to make those macros and templates match the behavior of maven verify and maven merge.
  • fix multiple jobs created using same job-template update same github check status due to hard coded status-context to Maven Verify. Now appending status-context with maven-version and java-version to make it unique. And create different status checks in the github. fix applied for maven verify and maven docker verify jobs
  • Fix log shipping script to not require a LOGS_SERVER. There was a regression that caused the log shipping script to start requiring a LOGS_SERVER which fails in the case of a system that does not have that optional environment variable set.
  • Handle multiple search extension or patterns passed by upstream JJB ARCHIVE_ARTIFACTS param as a single string by spliting these values before being passed to lftools deploy archives.

    ARCHIVE_ARTIFACTS="**/*.prop \
                      **/*.log \
                      **/target/surefire-reports/*-output.txt \
                      **/target/failsafe-reports/failsafe-summary.xml \
                      **/hs_err_*.log **/target/feature/feature.xml"
    

    For example, the above env variable passed to the script and to lftools deploy archives as:

    lftools deploy archives -p **/*.prop \
                      **/*.log \
                      **/target/surefire-reports/*-output.txt \
                      **/target/failsafe-reports/failsafe-summary.xml \
                      **/hs_err_*.log **/target/feature/feature.xml \
                      "$NEXUS_URL" \
                      "$NEXUS_PATH" \
                      "$WORKSPACE"
    

    The correct way of passing this as per lftools implmentation is:

    lftools deploy archives -p '**/*.prop' \
                      -p '**/*.log' \
                      -p '**/target/surefire-reports/*-output.txt' \
                      -p '**/target/failsafe-reports/failsafe-summary.xml' \
                      -p '**/hs_err_*.log' \
                      -p '**/target/feature/feature.xml' \
                      "$NEXUS_URL" \
                      "$NEXUS_PATH" \
                      "$WORKSPACE"
    
  • Fix error with handling unbound arrays for search extensions, when using set -u. The correct way of using this.

    set -u
    arr=()
    echo "output: '${arr[@]}'"
    bash: arr[@]: unbound variable
    echo "output: '${arr[@]:-}'"
    foo: ''
    
  • lf-maven-versions-plugin builder step needs to run before maven-patch-release.sh as this second script contains a condition to confirm if the maven vesions plugin was selected as a way to remove the ‘SNAPSHOT’ pattern from the pom.xml files. lf-maven-docker-stage was based on lf-maven-stage and it seems that these particular builder steps were switched in place accidentally.
  • The logs-deploy.sh script now allows ARCHIVE_ARTIFACTS to contain zero or more files.
  • request-2.22.0 does not work with python-3.4.9, so pin requests to v2.21.0 to address the tox failures.

v0.37.2

Bug Fixes

  • Use maven goal install (not deploy) in the maven + docker verify job. An image cannot be pushed by a verification job, and the deploy target directs the plugin to push.

v0.37.1

Bug Fixes

  • Remove maven-versions-plugin-set-version variable in newly added macro. This is a variable that does not need to be defined by the users of the jobs. The version needed in this builder step is inherited from versions.properties as “release_version”.

v0.37.0

New Features

  • Add verify, merge and stage templates for Java projects that build and wrap a JAR (e.g., a Spring-Boot application) inside a Docker image, and do not need to deploy any JAR libraries or POM files.

Upgrade Notes

  • The next release of common-packer will require a minimum version of 1.3.2 for packer. The current release of packer is 1.4.0.

Bug Fixes

  • The packer-merge job for Gerrit systems was improperly configured to use the Gerrit Trigger choosing strategy and not default. This caused issues unexpected issues with retriggering merged changes when the expectation was that it would pick up the lastest change as per normal.
  • This is a variable that does not need to be defined by the users of the jobs. The version needed in this builder step is inherited from versions.properties as “release_version” and it is fixed as that. This also helps teams not having to define this version in 2 places and just rely on version.properties.
  • Projects using maven versions plugin let this plugin take care or updating their versions in the pom.xml. When maven-versions-plugin is set to “true”, skip the stripping of SNAPSHOTS from the pom.xml files. maven-versions-plugin is set to “false” by default.

Other Notes

  • Update example Jenkins Init Script in README to redirect all output to a log file.

v0.36.0

Prelude

WhiteSource is a security and license compliance management platform. It is used to perform scans on a great variety of coding and scripting languages.

New Features

  • New comment-to-gerrit builder will comment back to gerrit patchset if a file called gerrit_comment.txt is created by the build.
  • Allows maven to run a clean install step before the WhiteSource scan runs the Unified Agent to fetch additional dependencies. Set to false by default.
  • Job {project-name}-whitesource-scan-{stream} uses the WhiteSource Unified Agent scanner CLI tool to perform the code scan and report the results into the WhiteSource dashboard.

Bug Fixes

  • Tag releases will now trigger a docs build to regenerate and update the release note link.
  • Update jenkins-cfg-verify job to validate new images names obtained from $GERRIT_REFSPEC instead of the master branch.
  • Hardcode project version to the “GERRIT_BRANCH”. Follow previous convention from CLM where reports were versioned after the branch name. Fix minor nits with bash varaibles.
  • wss-unified-agent.config file should not be opened for configuration to tech teams. The config file should be part of Jenkins Settings Files and called via Managed Files. wss-unified-agent.config must be created in Jenkins config files based on wss-unified-agent.config.example.

Other Notes

  • To run this job, a configuration file is needed (wss-unified-agent.config.example). A new secret text credential will need to be created. (ID=wss-apiKey Secret=WhiteSource organization API key)
  • Update lftools version to v0.23.1.

v0.35.0

New Features

  • The jjb-merge job now has a new parameter jjb-workers to allow configuration of the number of threads to run update with. Default is 0 which is equivalent to the number of CPU cores available on the system.
  • New info-vote-verify macro Will count votes against an INFO.yaml change and sumbit automatically if a majority of committers vote +1 or +2 on the change. Job is triggered by +2 votes or a comment of “vote”

Other Notes

  • The Maven Verify job will now call -Dmaven.source.skip to skip source jar generation in the verify job. This saves us some time in the verify build as the source artifacts are not useful in a verify job.

v0.34.0

New Features

  • jenkiins-init-scripts The ‘ciman’ repo is not longer required to be located in ‘/opt/ciman’.
  • lf-maven-set-version conditional step for lf-maven-stage to allow teams to run Maven versions plugin to update their artifact versions. Step will run if maven-versions-plugin is set to true.
  • Support for the Throttle Plugin is added to JJB jobs so static build servers can restrict the number of concurrent JJB jobs ran at the same time.

    This must be explicitly enabled by setting throttle-enabled on the jobs.

Bug Fixes

  • Adapt maven path search for files and dirs. The “-f” maven param can specify both a directory, in which case it will look for “pom.xml” in the directory, or a specific file. The original version of this search was only compatible with directories that contain a pom.xml file.
  • Update the lf-maven-cental macro documentation and example templates with the missing requireed params.
  • Fix JAVA_HOME for openjdk11 on CentOS 7 to use the OpenJDK version installed in /usr/lib/jvm/java-11-openjdk.
  • The JJB Deploy Job is configured to trigger only if the Gerrit comment starts with the jjb-deploy keyword.

    Without the regex being optimized the job triggers on any occurance of the jjb-deploy keyword in a Gerrit comment, with is waste infra resources.

    Example of a valid command in Gerrit comment that triggers the job:

    jjb-deploy builder-jjb-*

    Example of a invalid command in Gerrit comment that would _not_ trigger the job:

    Update the job. jjb-deploy builder-jjb-*

v0.33.0

New Features

  • jenkins-init-scripts If the environmental variable ‘SWAP_SIZE’ is set when the ‘init.sh’ script is called, then a ‘SWAP_SIZE’ GB swap space will be configured. If ‘SWAP_SIZE’ is ‘0’ or is not a valid integer, then no swap space is configured. If it is unset then 1GB of swap will be configured. Previously the swap size was fixed at 1GB.
  • jenkins-init-scripts If the work directory or volume (/w) aleady exists, the ownership will be recursivly set to ‘jenkins:jenkins’. Previously only the top directory /w was owned by ‘jenkins:jenkins’
  • lf-sigul-sign-dir macros now supports a sign-mode parameter which allows jobs to choose to sign artifacts using either parallel mode or serial mode (default).

Upgrade Notes

  • lf-sigul-sign-dir users need to add a new parameter sign-mode to their job-templates setting either parallel or serial as the value, we recommend setting serial mode for this setting.

    {project-name}-maven-stage-{stream}’s Sigul signer now defaults to serial mode instead of the previous parallel behaviour. To change this back to the previous behaviour pass the “sign-mode” parameter to the job template:

    - project:
        name: parallel-sign
        jobs:
          - gerrit-maven-stage:
              sign-mode: parallel
    

v0.31.0

New Features

  • New job-template {project-name}-release-announce for lf-releng projects to automate release announcement emails.
  • Add support for pushing Sonar results to SonarCloud. Refer to Maven Sonar docs for details.

Upgrade Notes

  • Jobs using the lf-maven-stage macro now need to update to the new usage. Preparation calls to lf-provide-maven-settings, lf-infra-create-netrc, and lf-provide-maven-settings-cleanup are no longer necessary to prepare the lf-maven-stage macro.

    Usage:

    - lf-maven-stage:
       mvn-global-settings: 'global-settings'
       mvn-settings: 'settings'
       mvn-staging-id: 'staging profile id'
    

v0.30.0

New Features

  • Packer merge jobs now include the image name in the Jenkins build description.

Bug Fixes

  • Extend ${JOB_NAME} to include {java-version} parameter to support jobs to build with multiple versions of openjdk{8,11}.
  • Modified lf-maven-jobs.yaml sonar cron entry to ‘{obj:cron}’ to pass value from custom user config file.

v0.29.0

New Features

  • Add a puppet-verify job to lf-ci-jobs. This job will perform Puppet linting on the specified repository.

    - project:
        name: lf-infra-puppet-mymodule
        project-name: lf-infra-puppet
        project: puppet/modules/mymodule
        jobs:
          - gerrit-puppet-verify
    

Bug Fixes

  • maven-fetch-metadata.sh was not respecting the “-f” (for file path) flag in MAVEN_PARAMS, causing lf-maven-merge jobs that utilize this flag to fail. It will now set a path based on this flag if it is present, or default to the current working directory.
  • Check openjdk $VERSION before setting $JAVA_HOME. This enables jobs to pass “openjdk10” or “openjdk11” on CentOS 7 images to use the OpenJDK version installed in /opt.

v0.28.3

Bug Fixes

  • Compress and upload all jjb-verify XML files to Nexus, to ease out the IOPs on cron jobs that manage the logs on Nexus and optimize job performace by ~8 mins. This is because the job generates around ~2.3K XML files (small files) which is uploaded to Nexus in every run of jjb-verify. Doing this is faster as compared to the Nexus Unpack plugin in the Nexus end unpacking the zip file we upload takes longer.

v0.28.1

Other Notes

  • Update lftools version to v0.19.0.

v0.28.0

New Features

  • New lf-stack-create macro allows job-templates to setup a OpenStack Heat stack, useful for spinning up CSIT labs to run integration tests against. Use with the lf-stack-delete macro.
  • Concurrency for the gerrit-jjb-verify job can now be configured by setting the ‘build-concurrent’ parameter.
  • New macro lf-maven-central is available to deploy artifacts to OSSRH staging for jobs that want to eventually deploy to Maven Central.

    ---
    - job-template:
        name: lf-maven-central-macro-test
    
        #####################
        # Default variables #
        #####################
    
        mvn-central: true
        mvn-global-settings: ""
        mvn-settings: ""
        ossrh-profile-id: ""
    
        #####################
        # Job configuration #
        #####################
    
        builders:
          - lf-maven-central:
              mvn-central: "{mvn-central}"
              mvn-global-settings: "{mvn-global-settings}"
              mvn-settings: "{mvn-settings}"
              ossrh-profile-id: "{ossrh-profile-id}"
    
  • The GERRIT_REFSPEC build parameter can now be used to trigger a test build from the Jenkins Sandbox system against a work in progress packer image patch from a GitHub Pull Request.

Upgrade Notes

  • lf-stack-delete has been modified to be a companion macro to lf-stack-create in order to cleanup the stack at the end of a job run. It now includes a required parameter openstack-cloud to choose the clouds.yaml cloud configuration for the project. Existing users of this macro will need to update their job templates accordingly.
  • Requires JJB 2.8.0 for the jenkins-sandbox-cleanup job to not fail.

    Note

    Despite the failure if JJB 2.8.0 is not available the job will successfully delete all jobs and views, the primary purpose of this job.

Bug Fixes

  • RELENG-1450 All view disappears on Jenkins Sandbox after views are deleted. The All view is now recreated after delete-all is run.

v0.27.0

New Features

  • Add the ability to configure the location of JJB’s cache directory for CI jobs.
  • New view-templates project-view, common-view, and csit-view are available for projects to manage Jenkins views through code.

    To use the project-view template in a project:

    - project:
        name: aaa-view
        views:
          - project-view
    
        project-name: aaa
    

    To use the common-view template in a project:

    - project:
        name: daily-builds
        views:
          - common-view
    
        view-name: Periodic
        view-regex: '.*-periodic-.*'
    

    To use the csit-view template in a project:

    - project:
        name: csit
        views:
          - csit-view
    
        view-name: CSIT
        view-regex: '.*csit.*'
    
    - project:
        name: csit-1node
        views:
          - csit-view
    
        view-name: CSIT-1node
        view-regex: '.*-csit-1node-.*'
    
  • Add support to maven-stage jobs to publish to Maven Central via OSSRH.

    This is accomplished by adding these 2 new optional parameters to the job configuration.

    - gerrit-maven-stage:
        mvn-central: true
        ossrh-profile-id: 7edbe315063867
    
  • The openstack-cron job now has the ability to remove images older than a specified age (default: 30).
  • The openstack-cron job now has the ability to remove orphaned servers.
  • The openstack-cron job now has the ability to remove orphaned stacks.
  • The openstack-cron job now has the ability to remove orphaned volumes.
  • lf-infra-gerrit-scm and lf-infra-github-scm now require a submodule-timeout parameter to provide a timeout value (in minutes) for git fetch operations.
  • All job-templates now provide an optional submodule-timeout parameter for git fetch operations, defaulting to 10 minutes.

Upgrade Notes

  • Some LF projects are already using a common-view template in their local ci-management repo. This common-view is called project-view in global-jjb so rename all instances of common-view to project-view when upgrading and remove the local common-view view-template definition from ci-management.
  • The openstack-cron job now requires a new parameter configured jenkins-urls in order to use the job.
  • Any project using the lf-infra-gerrit-scm and lf-infra-github-scm macros in global-jjb should need to add a submodule-timeout value. It is recommended to default this value to 10 since it is the default used by the Jenkins Git Plugin.

Bug Fixes

  • The jenkins-init scripts dir is now updated to reflect changes recommended for the v0.25.0 release. We unfortunately missed this critical piece in the v0.25.0 release.
  • Specify refspec to be blank for SCM config on github-maven-merge job. Setting the refspec to +refs/pull/*:refs/remotes/origin/pr/* causes there to be no merge job triggered.

Other Notes

  • lftools’ openstack module will now be installed as part of pre-build.
  • The openstack-cron job now runs every hour instead of daily. This is because stack cleanup should happen more regularly.

v0.26.0

New Features

  • Add a new nexus-iq-namespace optional parameter to insert a namespace into Nexus IQ AppID. This is useful for shared Nexus IQ systems where projects might have concern about namespace collision.

    Note

    We recommend when using the namespace to add a trailing - to the value. Eg. ‘odl-‘, this is to make the namespace look nice for example “odl-aaa” is the result of namespace odl-, and project name aaa.

  • Add lf-infra-publish-windows. A publisher for use at the end of Windows based job-templates.

Bug Fixes

  • Fix packer-verify job to correctly work with clouds.yaml config model implemented in global-jjb v0.25.0.

v0.25.1

Bug Fixes

  • jjb-cleanup.sh may be merged with other shell scripts that set -u which causes Jenkins to fail when activating virtualenvs

v0.25.0

New Features

  • Add support to the packer-build job to use clouds.yaml for openstack builder configuration rather than through the cloud-env file. This allows us to simplify the template configuration for openstack builders moving forward.
  • New macro lf-sigul-sign-dir available to sign artifacts in a provided directory using Sigul.

    Usage:

    - lf-sigul-sign-dir:
        sign-dir: '$WORKSPACE/m2repo'
    

    This macro also requires a boolean variable to SIGN_ARTIFACTS to be set to true to activate the macro. We recommend the job-template that uses this macro to define it in the job parameters section.

    Example:

    - bool:
        name: SIGN_ARTIFACTS
        default: '{sign-artifacts}'
        description: Use Sigul to sign artifacts.
    
  • Add Sigul signing support to the maven-staging job. To activate Sigul signing make sure to set sign-artifacts: true. Example:

    - project:
        name: abc
        jobs:
          - gerrit-maven-stage
    
        sign-artifacts: true
    
  • Add lf-stack-delete macro to delete an openstack heat stack at the end of the job.

    This macro requires a parameter defined in the job named STACK_NAME containing the name of the stack to delete.

  • Add lf-infra-wrappers-windows to handle Windows specific wrapper configuration.
  • Refactor lf-infra-wrappers to be for Linux systems and split out the non-linux specific components into a new lf-infra-wrappers-common. This change is seamless for current users of lf-infra-wrappers.

Upgrade Notes

  • Upgrade to global-jjb v0.24.6 before performing this upgrade. This ensures that jjb-verify job pulls in a regex fix that will allow it to verify the v0.25.0 upgrade.
  • Global JJB now has non-JJB YAML configuration and requires action on the ci-management repo when upgrading to this version of Global JJB to prevent JJB from picking up these YAMLs as config. Follow the instructions below BEFORE upgrading globall-jjb:

    cd <git-root>
    git mv jjb/global-jjb global-jjb
    mkdir jjb/global-jjb
    ln -s ../../global-jjb/shell jjb/global-jjb/shell
    ln -s ../../global-jjb/jjb jjb/global-jjb/jjb
    git add jjb/global-jjb
    git commit -sm "Prepare repo for global-jjb v0.25.0"
    
  • Minimum packer version 1.2.5 is now required for the packer-build job.
  • lf-infra-packer-build macro now requires 2 new variables to be passed.
    1. openstack: Set to true if template is built using the openstack builder
    2. openstack-cloud: The clouds.yaml cloud to use when running packer build

Deprecation Notes

  • lftools-install.sh is deprecated and will be removed in a future release. We recommend installing lftools via pip install –user lftools to install instead of using this script.

Bug Fixes

  • Fix pip install pip setuptools which seems to fail against the Nexus 3 proxy. Run them as separate calls to make things happier.
  • jjb-verify will now test on all changes in the jjb directory. The previous pattern was too specific and sometimes missed verifying patches that should be verified.
  • Replace jjb-verify to test on all changes in the shell/* directory.
  • Fix the lftools virtualenv workaround we had to put in place in the tox-verify job by using pip install --user for global tool installs.
  • Fix jobs failing with UNSTABLE build due to install pip==18.0 missing. This change moves all the jobs to using lf-infra-pre-build to install lftools via –user command.
  • Use python -m pip to ensure that we are using the pip version that was installed rather than the OS wrapper version of pip.
  • Fix package listing script in post-builder from causing UNSTABLE build due to difference in the two files being compared.
  • Fix RTD job failing to find PBR install.

Other Notes

  • Update lftools to ~ 0.17.1