Release Notes

v0.39.0

Critical Issues

  • lftools v0.24.0 introduced a major issue which caused the Jenkins cloud configuration merge job for OpenStack clouds to fail. This has been corrected in lftools v0.25.1

Bug Fixes

  • Add missing git-url variable in {project-name}-releases-merge-{stream} job.
  • Add the {stream} name in releases-verify and releases-merge jobs.
  • Some ONAP components like DCAEGEN2 do not host a version.properties file in the root of their repos. We need to be able to provide a location and/or different name for the version.properties file for jobs using the lf-maven-versions-plugin builder step.

v0.38.4

New Features

  • Group {project-name}-releases-verify and {project-name}-releases-merge into {project-name}-gerrit-release-jobs.

    Add test jobs for lf-release-jobs.

Bug Fixes

  • Change parameter names used to specify container tag method, with a new default to use the fixed string ‘latest’ as a tag. Merge two shell scripts into one instead of using Jenkins conditional steps. Extend to accept a custom directory for the container-tag.yaml if the yaml-file method is used to set the docker tag information; this is an optional variable which is set to empty by default, and falls back to DOCKER_ROOT.
  • Add missing scm block in gerrit-releases-merge job definition. Add missing submodule-disable variable for jobs using lf-infra-gerrit-scm. Update documentation for gerrit-releases-merge and gerrit-releases-verify to remove submodule options as optional parameters.
  • Update lftools version to v0.25.0
  • Add missing $ to variable tag_file so the yq query can pull the tag from the container-tag.yaml file.
  • Add -l to /bin/bash shebang line at top of docker-get-container-tag.sh to make it a login shell, which automatically includes /home/jenkins/.local/bin on the path, because that is where pip installs the yq command.

v0.38.3

Bug Fixes

  • Add trigger on cron to docker merge macro to support regular rebuilds. This makes the merge macro match the behavior of most other jobs.
  • Add yamllint verification to INFO.yaml files.

Other Notes

  • Update lftools version to v0.24.0.

v0.38.2

Bug Fixes

  • Add missing config for triggering on file paths to docker macros and templates, namely gerrit_trigger_file_paths and github_included_regions, to make the verify and merge macros and templates match the behavior of other jobs.

v0.38.1

Bug Fixes

  • Install yq to be used to read yaml files. In specific, it will be needed to read container-tag.yaml.
  • When calling builder step macros “lf-docker-get-container-tag”, “lf-docker-build” and “lf-docker-push”, make sure the needed variables are also passed explicitly to avoid these variables appear as undefined.
  • Allow DOCKER_ARGS to be empty in docker-build.sh. This is not a required parameter, it can be empty.

    Remove container reference in docker-get-git-describe.sh. The CONTAINER_PUSH_REGISTRY already gets added in the docker-push script. No need to add it again.

    Rename image_name to image_build_tag in docker-get-yaml-tag to match docker-get-git-describe. Add missing “DOCKER_NAME” in the DOCKER_IMAGE.

    docker-get-git-describe.sh and docker-get-yaml-tag.sh should only export the tag variable. Let docker-build.sh process DOCKER_NAME

Other Notes

  • Add yq install as part of python-tools-install.sh Allow future scripts to use yq package.

v0.38.0

New Features

  • gerrit-tox-verify now has a new parameter gerrit-skip-vote (bool) to control whether Jenkins should skip voting depending on the build outcome. It defaults to false since it is the default used by the Jenkins Gerrit Trigger Plugin.
  • gerrit-docker-verify runs for new commits and runs a build of the affected Docker images.
  • gerrit-docker-merge runs for merged commits, runs a build of the affected Docker images and pushes the images to a specified Docker registry.
  • New lf-release-job-merge and lf-release-job-verify templates allow projects to have self-serve releases. Project will create a tagname.yaml file in the releases/ directory of their git repo. example:

    $ cat releases/4.0.0.yaml
    ---
    distribution_type: 'maven'
    version: '4.0.0'
    project: 'odlparent'
    log_dir: 'odlparent-maven-release-master/11/'
    #below is optional
    maven_central_url: 'oss.sonatype.org'
    
  • lf-infra-gerrit-scm and lf-infra-github-scm now require a submodule-disable parameter (bool) to control whether submodules are ignored or not during git fetch operations.
  • All job-templates now provide an optional submodule-disable parameter for git fetch operations, defaulting to false.

Upgrade Notes

  • Any project using the lf-infra-gerrit-scm and lf-infra-github-scm macros in global-jjb should need to add a submodule-disable value. It is recommended to default this value to false since it is the default used by the Jenkins Git Plugin.
  • Update gerrit comment trigger to use a more standard regex and avoid triggering jobs, when these keywords are intended to be used as code review comments between users. Also improve the regexs to make them more succinct and readable.

Bug Fixes

  • Add missing config for triggering on file paths to maven stage macros and templates, namely gerrit_trigger_file_paths nad github_included_regions, to make those macros and templates match the behavior of maven verify and maven merge.
  • fix multiple jobs created using same job-template update same github check status due to hard coded status-context to Maven Verify. Now appending status-context with maven-version and java-version to make it unique. And create different status checks in the github. fix applied for maven verify and maven docker verify jobs
  • Fix log shipping script to not require a LOGS_SERVER. There was a regression that caused the log shipping script to start requiring a LOGS_SERVER which fails in the case of a system that does not have that optional environment variable set.
  • Handle multiple search extension or patterns passed by upstream JJB ARCHIVE_ARTIFACTS param as a single string by spliting these values before being passed to lftools deploy archives.

    ARCHIVE_ARTIFACTS="**/*.prop \
                      **/*.log \
                      **/target/surefire-reports/*-output.txt \
                      **/target/failsafe-reports/failsafe-summary.xml \
                      **/hs_err_*.log **/target/feature/feature.xml"
    

    For example, the above env variable passed to the script and to lftools deploy archives as:

    lftools deploy archives -p **/*.prop \
                      **/*.log \
                      **/target/surefire-reports/*-output.txt \
                      **/target/failsafe-reports/failsafe-summary.xml \
                      **/hs_err_*.log **/target/feature/feature.xml \
                      "$NEXUS_URL" \
                      "$NEXUS_PATH" \
                      "$WORKSPACE"
    

    The correct way of passing this as per lftools implmentation is:

    lftools deploy archives -p '**/*.prop' \
                      -p '**/*.log' \
                      -p '**/target/surefire-reports/*-output.txt' \
                      -p '**/target/failsafe-reports/failsafe-summary.xml' \
                      -p '**/hs_err_*.log' \
                      -p '**/target/feature/feature.xml' \
                      "$NEXUS_URL" \
                      "$NEXUS_PATH" \
                      "$WORKSPACE"
    
  • Fix error with handling unbound arrays for search extensions, when using set -u. The correct way of using this.

    set -u
    arr=()
    echo "output: '${arr[@]}'"
    bash: arr[@]: unbound variable
    echo "output: '${arr[@]:-}'"
    foo: ''
    
  • lf-maven-versions-plugin builder step needs to run before maven-patch-release.sh as this second script contains a condition to confirm if the maven vesions plugin was selected as a way to remove the ‘SNAPSHOT’ pattern from the pom.xml files. lf-maven-docker-stage was based on lf-maven-stage and it seems that these particular builder steps were switched in place accidentally.
  • The logs-deploy.sh script now allows ARCHIVE_ARTIFACTS to contain zero or more files.
  • request-2.22.0 does not work with python-3.4.9, so pin requests to v2.21.0 to address the tox failures.

v0.37.2

Bug Fixes

  • Use maven goal install (not deploy) in the maven + docker verify job. An image cannot be pushed by a verification job, and the deploy target directs the plugin to push.

v0.37.1

Bug Fixes

  • Remove maven-versions-plugin-set-version variable in newly added macro. This is a variable that does not need to be defined by the users of the jobs. The version needed in this builder step is inherited from versions.properties as “release_version”.

v0.37.0

New Features

  • Add verify, merge and stage templates for Java projects that build and wrap a JAR (e.g., a Spring-Boot application) inside a Docker image, and do not need to deploy any JAR libraries or POM files.

Upgrade Notes

  • The next release of common-packer will require a minimum version of 1.3.2 for packer. The current release of packer is 1.4.0.

Bug Fixes

  • The packer-merge job for Gerrit systems was improperly configured to use the Gerrit Trigger choosing strategy and not default. This caused issues unexpected issues with retriggering merged changes when the expectation was that it would pick up the lastest change as per normal.
  • This is a variable that does not need to be defined by the users of the jobs. The version needed in this builder step is inherited from versions.properties as “release_version” and it is fixed as that. This also helps teams not having to define this version in 2 places and just rely on version.properties.
  • Projects using maven versions plugin let this plugin take care or updating their versions in the pom.xml. When maven-versions-plugin is set to “true”, skip the stripping of SNAPSHOTS from the pom.xml files. maven-versions-plugin is set to “false” by default.

Other Notes

  • Update example Jenkins Init Script in README to redirect all output to a log file.

v0.36.0

Prelude

WhiteSource is a security and license compliance management platform. It is used to perform scans on a great variety of coding and scripting languages.

New Features

  • New comment-to-gerrit builder will comment back to gerrit patchset if a file called gerrit_comment.txt is created by the build.
  • Allows maven to run a clean install step before the WhiteSource scan runs the Unified Agent to fetch additional dependencies. Set to false by default.
  • Job {project-name}-whitesource-scan-{stream} uses the WhiteSource Unified Agent scanner CLI tool to perform the code scan and report the results into the WhiteSource dashboard.

Bug Fixes

  • Tag releases will now trigger a docs build to regenerate and update the release note link.
  • Update jenkins-cfg-verify job to validate new images names obtained from $GERRIT_REFSPEC instead of the master branch.
  • Hardcode project version to the “GERRIT_BRANCH”. Follow previous convention from CLM where reports were versioned after the branch name. Fix minor nits with bash varaibles.
  • wss-unified-agent.config file should not be opened for configuration to tech teams. The config file should be part of Jenkins Settings Files and called via Managed Files. wss-unified-agent.config must be created in Jenkins config files based on wss-unified-agent.config.example.

Other Notes

  • To run this job, a configuration file is needed (wss-unified-agent.config.example). A new secret text credential will need to be created. (ID=wss-apiKey Secret=WhiteSource organization API key)
  • Update lftools version to v0.23.1.

v0.35.0

New Features

  • The jjb-merge job now has a new parameter jjb-workers to allow configuration of the number of threads to run update with. Default is 0 which is equivalent to the number of CPU cores available on the system.
  • New info-vote-verify macro Will count votes against an INFO.yaml change and sumbit automatically if a majority of committers vote +1 or +2 on the change. Job is triggered by +2 votes or a comment of “vote”

Other Notes

  • The Maven Verify job will now call -Dmaven.source.skip to skip source jar generation in the verify job. This saves us some time in the verify build as the source artifacts are not useful in a verify job.

v0.34.0

New Features

  • jenkiins-init-scripts The ‘ciman’ repo is not longer required to be located in ‘/opt/ciman’.
  • lf-maven-set-version conditional step for lf-maven-stage to allow teams to run Maven versions plugin to update their artifact versions. Step will run if maven-versions-plugin is set to true.
  • Support for the Throttle Plugin is added to JJB jobs so static build servers can restrict the number of concurrent JJB jobs ran at the same time.

    This must be explicitly enabled by setting throttle-enabled on the jobs.

Bug Fixes

  • Adapt maven path search for files and dirs. The “-f” maven param can specify both a directory, in which case it will look for “pom.xml” in the directory, or a specific file. The original version of this search was only compatible with directories that contain a pom.xml file.
  • Update the lf-maven-cental macro documentation and example templates with the missing requireed params.
  • Fix JAVA_HOME for openjdk11 on CentOS 7 to use the OpenJDK version installed in /usr/lib/jvm/java-11-openjdk.
  • The JJB Deploy Job is configured to trigger only if the Gerrit comment starts with the jjb-deploy keyword.

    Without the regex being optimized the job triggers on any occurance of the jjb-deploy keyword in a Gerrit comment, with is waste infra resources.

    Example of a valid command in Gerrit comment that triggers the job:

    jjb-deploy builder-jjb-*

    Example of a invalid command in Gerrit comment that would _not_ trigger the job:

    Update the job. jjb-deploy builder-jjb-*

v0.33.0

New Features

  • jenkins-init-scripts If the environmental variable ‘SWAP_SIZE’ is set when the ‘init.sh’ script is called, then a ‘SWAP_SIZE’ GB swap space will be configured. If ‘SWAP_SIZE’ is ‘0’ or is not a valid integer, then no swap space is configured. If it is unset then 1GB of swap will be configured. Previously the swap size was fixed at 1GB.
  • jenkins-init-scripts If the work directory or volume (/w) aleady exists, the ownership will be recursivly set to ‘jenkins:jenkins’. Previously only the top directory /w was owned by ‘jenkins:jenkins’
  • lf-sigul-sign-dir macros now supports a sign-mode parameter which allows jobs to choose to sign artifacts using either parallel mode or serial mode (default).

Upgrade Notes

  • lf-sigul-sign-dir users need to add a new parameter sign-mode to their job-templates setting either parallel or serial as the value, we recommend setting serial mode for this setting.

    {project-name}-maven-stage-{stream}’s Sigul signer now defaults to serial mode instead of the previous parallel behaviour. To change this back to the previous behaviour pass the “sign-mode” parameter to the job template:

    - project:
        name: parallel-sign
        jobs:
          - gerrit-maven-stage:
              sign-mode: parallel
    

v0.31.0

New Features

  • New job-template {project-name}-release-announce for lf-releng projects to automate release announcement emails.
  • Add support for pushing Sonar results to SonarCloud. Refer to Maven Sonar docs for details.

Upgrade Notes

  • Jobs using the lf-maven-stage macro now need to update to the new usage. Preparation calls to lf-provide-maven-settings, lf-infra-create-netrc, and lf-provide-maven-settings-cleanup are no longer necessary to prepare the lf-maven-stage macro.

    Usage:

    - lf-maven-stage:
       mvn-global-settings: 'global-settings'
       mvn-settings: 'settings'
       mvn-staging-id: 'staging profile id'
    

v0.30.0

New Features

  • Packer merge jobs now include the image name in the Jenkins build description.

Bug Fixes

  • Extend ${JOB_NAME} to include {java-version} parameter to support jobs to build with multiple versions of openjdk{8,11}.
  • Modified lf-maven-jobs.yaml sonar cron entry to ‘{obj:cron}’ to pass value from custom user config file.

v0.29.0

New Features

  • Add a puppet-verify job to lf-ci-jobs. This job will perform Puppet linting on the specified repository.

    - project:
        name: lf-infra-puppet-mymodule
        project-name: lf-infra-puppet
        project: puppet/modules/mymodule
        jobs:
          - gerrit-puppet-verify
    

Bug Fixes

  • maven-fetch-metadata.sh was not respecting the “-f” (for file path) flag in MAVEN_PARAMS, causing lf-maven-merge jobs that utilize this flag to fail. It will now set a path based on this flag if it is present, or default to the current working directory.
  • Check openjdk $VERSION before setting $JAVA_HOME. This enables jobs to pass “openjdk10” or “openjdk11” on CentOS 7 images to use the OpenJDK version installed in /opt.

v0.28.3

Bug Fixes

  • Compress and upload all jjb-verify XML files to Nexus, to ease out the IOPs on cron jobs that manage the logs on Nexus and optimize job performace by ~8 mins. This is because the job generates around ~2.3K XML files (small files) which is uploaded to Nexus in every run of jjb-verify. Doing this is faster as compared to the Nexus Unpack plugin in the Nexus end unpacking the zip file we upload takes longer.

v0.28.1

Other Notes

  • Update lftools version to v0.19.0.

v0.28.0

New Features

  • New lf-stack-create macro allows job-templates to setup a OpenStack Heat stack, useful for spinning up CSIT labs to run integration tests against. Use with the lf-stack-delete macro.
  • Concurrency for the gerrit-jjb-verify job can now be configured by setting the ‘build-concurrent’ parameter.
  • New macro lf-maven-central is available to deploy artifacts to OSSRH staging for jobs that want to eventually deploy to Maven Central.

    ---
    - job-template:
        name: lf-maven-central-macro-test
    
        #####################
        # Default variables #
        #####################
    
        mvn-central: true
        mvn-global-settings: ''
        mvn-settings: ''
        ossrh-profile-id: ''
    
        #####################
        # Job configuration #
        #####################
    
        builders:
          - lf-maven-central:
              mvn-central: '{mvn-central}'
              mvn-global-settings: '{mvn-global-settings}'
              mvn-settings: '{mvn-settings}'
              ossrh-profile-id: '{ossrh-profile-id}'
    
  • The GERRIT_REFSPEC build parameter can now be used to trigger a test build from the Jenkins Sandbox system against a work in progress packer image patch from a GitHub Pull Request.

Upgrade Notes

  • lf-stack-delete has been modified to be a companion macro to lf-stack-create in order to cleanup the stack at the end of a job run. It now includes a required parameter openstack-cloud to choose the clouds.yaml cloud configuration for the project. Existing users of this macro will need to update their job templates accordingly.
  • Requires JJB 2.8.0 for the jenkins-sandbox-cleanup job to not fail.

    Note

    Despite the failure if JJB 2.8.0 is not available the job will successfully delete all jobs and views, the primary purpose of this job.

Bug Fixes

  • RELENG-1450 All view disappears on Jenkins Sandbox after views are deleted. The All view is now recreated after delete-all is run.

v0.27.0

New Features

  • Add the ability to configure the location of JJB’s cache directory for CI jobs.
  • New view-templates project-view, common-view, and csit-view are available for projects to manage Jenkins views through code.

    To use the project-view template in a project:

    - project:
        name: aaa-view
        views:
          - project-view
    
        project-name: aaa
    

    To use the common-view template in a project:

    - project:
        name: daily-builds
        views:
          - common-view
    
        view-name: Periodic
        view-regex: '.*-periodic-.*'
    

    To use the csit-view template in a project:

    - project:
        name: csit
        views:
          - csit-view
    
        view-name: CSIT
        view-regex: '.*csit.*'
    
    - project:
        name: csit-1node
        views:
          - csit-view
    
        view-name: CSIT-1node
        view-regex: '.*-csit-1node-.*'
    
  • Add support to maven-stage jobs to publish to Maven Central via OSSRH.

    This is accomplished by adding these 2 new optional parameters to the job configuration.

    - gerrit-maven-stage:
        mvn-central: true
        ossrh-profile-id: 7edbe315063867
    
  • The openstack-cron job now has the ability to remove images older than a specified age (default: 30).
  • The openstack-cron job now has the ability to remove orphaned servers.
  • The openstack-cron job now has the ability to remove orphaned stacks.
  • The openstack-cron job now has the ability to remove orphaned volumes.
  • lf-infra-gerrit-scm and lf-infra-github-scm now require a submodule-timeout parameter to provide a timeout value (in minutes) for git fetch operations.
  • All job-templates now provide an optional submodule-timeout parameter for git fetch operations, defaulting to 10 minutes.

Upgrade Notes

  • Some LF projects are already using a common-view template in their local ci-management repo. This common-view is called project-view in global-jjb so rename all instances of common-view to project-view when upgrading and remove the local common-view view-template definition from ci-management.
  • The openstack-cron job now requires a new parameter configured jenkins-urls in order to use the job.
  • Any project using the lf-infra-gerrit-scm and lf-infra-github-scm macros in global-jjb should need to add a submodule-timeout value. It is recommended to default this value to 10 since it is the default used by the Jenkins Git Plugin.

Bug Fixes

  • The jenkins-init scripts dir is now updated to reflect changes recommended for the v0.25.0 release. We unfortunately missed this critical piece in the v0.25.0 release.
  • Specify refspec to be blank for SCM config on github-maven-merge job. Setting the refspec to +refs/pull/*:refs/remotes/origin/pr/* causes there to be no merge job triggered.

Other Notes

  • lftools’ openstack module will now be installed as part of pre-build.
  • The openstack-cron job now runs every hour instead of daily. This is because stack cleanup should happen more regularly.

v0.26.0

New Features

  • Add a new nexus-iq-namespace optional parameter to insert a namespace into Nexus IQ AppID. This is useful for shared Nexus IQ systems where projects might have concern about namespace collision.

    Note

    We recommend when using the namespace to add a trailing - to the value. Eg. ‘odl-‘, this is to make the namespace look nice for example “odl-aaa” is the result of namespace odl-, and project name aaa.

  • Add lf-infra-publish-windows. A publisher for use at the end of Windows based job-templates.

Bug Fixes

  • Fix packer-verify job to correctly work with clouds.yaml config model implemented in global-jjb v0.25.0.

v0.25.1

Bug Fixes

  • jjb-cleanup.sh may be merged with other shell scripts that set -u which causes Jenkins to fail when activating virtualenvs

v0.25.0

New Features

  • Add support to the packer-build job to use clouds.yaml for openstack builder configuration rather than through the cloud-env file. This allows us to simplify the template configuration for openstack builders moving forward.
  • New macro lf-sigul-sign-dir available to sign artifacts in a provided directory using Sigul.

    Usage:

    - lf-sigul-sign-dir:
        sign-dir: '$WORKSPACE/m2repo'
    

    This macro also requires a boolean variable to SIGN_ARTIFACTS to be set to true to activate the macro. We recommend the job-template that uses this macro to define it in the job parameters section.

    Example:

    - bool:
        name: SIGN_ARTIFACTS
        default: '{sign-artifacts}'
        description: Use Sigul to sign artifacts.
    
  • Add Sigul signing support to the maven-staging job. To activate Sigul signing make sure to set sign-artifacts: true. Example:

    - project:
        name: abc
        jobs:
          - gerrit-maven-stage
    
        sign-artifacts: true
    
  • Add lf-stack-delete macro to delete an openstack heat stack at the end of the job.

    This macro requires a parameter defined in the job named STACK_NAME containing the name of the stack to delete.

  • Add lf-infra-wrappers-windows to handle Windows specific wrapper configuration.
  • Refactor lf-infra-wrappers to be for Linux systems and split out the non-linux specific components into a new lf-infra-wrappers-common. This change is seamless for current users of lf-infra-wrappers.

Upgrade Notes

  • Upgrade to global-jjb v0.24.6 before performing this upgrade. This ensures that jjb-verify job pulls in a regex fix that will allow it to verify the v0.25.0 upgrade.
  • Global JJB now has non-JJB YAML configuration and requires action on the ci-management repo when upgrading to this version of Global JJB to prevent JJB from picking up these YAMLs as config. Follow the instructions below BEFORE upgrading globall-jjb:

    cd <git-root>
    git mv jjb/global-jjb global-jjb
    mkdir jjb/global-jjb
    ln -s ../../global-jjb/shell jjb/global-jjb/shell
    ln -s ../../global-jjb/jjb jjb/global-jjb/jjb
    git add jjb/global-jjb
    git commit -sm "Prepare repo for global-jjb v0.25.0"
    
  • Minimum packer version 1.2.5 is now required for the packer-build job.
  • lf-infra-packer-build macro now requires 2 new variables to be passed.
    1. openstack: Set to true if template is built using the openstack builder
    2. openstack-cloud: The clouds.yaml cloud to use when running packer build

Deprecation Notes

  • lftools-install.sh is deprecated and will be removed in a future release. We recommend installing lftools via pip install –user lftools to install instead of using this script.

Bug Fixes

  • Fix pip install pip setuptools which seems to fail against the Nexus 3 proxy. Run them as separate calls to make things happier.
  • jjb-verify will now test on all changes in the jjb directory. The previous pattern was too specific and sometimes missed verifying patches that should be verified.
  • Replace jjb-verify to test on all changes in the shell/* directory.
  • Fix the lftools virtualenv workaround we had to put in place in the tox-verify job by using pip install --user for global tool installs.
  • Fix jobs failing with UNSTABLE build due to install pip==18.0 missing. This change moves all the jobs to using lf-infra-pre-build to install lftools via –user command.
  • Use python -m pip to ensure that we are using the pip version that was installed rather than the OS wrapper version of pip.
  • Fix package listing script in post-builder from causing UNSTABLE build due to difference in the two files being compared.
  • Fix RTD job failing to find PBR install.

Other Notes

  • Update lftools to ~ 0.17.1