Release Notes

v0.25.5-2

New Features

  • –team now lists members of a specific team
  • check_votes now takes click.option(‘–github_repo’) Used in automation to determine is 50% of committers have voted on an INFO.yaml change
  • nexus release now checks “{}/staging/repository/{}/activity” Ensures that Repository is in closed state Checks if Repository is already released (exit 0) Check for failures, if found (exit 1) Added click.option(‘-v’, ‘–verify-only’, is_flag=True, required=False) if -v is passed, only checks for errors, skips release

Bug Fixes

  • Fixes invite to team

v0.25.5

New Features

  • Support multiple nexus sections in lftools.ini

    In the format:

    [nexus.example.org]
    username=
    password=
    
    [nexus.example1.org]
    username=
    password=
    

    [nexus] section is taken from -s “server” passed to release job. https part of passed url is stripped before match.

Upgrade Notes

  • current [nexus] section of lftools.ini must be changed to [nexus.example.com]

    where nexus.example.com matches the “server” string passed to lftools nexus release -s https://nexus.example.com The https part of passed url is stripped before match. example provided would require auth section in lftools.ini of [nexus.example.org]

v0.25.4

Bug Fixes

  • Remove drop of staging repos on release The api returns that the relese is completed. in the background java threads are still running. Then we call drop and nexus has threads promoting and dropping at the same time. In this way we lose data. Something else needs to drop, the api does not correctly handle this.

v0.25.3

Known Issues

  • Pytest 5 has come out and requires Python >= 3.5 which we’re not presently testing on. Pytest is now pinned to 4.6.4 until we update.

Bug Fixes

  • Change out lfidapi module print statements to use the logger facility. This allows us to split appart information, debugging, and error log statements so that they can be easily enabled and captured on the correct streams.
  • There was a subtle bug where a function call was being overwritten by a local variable of the same name and then a call to the function was being attempted.

v0.25.2

Bug Fixes

  • Use requests.put rather than requests.post for deploy_nexus in order to fix Nexus 3 compatibility. This does not affect Nexus 2 compatibility.

v0.25.1

New Features

  • Add a --force option to delete stacks command. This will help with re-factoring the code in global-jjb scripts using in builder-openstack-cron job to remove orphaned stacks/node and continue with the next stack to delete.

Critical Issues

  • A problem was found with the Jenkins OpenStack cloud configuration job routines that were relying on the long name form of installed plugins. As the long name form is more human friendly we are reverting to that configuration.

v0.25.0

New Features

  • Github list and create repositories.

    Usage: Usage: lftools github [OPTIONS] COMMAND [ARGS]…

    Commands:
        audit   List Users for an Org that do not have 2fa enabled.
        create  Create a Github repo for within an Organizations.
        list    List and Organizations GitHub repos.
    
    Options:
      --help    Show this message and exit.
    

Bug Fixes

  • There is a possibility that there exists a file called Archives, and if so, there will be an OSError crash
    02:15:01 File “/home/jenkins/.local/lib/python2.7/site-packages/lftools/deploy.py”, line 236, in deploy_archives 02:15:01 copy_archives(workspace, pattern) 02:15:01 File “/home/jenkins/.local/lib/python2.7/site-packages/lftools/deploy.py”, line 170, in copy_archives 02:15:01 for file_or_dir in os.listdir(archives_dir): 02:15:01 OSError: [Errno 20] Not a directory: ‘/w/workspace/autorelease-update-validate-jobs-fluorine/archives’
    This fix raises an Exception, and exists lftools with (1), if there is any issues with the Archive directory
    (missing, a file instead of directory, or something else)
  • Fix OSError in lftools deploy archives due to pattern

    If the pattern is not properly done, the resulting file list might contain duplicated files.

    This fix will remove the duplicated patterns, as well as the duplicated matched files.

    This fix should fix the following crash 08:24:05 File “/home/jenkins/.local/lib/python2.7/site-packages/lftools/deploy.py”, line 204, in copy_archives 08:24:05 os.makedirs(os.path.dirname(dest)) 08:24:05 File “/usr/lib64/python2.7/os.py”, line 157, in makedirs 08:24:05 mkdir(name, mode) 08:24:05 OSError: [Errno 17] File exists: ‘/tmp/lftools-da.m80YHz/features/benchmark/odl-benchmark-api/target/surefire-reports’

  • Handle config parser correctly which defaults to “[jenkins]” section when no server is passed. This fixes the issue with checking if the key exists in the configuration read before reading the key-value.

    The issue is reproducible by running lftools jenkins plugins –help or tox -e docs, with jenkins.ini missing the “[jenkins]” section.

  • lfidapi create group checks if group exists before posting
  • Unicode compatibility in deploy_logs for Python 2 and 3 was improved in several ways. The former method to pull and write log files did not work properly in Python 3, and was not very robust for Python 2. Both reading and writing logs is now handled in a unicode-safe, 2/3 compatible way.

v0.24.0

New Features

  • List active plugins that have a known vulnerability.

Bug Fixes

  • Add file extensions .html and .xml to ensure they are compressed. .xml files pushed to the log server can be quite large, so the fix ensures that the logs uploaded to Nexus have a smaller foot print.

v0.23.1

New Features

  • Add sigul signing for git tags and Docker containers.

v0.22.2

Bug Fixes

  • Fix the unhelpful stack trace when a deploy nexus-zip fails to upload.

    Traceback (most recent call last):
      File "/home/jenkins/.local/bin/lftools", line 10, in <module>
        sys.exit(main())
      File "/home/jenkins/.local/lib/python2.7/site-packages/lftools/cli/__init__.py", line 110, in main
        cli(obj={})
      File "/usr/lib/python2.7/site-packages/click/core.py", line 721, in __call__
        return self.main(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/click/core.py", line 696, in main
        rv = self.invoke(ctx)
      File "/usr/lib/python2.7/site-packages/click/core.py", line 1065, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/lib/python2.7/site-packages/click/core.py", line 1065, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/lib/python2.7/site-packages/click/core.py", line 894, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/lib/python2.7/site-packages/click/core.py", line 534, in invoke
        return callback(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/home/jenkins/.local/lib/python2.7/site-packages/lftools/cli/deploy.py", line 63, in archives
        deploy_sys.deploy_archives(nexus_url, nexus_path, workspace, pattern)
      File "/home/jenkins/.local/lib/python2.7/site-packages/lftools/deploy.py", line 236, in deploy_archives
        deploy_nexus_zip(nexus_url, 'logs', nexus_path, archives_zip)
      File "/home/jenkins/.local/lib/python2.7/site-packages/lftools/deploy.py", line 362, in deploy_nexus_zip
        raise requests.HTTPError(e.value)
    AttributeError: 'HTTPError' object has no attribute 'value'
    

    Now instead it returns a much more helpful error message:

    ERROR: Failed to upload to Nexus with status code: 401.
    
    test.zip
    
  • Fixes an OSError exception that is not handled, in the lftools command:

    lftools deploy archives

    The code resides in the copy_archives function in deploy.py file.

    This exception is caused by a missing archives directory, which a for loop expects to be there. The fix is simply to verify if archives file/directory exists, and if it does then perform the for loop.

    12:07:36 File “/home/jenkins/.local/lib/python2.7/site-packages/lftools/deploy.py”, line 166, in copy_archives 12:07:36 for file_or_dir in os.listdir(archives_dir): 12:07:36 OSError: [Errno 2] No such file or directory: ‘/w/workspace/music-mdbc-master-verify-java/archives’

v0.22.0

New Features

  • check-votes

    Usage: lftools infofile check-votes [OPTIONS] INFO_FILE GERRIT_URL

    Commands:
      Check for Majority of votes on a gerrit patchset that changes
      an INFO.yaml file.
    
    Options:
      --help    Show this message and exit.
    

v0.21.0

New Features

  • Allow passing serial as third argument to sign_dir

    Parallel-signing using sigul is resulting in NSPR reset errors, so allow passing “serial” to the sign_dir function as a third argument to request serial signing of directory contents.

v0.20.0

New Features

  • Gerrit project create and github enable replication commands.

    Usage: lftools gerrit [OPTIONS] COMMAND [ARGS]…

    Commands:
      create  Create and configure permissions for a new gerrit repo.
    
    Options:
      --enable  Enable replication to Github.
                This skips creating the repo.
      --parent  Specify parent other than "All-Projects"
      --help    Show this message and exit.
    
  • LFID Api Tools.

    Usage: lftools lfidapi [OPTIONS] COMMAND [ARGS]…

    Commands:
      create-group    Create group.
      invite          Email invitation to join group.
      search-members  List members of a group.
      user            Add and remove users from groups.
    
    Options:
      --help    Show this message and exit
    
  • Add Nexus command to release one or more staging repositories. Via the Nexus 2 REST API, this command performs both a “release” and a “drop” action on the repo(s), in order to best reproduce the action of manually using the “Release” option in the Nexus UI.

    Usage: lftools nexus release [OPTIONS] [REPOS]…

    Options:
    -s, --server TEXT
     

    Nexus server URL. Can also be set as NEXUS_URL in the environment. This will override any URL set in settings.yaml.

  • Add command to list openstack containers.

    Usage:

    lftools openstack --os-cloud example object list-containers
    
  • This command will collect all tags from both Nexus3 and Docker Hub, for a particular org (for instance ‘onap’), as well as a repo (default all repos). With this information, it will calculate a list of valid tags that needs to be copied to Docker Hub from Nexus3.

    Usage:

    lftools nexus docker releasedockerhub

    Options:
    -o, --org TEXT

    Specify repository organization. [required]

    -r, --repo TEXT
     

    Only repos containing this string will be selected. Default set to blank string, which is every repo.

    -s, --summary

    Prints a summary of missing docker tags.

    -v, --verbose

    Prints all collected repo/tag information.

    -c, --copy

    Copy missing tags from Nexus3 repos to Docker Hub repos.

    -p, --progbar

    Display a progress bar for the time consuming jobs.

  • Verify YAML Schema.

    Usage: Usage: lftools schema verify [OPTIONS] YAMLFILE SCHEMAFILE

    Commands:
      verify a yaml file based on a schema file.
    
    Options:
      --help    Show this message and exit.
    

Known Issues

  • Currently, if the Docker Hub repo is missing, it is not created specifically, but implicitly by docker itself when we push the docker image to an non- existing Docker Hub repo.

    The command handles any org (onap or hyperledger for instance), “BUT” it requires that the versioning pattern is #.#.# (1.2.3) for the project. In regexp terms : ^d+.d+.d+$

Critical Issues

  • Before you give the “lftools nexus docker releasedockerhub” command please ensure you have manually logged in to both Nexus as well as to Docker.

    sudo docker login —> DOCKER Credentials sudo docker login nexus3.onap.org:10002 -u <yourLFID>

v0.19.0

New Features

  • Provide additional methods to pass LFID to lftools than lftools.ini
    1. Via explicit --password parameter
    2. Via environment variable LFTOOLS_PASSWORD
    3. At runtime if --interactive mode is set
  • Refactored deploy_nexus function from shell/deploy to pure Python to be more portable with Windows systems. Also added a number of unit tests to cover all executable branches of the code.
  • Refactored deploy_nexus_stage function from shell/deploy to pure Python to be more portable with Windows systems. Also added a number of unit tests to cover all executable branches of the code.
  • Add --conf parameter to jenkins subcommand to allow choosing a jjb config outside of the default paths.
  • Docker list and delete commands for Nexus docker repos.

    Usage: lftools nexus docker [OPTIONS] COMMAND [ARGS]…

    Commands:
      delete  Delete all images matching the PATTERN.
      list    List images matching the PATTERN.
    
  • The shell/deploy file’s copy_archives() function has been reimplemented in pure Python for better portability to Windows systems.
  • Refactored deploy_archives() function from shell/deploy to pure Python to be more portable with Windows systems.
  • Refactored deploy_logs() function from shell/deploy to pure Python to be more portable with Windows systems.
  • Refactored deploy_nexus_zip() function from shell/deploy to pure Python to be more portable with Windows systems.
  • Refactored nexus_stage_repo_close(), and nexus_repo_stage_create() function from shell/deploy to pure Python to be more portable with Windows systems. Also added a number of unit tests to cover all executable branches of the code.
  • Refactored upload_maven_file_to_nexus function from shell/deploy to pure Python to be more portable with Windows systems. Also added a number of unit tests to cover all executable branches of the code.

Deprecation Notes

  • shell/deploy script’s deploy_nexus function is now deprecated and will be removed in a future release.
  • shell/deploy script’s deploy_nexus_stage function is now deprecated and will be removed in a future release.
  • The shell/deploy script’s copy_archives() function is now deprecated and will be removed in a later version. We recommend migrating to the lftools pure Python implementation of this function.
  • shell/deploy script’s deploy_archives() function is now deprecated and will be removed in a future release.
  • shell/deploy script’s deploy_logs() function is now deprecated and will be removed in a future release.
  • shell/deploy script’s deploy_nexus_zip() function is now deprecated and will be removed in a future release.
  • shell/deploy script’s nexus_stage_repo_close() and nexus_stage_repo_create() function is now deprecated and will be removed in a future release.
  • shell/deploy script’s upload_maven_file_to_nexus function is now deprecated and will be removed in a future release.

Bug Fixes

  • Running the lftools CLI was unexpectedly creating unnecessary gpg-signatures directories in the /tmp directory and not cleaning them up.

v0.18.0

New Features

  • Add new cmd to fetch Jenkins token from user account. An optional --change parameter can be passed to have Jenkins change the API token.

    Usage: lftools jenkins token [OPTIONS]

    Get API token.

    Options:
    --change

    Generate a new API token.

    --help

    Show this message and exit.

  • Add jenkins token init command to initialize a new server section in jenkins_jobs.ini. This command uses credentials found in lftools.ini to initialize the new Jenkins server configuration.

    Usage: lftools jenkins token init [OPTIONS] NAME URL

  • Add jenkins token reset command to automatically reset API tokens for all Jenkins systems configured in jenkins_jobs.ini.

    Usage: lftools jenkins token reset [OPTIONS] [SERVER]

  • We now support locating the jenkins_jobs.ini in all the same default search paths as JJB supports. Specifically in this order:
    1. $PWD/jenkins_jobs.ini
    2. ~/.config/jenkins_jobs/jenkins_jobs.ini
    3. /etc/jenkins_jobs/jenkins_jobs.ini
  • Add a new delete-stale option to the stack command.

    This function compares running builds in Jenkins to active stacks in OpenStack and determines if there are orphaned stacks and removes them.

  • Add an openstack image share sub-command to handle sharing images between multiple tenants. Command accepts a space-separated list of tenants to share the provided image with.

    Usage: lftools openstack image share [OPTIONS] IMAGE [DEST]...

  • Add an openstack image upload sub-command to handle uploading images to openstack.

    Usage: Usage: lftools openstack image upload [OPTIONS] IMAGE NAME...

Bug Fixes

  • The get-credentials command is now fixed since it was was broken after refactoring done in Gerrit patch I2168adf9bc992b719da6c0350a446830015e6df6.

Other Notes

  • Refactored the Jenkins object into a class to allow us to reuse it outside of the Jenkins command group.

v0.17.0

New Features

  • Add support to the jenkins command to parse jenkins_jobs.ini for configuration if server parameter passed is not a URL.
  • Add a jobs sub-command to jenkins command to enable or disable Jenkins Jobs that match a regular expression.

Other Notes

  • Enhance logger subsystem to work better as a CLI program. This is a first step to migrating all lftools subsystems to use the logger instead of print statements everywhere.

v0.16.1

Bug Fixes

  • The v0.16.0 pulled in a new ldap module which breaks if the ldap devel libraries are not available on the system trying to use it. This hotfix makes the ldap module optional.

v0.16.0

New Features

  • Add a new --debug flag to enable extra troubleshooting information. This flag can also be set via environment variable DEBUG=True.
  • $ lftools ldap

    Usage: lftools ldap [OPTIONS] COMMAND [ARGS]…

    Commands:
      autocorrectinfofile  Verify INFO.yaml against LDAP group.
      csv                  Query an Ldap server.
      inactivecommitters   Check committer participation.
      yaml4info            Build yaml of commiters for your INFO.yaml.
    
  • $ lftools infofile

    Commands:
      get-committers   Extract Committer info from INFO.yaml or LDAP...
      sync-committers  Sync committer information from LDAP into...
    

Deprecation Notes

  • Remove support for modifying the logger via logging.ini. It was a good idea but in practice this is not really used and adds extra complexity to lftools.

Bug Fixes

  • Fix broken openstack and sign help command output in docs.