Release Notes

v0.17.5

Known Issues

  • Resolved “Invalid Image Id” errors in Jenkins EC2 plugin when attempting to provision Ubuntu and CentOS Stream 9 AMIs.

  • Resolved “No space left on device” errors during Jenkins builds by increasing root volume size to 10GB.

  • Fixed AWS AMI builds that were failing due to package repository availability issues on Ubuntu (stale package lists) and CentOS Stream 9 (obsolete package references).

Upgrade Notes

  • No action required for existing deployments. These changes fix broken AWS AMI build functionality. New AMIs built with these fixes will:

    • Have SR-IOV enhanced networking enabled (ENA support)

    • Have 10GB root volumes instead of 8GB

    • Be compatible with Jenkins EC2 plugin for automatic provisioning

    • Use current Ubuntu base images from Canonical

    • Have correct package dependencies for CentOS Stream 9

    Note: Ansible role fixes for lf-recommended-tools and lf-dev-libs need to be submitted separately to the ansible/roles repository for CentOS Stream 9 support.

Bug Fixes

  • Fixed AWS AMI builds for Ubuntu 20.04, 22.04, 24.04, and CentOS Stream 9 that were failing due to incorrect source AMI filters and package availability issues.

    Ubuntu AMI Build Fixes:

    • Updated source AMI filters to use Canonical owner ID (099720109477) instead of aws-marketplace

    • Removed product code filtering which is not applicable for Canonical Ubuntu AMIs

    • Updated AMI name patterns to match current Ubuntu naming conventions: ubuntu-20.04, ubuntu-22.04, ubuntu-24.04

    • Added apt-get update to provision/install-python.sh to ensure package lists are current before installing dependencies

    CentOS Stream 9 Build Fixes:

    • Fixed version-specific conditionals in provision/install-base-pkgs-RedHat.yaml to use version('8', '==') instead of version('8', '>=') to prevent incompatible package installation on Stream 9

    • Removed obsolete packages not available in CentOS Stream 9: compat-openssl10*, python36*, python36-devel, python36-pip, python36-setuptools, python36-virtualenv, openssl11*, openssl11-devel

    • Updated python3-virtualenv dependency handling for Stream 9 compatibility

    AWS-Specific Enhancements:

    • Added enable_ena_support = true to enable SR-IOV enhanced networking on all AMIs for better performance and Jenkins EC2 plugin compatibility

    • Increased root volume size from 8GB to 10GB to prevent “No space left on device” errors during Jenkins builds

    • Added IMDS v2 metadata support configuration

    • Standardized instance types to t3.small for consistent builds

    Verified Working AMIs:

    • Ubuntu 22.04: ami-049dc0c4ecd9335ae (tested in Jenkins)

    • CentOS Stream 9: ami-0541c45aeac2e4a04 (build successful)

    Files Modified:

    • provision/install-base-pkgs-RedHat.yaml

    • provision/install-python.sh

    • templates/builder-aws.pkr.hcl

    • vars/centos-cs-9.pkrvars.hcl

    • vars/ubuntu-20.04-arm64.pkrvars.hcl

    • vars/ubuntu-20.04.pkrvars.hcl

    • vars/ubuntu-22.04.pkrvars.hcl

    • vars/ubuntu-24.04-arm64.pkrvars.hcl

    • vars/ubuntu-24.04.pkrvars.hcl

v0.17.4

Bug Fixes

  • Fix CentOS Stream 9 Ansible provisioning failures through bastion hosts.

    Problem: CentOS Stream 9 packer builds were failing during Ansible provisioning with SCP/SFTP transfer errors when building through bastion/jump hosts:

    TASK [Enable pki-core] *************************************
fatal: [default]: FAILED! => {}
MSG:
failed to transfer file to /home/cloud-user/.ansible/tmp/.../
AnsiballZ_command.py:

    This was a platform-specific issue - Ubuntu 24.04 builds worked fine with the same configuration, but CentOS Stream 9 consistently failed at Ansible file transfer operations.

    Root Cause: CentOS Stream 9 cloud images have different SSH/SCP/SFTP configurations compared to Ubuntu. Even with ANSIBLE_SCP_IF_SSH=True and --scp-extra-args '-O' flags, Ansible file transfers failed when trying to copy Python module wrappers to the remote system.

    Resolution: Enabled Ansible pipelining (ANSIBLE_PIPELINING=True) when local_build=true (bastion/jump host builds). Pipelining completely bypasses the problematic SCP/SFTP file transfer mechanism by sending Python code directly over SSH stdin, eliminating the file transfer step entirely.

    Benefits:

    • ✅ Fixes CentOS Stream 9 Ansible provisioning failures

    • ✅ Reduces number of SSH connections (performance improvement)

    • ✅ Avoids platform-specific SCP/SFTP incompatibilities

    • ✅ Works across all Linux distributions (Ubuntu, CentOS, RHEL)

    • ✅ Backward compatible with Jenkins builds (local_build=false unchanged)

    Technical Details:

    Pipelining works by:

    1. Ansible generates Python module code

    2. Sends it directly over SSH stdin (no temp files)

    3. Remote Python interpreter executes it from stdin

    4. Results returned over SSH stdout

    This eliminates the need for:

    • Creating temporary files on the remote system

    • Transferring files via SCP/SFTP

    • Cleaning up temporary files

    Compatibility: Pipelining requires that requiretty is disabled in /etc/sudoers on the target system. Modern cloud images (Ubuntu 24.04, CentOS Stream 9) have this disabled by default, so no additional configuration is needed.

    Impact:

    • local_build=true (bastion builds): Pipelining enabled

    • local_build=false (Jenkins builds): No change, pipelining disabled

    Templates updated:

    • templates/builder.pkr.hcl

    • templates/docker.pkr.hcl

    • templates/devstack.pkr.hcl

    • templates/devstack-pre-pip-yoga.pkr.hcl

    • templates/windows-builder.pkr.hcl

    Reference: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#ansible-pipelining

v0.17.3

Bug Fixes

  • Fix Ansible file transfer failures when building through bastion/jump hosts.

    Problem: Packer builds with local_build=true (bastion/jump host mode) were failing during Ansible provisioning with SFTP errors:

    failed to transfer file to /home/ubuntu/.ansible/tmp/...
bash: line 1: /usr/lib/sftp-server: No such file or directory
scp: Connection closed

    Root Cause: Ansible defaults to SFTP for file transfers, not SCP. The existing --scp-extra-args '-O' flag was being ignored because Ansible was using SFTP, not SCP. Many cloud instances and bastion hosts don’t have sftp-server installed, causing transfer failures.

    Resolution: Added ANSIBLE_SCP_IF_SSH=True environment variable when local_build=true to force Ansible to use SCP mode instead of SFTP. This makes the --scp-extra-args '-O' flag take effect, which forces the legacy SCP protocol that works through bastion hosts.

    Implementation: Created conditional ansible_env_vars in the locals block of all templates:

    • When local_build=true: Sets ANSIBLE_SCP_IF_SSH=True

    • When local_build=false: Standard Ansible environment (no change)

    Backward Compatibility: This change only affects builds with local_build=true (packer-build-action through bastion). Existing Jenkins builds continue to work unchanged since local_build defaults to false.

    Templates updated:

    • templates/builder.pkr.hcl

    • templates/docker.pkr.hcl

    • templates/devstack.pkr.hcl

    • templates/devstack-pre-pip-yoga.pkr.hcl

    • templates/windows-builder.pkr.hcl

    Usage: When building through a bastion/jump host, set:

    packer build -var local_build=true ...

    Reference: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html

v0.17.2

Bug Fixes

  • Fix Ansible provisioner user parameter in OpenStack templates.

    The Ansible provisioner in OpenStack templates (builder.pkr.hcl, docker.pkr.hcl, devstack.pkr.hcl, devstack-pre-pip-yoga.pkr.hcl, and windows-builder.pkr.hcl) was missing the user parameter, causing Ansible to default to the wrong user account when connecting to instances through SSH.

    This resulted in SCP failures during the “Gathering Facts” task with errors like:

    fatal: [default]: FAILED! => {}
MSG:
failed to transfer file to /home/runner/.ansible/tmp/...
scp: dest open '~runner/.ansible/tmp/...': No such file or directory

    Root Cause: During the HCL2 conversion (commit 6e2fdc9), AWS templates were created with user = "${var.ssh_user}" but OpenStack templates were not. This was an oversight that went undetected until Ansible provisioning was tested through bastion/jump hosts in CI/CD environments.

    Resolution: Added user = "${var.ssh_user}" parameter to the Ansible provisioner block in all affected OpenStack templates. This ensures Ansible uses the correct SSH user (ubuntu/centos/cloud-user) as defined in the respective vars files.

    Backward Compatibility: This change is fully backward compatible:

    • The ssh_user variable already exists in all vars files

    • AWS templates already use this pattern successfully

    • No changes required to existing Jenkins jobs or workflows

    • The fix aligns OpenStack templates with the existing AWS template pattern

    Templates updated:

    • templates/builder.pkr.hcl

    • templates/docker.pkr.hcl

    • templates/devstack.pkr.hcl

    • templates/devstack-pre-pip-yoga.pkr.hcl

    • templates/windows-builder.pkr.hcl

v0.17.1

Upgrade Notes

  • No action required for existing deployments. All changes are backward compatible with Packer 1.9.x and 1.10.x. If upgrading to or using Packer 1.11.x, ensure packer init is run before packer build or packer validate to download the required plugins (this is already handled in the CI/CD workflow).

Bug Fixes

  • Fixed compatibility with Packer 1.11.x by adding explicit plugin declarations to all template files. Packer 1.11.x introduced stricter plugin validation requiring all plugins to be declared in required_plugins blocks.

    Updated templates with required_plugins blocks:

    • builder.pkr.hcl (openstack, docker, ansible)

    • docker.pkr.hcl (openstack, docker, ansible)

    • devstack.pkr.hcl (openstack, docker, ansible)

    • devstack-pre-pip-yoga.pkr.hcl (openstack, docker, ansible)

    • windows-builder.pkr.hcl (openstack, ansible)

    • builder-aws.pkr.hcl (amazon, ansible)

    • docker-aws.pkr.hcl (amazon, ansible)

    This fix maintains backward compatibility with Packer 1.9.x and 1.10.x while ensuring builds work correctly with Packer 1.11.x and future versions. The GitHub Actions workflow already includes the necessary packer init command to download plugins.

    Added .packer.d/ directory to .gitignore to exclude plugin cache.

  • Fixed ansible-galaxy.sh script failing on Ubuntu 24.04 with error python3.10: command not found. The script now uses the system’s default Python 3 instead of requiring a specific version.

v0.17.0

New Features

  • Add SSH bastion/jump host support for OpenStack builder template. The following new variables are now available for configuring SSH bastion connections:

    • ssh_bastion_host - IP address or hostname of the bastion host

    • ssh_bastion_username - Username for bastion authentication

    • ssh_bastion_port - SSH port on bastion (default: 22)

    • ssh_bastion_agent_auth - Use SSH agent for authentication (default: true)

    • ssh_bastion_private_key_file - Path to SSH private key file

    • ssh_bastion_password - Password for bastion authentication (not recommended)

    All bastion variables are optional with empty string defaults, making them backward compatible with existing builds that don’t require bastion access.

    Example usage:

    packer build \\
  -var=ssh_bastion_host=100.64.183.39 \\
  -var=ssh_bastion_username=root \\
  -var-file=vars/ubuntu-22.04.pkrvars.hcl \\
  templates/builder.pkr.hcl

    This enables Packer builds to access OpenStack instances through a bastion/jump host, which is required when direct access to OpenStack networks is not available (e.g., in CI/CD environments using Tailscale or other ephemeral bastion solutions).

    Reference: https://developer.hashicorp.com/packer/integrations/hashicorp/openstack/latest/components/builder/openstack

  • Interactive JJB Build Script: Added comprehensive build-packer-images.sh script that integrates with Jenkins Job Builder (JJB) configurations to validate and execute only approved platform+template combinations.

    Key features include: - Parses 21 validated combinations from jjb/releng-packer-jobs.yaml - Automatically excludes End-of-Life platforms (ubuntu-18.04) - Individual timestamped log files for each build in /tmp/ - Three execution modes: interactive, dry-run, and background - Real-time build progress tracking with status reporting - Smart file discovery and validation

  • Conditional SSH Compatibility System: Implemented dynamic SSH argument handling using HCL conditional expressions to support both local development and CI/CD environments without breaking existing workflows.

    Templates updated with local_build variable: - templates/builder.pkr.hcl - templates/builder-aws.pkr.hcl - templates/docker.pkr.hcl - templates/docker-aws.pkr.hcl - templates/devstack.pkr.hcl - templates/devstack-pre-pip-yoga.pkr.hcl - templates/windows-builder.pkr.hcl

  • Comprehensive Documentation: Added complete README.md with usage instructions, prerequisites, troubleshooting guide, and examples for all supported build combinations and execution modes.

Known Issues

  • SSH Compatibility Problems: Previous Packer template configurations used hardcoded SSH arguments that were incompatible with newer SSH versions and local development environments, causing SCP file transfer failures during Ansible provisioning steps.

  • Build Combination Management: No systematic way to ensure builds only used Jenkins Job Builder validated platform+template combinations, leading to potential builds of unsupported or End-of-Life platform images.

  • Build Process Visibility: Limited visibility into build progress and no centralized logging mechanism for troubleshooting failed builds across multiple platform+template combinations.

Upgrade Notes

  • Existing builds using ssh_proxy_host will continue to work without changes. The legacy proxy support is maintained for backward compatibility. New deployments should use the SSH bastion variables for native jump host support instead of proxy-based SSH tunneling.

  • Migration to JJB-Validated Builds: The new build script automatically restricts builds to only JJB-approved combinations (21 total), excluding EOL platforms. This ensures consistency with CI/CD infrastructure.

    Backward Compatibility: All existing manual packer build commands continue to work unchanged. The local_build variable defaults to false, maintaining existing SSH behavior for automated builds.

    Enhanced Local Development: Set local_build=true when building locally to enable SSH compatibility options for modern SSH versions.

Deprecation Notes

  • Manual Build Combination Discovery: While still functional, manual discovery of valid platform+template combinations is deprecated in favor of the JJB-validated approach provided by build-packer-images.sh script.

Security Issues

  • SSH Algorithm Updates: Enhanced SSH compatibility includes support for modern SSH key algorithms while maintaining backward compatibility with older SSH implementations used in CI/CD environments.

Bug Fixes

  • Fixed netselect package download failure in packer build by updating to available version (0.3.ds1-30.1) and using reliable mirror (deb.debian.org) instead of failing ftp.au.debian.org URL.

  • SCP Upload Failures: Resolved critical SSH/SCP compatibility issues that were causing Packer builds to fail with errors.

    The fix implements conditional SSH arguments: - For local builds: –scp-extra-args “’-O’” with enhanced SSH algorithms - For CI builds: Standard SSH arguments (backward compatible)

v0.16.5

Bug Fixes

  • Remove broken link to non-existent Docker template

v0.16.0

Upgrade Notes

  • Upgrade Ansible version to v2.15.9.

v0.15.2

Deprecation Notes

  • Deprecate JSON format var and templates

    With the release of common-paker v0.14.0 (commit 6e2fdc9fb5be1d9843d14b6a) the packer version v1.9.x supports only HCL format templates and var files. These older JSON format files are no longer used with the packer verify and merge jobs.

    All packer templates have been migrated from JSON to HCL2 format with v0.14.X release.

    JSON format templates are deprecated and no longer used with packer version > 1.9.x. Therefore remove the deprecated files from the repository.

v0.14.1

Known Issues

  • RSA/SHA-1 was deprecated in the latest OpenSSH release 8.8 causing builds fail with the following error:

    Data could not be sent to remote host “127.0.0.1”. Make sure this host can be reached over ssh: command-line: line 0: Bad configuration option: pubkeyacceptedalgorithms

Bug Fixes

  • Add workaroud and pass required HostKeyAlgorithms through ssh extra argumements.

v0.14.0

Prelude

As of packer version 1.7.0 HCL2 is the preferred way to write Packer templates. HCL2 preserves existing workflows while leveraging HCL2’s advanced features like variable interpolation and configuration composability.

New Features

  • Modify devstack templates for Ubuntu/Debian distributions.

Known Issues

Upgrade Notes

  • Migrate packer templates from JSON to HCL2 format. JSON format templates are deprecated and no longer works with packer version > 1.9.x.

    Existing JSON templates can be converted to ‘.pkr.hcl’ using:

    packer hcl2_upgrade -with-anotations <folder|filename>

    Packer version 1.9.1 will be minimum required version for packer jobs. This version requires installing the cloud specific plugin through packer config and needs to be initalize and download before running packer build.

    <temmplate>.pkr.hcl includes the sources and builds are defined. .auto.pkrvars.hcl includes variables that are loaded automatically. These variables load automatically from the same directory and are common across templates. variables.pkr.hcl includes variable declarations that are common across templates.

    Reference: https://developer.hashicorp.com/packer/guides/hcl/variables https://developer.hashicorp.com/packer/docs/templates/hcl_templates https://github.com/hashicorp/packer-plugin-openstack/blob/main/README.md

Deprecation Notes

  • Support for ‘.json’ templates will be removed from common-packer in subsequent release to give enough time for projects consuming to upgrade. All projects specific templates not available in this repository are required to convert existing ‘.json’ to ‘.pkr.hcl’ format.

v0.13.0

New Features

  • Add Openstack devstack templates and provisioner to common-packer.

  • Add GHA packer validation job to common-packer.

v0.12.1

Known Issues

  • Error running “./common-packer/ansible-playbook.sh –version”: exit status 1

Bug Fixes

  • Skip ansible provisioners version check. Packer provisioners invokes the ansible version check, which can be skipped since the provisioner is invoking a custom script.

v0.12.0

Prelude

Install ansible and ansible-playbook using PyPI.

Known Issues

  • Error:

    Problem: package ansible-5.4.0-3.el8.noarch requires (ansible-core >= 2.12.2 with ansible-core < 2.13), but none of the providers can be installed

Bug Fixes

  • Install ansible through system packages causes dependencies conflicts on CentOS 8 platform therefore update packer templates to use ansible provisioner created through venv.

v0.11.0

Upgrade Notes

  • Upgrade git v2.36 on CentOS 7.

v0.10.2

New Features

  • Add support for CentOS Stream 9

Bug Fixes

  • Import correct GPG keys for sigul and EL8

    Sigul 1.1.1 is signed by the Fedora infrastructure GPG key therefore import the Fedora infra key before installing the updated version on Sigul for CentOS8.

    Import the EL8 GPG keys without which would fail while installing several dependencies.

v0.10.0

New Features

  • Add support for CentOS Streams 8

v0.9.2

Bug Fixes

  • Add community.general as part of the required collections. Required for tasks defined in local-docker specific to Magma to enable and manage virtualization tools.

v0.9.1

Bug Fixes

v0.9.0

New Features

  • Feat: Add support for Docker builder on arm64

    The original change e24c07369afd514abdf3efb0f596f772261412ed missed updating arm64 var files, while the builder templates were updated. This breaks the packer verify jobs.

    Error:

    required variable not set: docker_source_image’

  • Add support for Docker builder on Windows

    The original change e24c07369afd514abdf3efb0f596f772261412ed missed updating Windows var files, while the builder templates were updated. This breaks the packer verify jobs.

    Error:

    required variable not set: docker_source_image’

Bug Fixes

  • All vars files have been updated to properly include the AMI product code filter. This is needed to properly pass global-jjb verification.

v0.8.0

New Features

  • Docker image builds are now supported by the packer templates.

Upgrade Notes

  • Requires first upgrading global-jjb to version v0.57.0 to pull in support for selecting a packer-builder in the packer-merge jobs, otherwise existing project packer-merge job builds may fail without the global-jjb updates if builds are run on a Jenkins node that does not support Docker.

  • Projects using AWS must ensure that the packer-merge jobs are updated to set packer-builder to aws.

    Example:

    - project:
    name: packer-builder-jobs
    jobs:
      - gerrit-packer-merge

    project: releng/builder
    project-name: builder
    branch: master
    archive-artifacts: "**/*.log"

    build-node: centos7-builder-2c-1g
    build-timeout: 90
    cron: "00 H 1 * *"

    platforms:
      - centos-7
      - centos-8

    packer-builder: aws
    templates: builder
    update-cloud-image: true

v0.7.6

Upgrade Notes

  • Upgrade lf-standard-* flavors to v3

    v3 flavors guarantees jobs spin on newer hardware that are faster and cost-efficient than the v2 flavors.

v0.7.1

Known Issues

  • Remove availability zone from Openstack templates. This was causing errors, and is unnecessary (there is only one AZ for these images).

v0.6.1

Bug Fixes

  • Updated Ubuntu 18.04 image as the previous base image was uploaded in qcow2 format. This caused timeouts in packer builds. New image is in raw format.

    Replaces: “LF - Ubuntu 18.04 LTS (2020-07-28)”

    With: “LF - Ubuntu 18.04 LTS (2019-12-11)”

v0.6.0

New Features

  • CentOS 7 builds will now include the Sigul client by default. This package is now in use by most projects, and is being downloaded whenever it is needed. This has particularly been a problem due to frequent connection issues with the kojipkgs servers that host the sigul package.

Upgrade Notes

v0.5.2

Bug Fixes

v0.5.0

Upgrade Notes

  • Add CentOS 8.x base image.

v0.4.2

Bug Fixes

  • Use netselect to choose a package mirror to install python-minimal in a reliable manner.

    apt{-get} does not refresh the package mirrors (for packer builds run within Jenkins), therefore fails with “E: Unable to locate package python-minimal” while installing python-minimal.

v0.3.1

Bug Fixes

  • The EC2 (aws) template had an extra configuration option that was added but had not been properly tested. This option is removed allowing aws based templates to properly validate and build.

  • Pygments release 2.4.0 which added a python requires that excludes all versions of Python < 3.5. The LFCI default 3 version is 3.4 so causes build failure.

  • Request-2.22.0 does not work with python-3.4.9, so pin requests to v2.21.0 to address the tox failures.

  • Ansible supports lists passed to package install that can avoid using with_items. Using with_items makes multiple calls to the packages manager slowing down the performance.

v0.3.0

New Features

  • More control over Openstack images is now allowed. This includes selection of cloud, availability zone, if volumes should be created and at what size and format.

Upgrade Notes

  • Packer 1.3.2 is now required to support Openstack block storage and disk format requirements

  • The CentOS 7.6 1811 base image is uploaded on the cloud provider. Switch the packer var files to build from the latest CentOS 7.6 1811 base image.

v0.1.0

Upgrade Notes

  • Requires Global JJB v0.26.0 minimum.