New Infra Bootstrap

This document uses example.org as the domain for all examples. Please change to point to the intended systems for your project.

Jenkins

Steps

  1. Login to Jenkins at https://jenkins.example.org

  2. Navigate to https://jenkins.example.org/pluginManager/

  3. Update all plugins

  4. Install required plugins as documented in global-jjb install guide

  5. Install the following plugins:

  6. Navigate to https://jenkins.example.org/configure

  7. Configure Jenkins as follows:

    # of executors: 0
Jenkins URL: https://jenkins.example.org
System Admin e-mail address: Jenkins <jenkins-dontreply@example.org>
Global Config user.name value: jenkins
Global Config user.email value: jenkins@example.org

    If using the Message Injector plugin set Message to inject to Logs: https://logs.example.org/SILO/HOSTNAME/$JOB_NAME/$BUILD_NUMBER and replace SILO and HOSTNAME as appropriate.

  8. Click Save

  9. Configure Jenkins security as described in Jenkins Security

  10. Navigate to https://jenkins.example.org/configureSecurity/

  11. Configure the following permissions for Anonymous Users

    • Overall:Read

    • Job:ExtendedRead

    • Job:Read

    • View:Read

    Note

    If the project is not yet public, hold off on these permissions or adjust as necessary for the project’s case.

  12. Setup Jenkins global environment variables as described in the global-jjb install guide

    Note

    Skip the ci-management step in as we will be discussing that in the next section.

  13. Setup a jobbuilder account

  14. Setup global-jjb required Jenkins Files

Setup Job Builder account

The ci-jobs in global-jjb require a jobbuilder account which has permissions to login to Jenkins.

  1. Navigate to and create an account for jobbuilder https://identity.linuxfoundation.org/

    Note

    This step mainly applies to LF projects. Use the relevant identity system as it applies to your local configuration.

  2. Navigate to https://jenkins.example.org/configureSecurity and configure permissions for the jobbuilder account as follows:

    • Overall: Administer

    • Job: Configure

    • Job: Create

    • Job: Delete

    • Job: Discover

    • Job: Read

    • View: Configure

    • View: Create

    • View: Delete

    • View: Read

Setup Sandbox Access

To allow people access to the Jenkins Sandbox, we require an LDAP group to exist with the appropriate people added. Use lftools lfidapi create-group to create a group called $project-jenkins-sandbox-access and add any initial members you might need.

  1. Go to https://jenkins.example.org/configureSecurity and add the group with:

    • Overall: Read

    • Job: Build

    • Job: Cancel

    • Job: Configure

    • Job: Create

    • Job: Delete

    • Job: Discover

    • Job: Read

    • Job: Workspace

    • View: Read

ci-management repo

Once Jenkins is available we can initialize a new ci-management repo.

Setup administrative files

  1. Create ci-management repo in the project SCM system

  2. Create a README.md file explaining the purpose of the repo

    # ci-management

This repo contains configuration files for Jenkins jobs for the EXAMPLE
project.

  3. Setup tox/coala linting for jjb/ and packer/ directories

    .yamllint.conf

    extends: default

rules:
  empty-lines:
    max-end: 1
  line-length:
    max: 120

    .coafile

    [Documentation]
bears = WriteGoodLintBear
files = *.md
allow_so_beginning = False
allow_there_is = False
allow_cliche_phrases = False

[GitCommit]
bears = GitCommitBear
ignore_length_regex = Signed-off-by,
    Also-by,
    Co-authored-by,
    http://,
    https://

[JSON]
bears = JSONFormatBear
files = packer/**.json
indent_size = 2

[ShellCheck]
bears = ShellCheckBear,
    SpaceConsistencyBear
files = jjb/**.sh,
    packer/**.sh
shell = bash
indent_size = 4
use_spaces = yeah

[YAML]
bears = YAMLLintBear
files = jjb/**/*.yaml
document_start = True
yamllint_config = .yamllint.conf

    tox.ini

    [tox]
minversion = 1.6
envlist = coala
skipsdist = true

[testenv:coala]
basepython = python3
deps =
    coala==0.11
    coala-bears==0.11
    nodeenv~=1.3.0
commands =
    nodeenv -p
    npm install --global write-good
    python3 -m nltk.downloader punkt maxent_treebank_pos_tagger averaged_perceptron_tagger
    coala --non-interactive

  4. Setup .gitignore

    .tox/
archives/
jenkins.ini

# Packer
.galaxy/
*.retry
cloud-env.json

  5. git commit -asm "Setup repo administrative files"

  6. git push files to the repository

  7. Run tox

    Note

    The jjb tox env will fail as the required jjb/ directory does not yet exist. This is fine and proves that tox is working before we continue in the next step.

Bootstrap common-packer and initial builder

Note

This section assumes the usage of an OpenStack cloud provider for Jenkins build nodes. Adjust as necessary if not using an OpenStack cloud.

  1. Navigate to the GIT_ROOT of the ci-management repo

  2. Install common-packer to GIT_ROOT/packer/common-packer

    git submodule add https://github.com/lfit/releng-common-packer.git packer/common-packer

  3. Follow common-packer doc to setup a template

  4. git commit -asm "Setup common-packer and initial builder"

  5. git push files to repository

  6. Upload a CentOS 7 cloudimg to use as a base for packer builds

    When uploading the cloudimg ensure it’s name matches the base_image name in common-packer/vars/centos-7.json.

  7. Run packer build -var-file=cloud-env.json -var-file=common-packer/vars/centos-7.json templates/builder.json

  8. Note down the image name from the packer build as we will need it later

  9. Navigate to https://jenkins.example.org/credentials/store/system/domain/_/newCredentials

  10. Configure the OpenStack cloud credential as follows:

    Kind: OpenStack auth v3
Project Domain: Default
Project Name: OPENSTACK_TENANT_ID
User Domain: Default
User Name: OPENSTACK_USERNAME
Password: OPENSTACK_PASSWORD
ID: os-cloud
Description: openstack-cloud-credential

    Note

    Replace ALL_CAPS instances with your Cattle account credential.

  11. Configure an ssh keypair for the Jenkins <-> OpenStack connection

    1. Generate a new SSH Keypair

      ssh-keygen -t rsa -C jenkins-ssh -f /tmp/jenkins

    2. Navigate to https://jenkins.example.org/credentials/store/system/domain/_/newCredentials

    3. Configure the Jenkins SSH Key as follows:

      Kind: SSH Username and private key
Scope: Global
Username: jenkins
Private Key: Enter directly
Passphrase:
ID: jenkins-ssh
Description: jenkins-ssh

      Copy the contents of /tmp/jenkins into the Key field.

    4. Navigate to https://openstack-cloud.example.org/project/key_pairs

    5. Import the contents of /tmp/jenkins.pub into the OpenStack cloud provider account with the keypair name jenkins-ssh

  12. Navigate to https://jenkins.example.org/configfiles/selectProvider

  13. Create a OpenStack User Data file with the following specs:

    Type: OpenStack User Data
ID: jenkins-init-script
Name: jenkins-init-script
Comment: jenkins-init-script

    With the contents (change the git clone URL as necessary for the project):

    #!/bin/bash
until host gerrit.example.org &>/dev/null
do
    echo "Waiting until gerrit.example.org is resolvable..."
done

git clone --recurse-submodules https://gerrit.example.org/r/ci-management /opt/ciman
/opt/ciman/jjb/global-jjb/jenkins-init-scripts/init.sh

    For Windows:

    Type: OpenStack User Data
ID: jenkins-init-script-windows
Name: jenkins-init-script-windows
Comment: jenkins-init-script-windows

    With the contents (change the git clone URL as necessary for the project):

    <powershell>
  # Resize first partition of first disk to maximum size
  Get-Partition -DiskNumber 0 -PartitionNumber 1
  $size = (Get-PartitionSupportedSize -DiskNumber 0 -PartitionNumber 1)
  Resize-Partition -DiskNumber 0 -PartitionNumber 1 -Size $size.SizeMax

  mkdir -Force "${SLAVE_JENKINS_HOME}"
  (new-object System.Net.WebClient).DownloadFile('${SLAVE_JAR_URL}','${SLAVE_JENKINS_HOME}\slave.jar')
  cd "${SLAVE_JENKINS_HOME}"
  java ${SLAVE_JVM_OPTIONS} -jar "slave.jar" -jnlpUrl "${SLAVE_JNLP_URL}" -secret "${SLAVE_JNLP_SECRET}"
</powershell>

  14. Configure cattle cloud

    1. Create cloud config directory mkdir -p jenkins-config/clouds/openstack/cattle

    2. Configure the OpenStack cloud connection details in jenkins-config/clouds/openstack/cattle/cloud.cfg

      Replace <BUILD_IMAGE_NAME> and <NETWORK_ID> in the below file with the details for your cloud. Find <NETWORK_ID> at https://dashboard.vexxhost.net/project/networks/

      jenkins-config/clouds/openstack/cattle/cloud.cfg
      # Cloud Configuration
CLOUD_CREDENTIAL_ID=os-cloud
CLOUD_URL=https://auth.vexxhost.net/v3/
CLOUD_IGNORE_SSL=false
CLOUD_ZONE=ca-ymq-1

# Default Template Configuration
IMAGE_NAME=<BUILD_IMAGE_NAME>
HARDWARE_ID=v3-standard-2
NETWORK_ID=<NETWORK_ID>
USER_DATA_ID=jenkins-init-script
INSTANCE_CAP=10
SANDBOX_CAP=4
FLOATING_IP_POOL=
SECURITY_GROUPS=default
STARTUP_TIMEOUT=600000
KEY_PAIR_NAME=jenkins-ssh
NUM_EXECUTORS=1
JVM_OPTIONS=
FS_ROOT=/w
RETENTION_TIME=0

    3. Create jenkins-config/clouds/openstack/cattle/centos7-builder-2c-1g.cfg

      IMAGE_NAME=ZZCI - CentOS 7 - builder - 20180604-1653
HARDWARE_ID=v3-standard-2

    4. Run global-jjb jenkins-cfg script to update Jenkins cloud config

      Note

      This step requires crudini tool, install from your package manager to avoid python 2 vs 3 problems in your virtualenv.

      Note

      This step requires having lftools available on your path and a ~/.config/jenkins_jobs/jenkins_jobs.ini configured with Jenkins credentials.

      Set jenkins_silos to match the config section name in the jenkins_jobs.ini file.

      Run the following commands:

      export WORKSPACE=$(pwd)
export jenkins_silos=production
bash ./jjb/global-jjb/shell/jenkins-configure-clouds.sh
# OPTIONAL: view the created script
cat archives/groovy-inserts/production-cloud-cfg.groovy

      Then navigate to https://jenkins.example.org/script and copy the contents of archives/groovy-inserts/production-cloud-cfg.groovy into the script console. This will initialize the OpenStack cloud configuration.

    5. Commit the jenkins-config directory

      git add jenkins-config/
git commit -sm "Add OpenStack cloud configuration"
git push

  15. Navigate to https://jenkins.example.org/configure and verify the cloud configuration.

Setup global-jjb and ci-jobs

  1. Install global-jjb to GIT_ROOT/jjb/global-jjb

    git submodule add https://github.com/lfit/releng-global-jjb.git jjb/global-jjb

  2. Setup jjb/defaults.yaml

    - defaults:
    name: global

    gerrit-server-name: Primary
    git-url: 'ssh://jenkins-$SILO@gerrit.example.org:29418'
    jenkins-ssh-credential: jenkins-ssh
    lftools-version: '<1.0.0'

  3. Create the CI Jobs in jjb/ci-management/ci-jobs.yaml

    - project:
    name: ci-jobs

    jobs:
      - '{project-name}-ci-jobs'

    project: ci-management
    project-name: ci-management
    build-node: centos7-builder-2c-1g

  4. Manually push the initial ci-management jobs to Jenkins

    jenkins-jobs update jjb/

  5. Git commit the current files and push to Gerrit

    git commit -sm "Setup global-jjb and ci-jobs"
git push

  6. Confirm verify jobs work

  7. Merge the patch and confirm merge job works

Setup packer jobs

  1. Create Initial CI Packer job in jjb/ci-management/ci-packer.yaml

    - project:
    name: packer-verify
    jobs:
      - gerrit-packer-verify

    project: ci-management
    project-name: ci-management
    build-node: centos7-builder-2c-1g

- project:
    name: packer-builder-jobs
    jobs:
      - gerrit-packer-merge

    project: ci-management
    project-name: ci-management
    build-node: centos7-builder-2c-1g

    templates: builder
    platforms:
      - centos-7
      - ubuntu-16.04

  2. Git commit and push the patch to ci-management for review

    git commit -sm "Add packer builder job"
git push ...

  3. Confirm packer verify job passes

  4. Merge patch and confirm merge job works

Nexus 2

Setup Server Config

  1. Navigate to https://nexus.example.org/#nexus-config

  2. SMTP Settings

    Hostname: localhost
Port: 25
Username:
Password:
Connection: Use plain SMTP
System Email: noreply@example.org

  3. Application Server Settings

    Base URL: https://nexus.example.org/
Force base URL: true
UI Timeout: 120

  4. PGP Key Server Information

    Server 1: http://pool.sks-keyservers.net:11371
Server 2: http://pgp.mit.edu:11371

Setup LDAP

  1. Navigate to https://nexus.example.org/#enterprise-ldap

  2. Click Add at the top menu bar

  3. Configure the LDAP connection as follows:

    Name: ldaps://ldap.example.org:636
Protocol: ldaps
Hostname: ldap.example.org
Port: 636
Search Base: dc=example,dc=org

Authentication: Anonymous Authentication

  4. Click on the User & Group Settings tab

  5. Configure the User & Group Settings as follows:

    Base DN: ou=Users
Object Class: inetOrgPerson
User ID Attribute: uid
Real Name Attribute: cn
E-Mail Attribute: mail

Group Type: Static Groups
Base DN: ou=groups
Object Class: groupOfNames
Group ID Attribute: cn
Group Member Attribute: member
Group Member Format: ${dn}

Setup Admin role

  1. Navigate to https://nexus.example.org/#security-roles

  2. Click Add > External Role Mapping

  3. Configure mapping as follows:

    Realm: LDAP
Role: lf-collab-admins

    Note

    If not an LF project replace lf-collab-admins with the relevant admin group for your case.

  4. Click Add and add the Nexus Administrator Role

From this point you should be able to login using your own account to administrate the server. Do that and then setup admin user email and deactivate the default deployment account as we will create separate project deployment accounts for each individual project.

  1. Navigate to https://nexus.example.org/#security-users

  2. Configure the admin user email to collab-it+PROJECT@linuxfoundation.org

    Note

    Replace email as necessary for your org.

  3. Set the default deployment user account Status to Disabled

Setup custom deployment role

LF projects use Nexus 2 as a server to host logs and requires the Nexus Unpack plugin configured. Since the default Nexus Deployment Role is not configurable, we will have to create our own custom one to ensure Unpack is available.

  1. Navigate to https://nexus.example.org/#security-roles

  2. Click Add > Nexus Role

  3. Configure the following settings:

    Role Id: lf-deployment
Name: LF Deployment Role
Description: LF modified deployment role

  4. Click Add and add the following roles:

    • Artifact Upload

    • Nexus Deployment Role

    • Unpack

Setup routing

  1. Navigate to https://nexus.example.org/#routes-config

  2. Clear all existing routes

  3. Click Add to add a new route

  4. Configure the route as follows:

    URL Pattern: ^/org/example/.*
Rule Type: Inclusive
Repository Group: All Repository Groups

Ordered Route Repositories:

  * Releases
  * Snapshots

Nexus 3

Setup Server Config

  1. Navigate to https://nexus3.example.org/#admin/system/emailserver

  2. SMTP Settings

    Enabled: true
Hostname: localhost
Port: 25
Username:
Password:
From address: noreply@example.org
Subject prefix:

Setup LDAP

  1. Navigate to https://nexus3.example.org/#admin/security/ldap

  2. Click Create connection

  3. Configure the LDAP connection as follows

    Name: ldaps://ldap.example.org:636
Protocol: ldaps
Hostname: ldap.example.org
Port: 636
Search base: dc=example,dc=org
Authentication method: Anonymous Authentication

  4. Click Verify connection and check that it works

  5. Click Next

  6. Configure the User & Group Settings as follows:

    Base DN: ou=Users
Object Class: inetOrgPerson
User ID Attribute: uid
Real Name Attribute: cn
E-Mail Attribute: mail
Map LDAP groups as roles: true

Group Type: Static Groups
Base DN: ou=groups
Object Class: groupOfNames
Group ID Attribute: cn
Group Member Attribute: member
Group Member Format: ${dn}

  7. Click Verify user mapping and confirm it works

  8. Click Create

Setup Admin role

  1. Navigate to https://nexus3.example.org/#admin/security/roles

  2. Click Create role > External Role Mapping

  3. Configure mapping as follows:

    Mapped Role: lf-collab-admins
Role Name: lf-collab-admins
Role description: lf-collab-admins
Privileges: nx-all

From this point you should be able to login using your own account to administrate the server. Do that and then setup admin user email and deactivate the default deployment account as we will create separate project deployment accounts for each individual project.

  1. Navigate to https://nexus3.example.org/#admin/security/users:admin

  2. Configure the admin user email to collab-it+PROJECT@linuxfoundation.org

    Note

    Replace email as necessary for your org.

Post bootstrap

With infrastructure bootstrapped, here is a list of tasks to consider that may be useful to setup.

GitHub

Nexus