From Fedora Project Wiki

Zuul Based CI

Goals

  • Bring CI infrastructure based on Zuul for projects hosted on pagure.io and src.fedoraproject.org.
    • Gerrit and Github are supported as well and we could for specific cases provide CI for projects hosted there.
  • Propose jobs and workflow of jobs around Pull Requests for Fedora packages (distgits on src.fedoraproject.org).

For Github projects, Packit should be also considered.

News

June 06, 2023

  • Added support for epel9-next branch.

May 22, 2023

  • Removed jobs for branch F36.

Oct 27, 2022

  • Removed rpm-test job from the tests template. This job was a duplicate of rpm-sti-test.

Sep 5, 2022

  • Added support for Fedora 37 branch
  • Removed support for Fedora 34 branch

Apr 8, 2022

  • Added "Contributing" section in this documentation.

Feb 11, 2022

  • Added support for Fedora 36 branch
  • Removed support for Fedora 33 branch

July 21, 2021

  • The tmt tests are now run via the new rpm-tmt-test check, if available in the dist-git repository.
  • Support for F32 has been removed as EOL.

February 18, 2021

February 10, 2021

  • Added support for F34 branch.

February 3, 2021

  • The Fedora community moved forward with the master branch removal. From our side it required some small changes to ensure PRs on main and rawhide branches are handled by Zuul. As we recently started to manage the Zuul YAML configuration files through dhall-lang, it was handy to simply modify the FZCI.dhall package. The generated updates on the YAML are here and here.

January 6, 2021

  • New jobs are available to run scratch build on Koji for each of the Fedora supported architectures. Those new jobs are attached along with the default distgit CI workflow. See this section for more information

December 8, 2020

November 30, 2020

November 13, 2020

November 02, 2020

  • In fedora-project-config, filenames to attach a repository to Zuul has changed. Now called fedora-distgits.yaml and fedora-sources.yaml.

September 09, 2020

  • Webhook configuration in project settings for distgit repositories is no longer needed. Indeed, in order to ease the consumption of the Zuul CI, especially for the Fedora distgit, we have setup, in the Software Factory Infrastructure, a service that listen to the fedora-messaging bus and forward messages to zuul-web. https://pagure.io/fm-gateway
  • This wiki page has been updated to reflect changes to activate Zuul CI for a distgit. Process is as simple as opening a Pull-Request: Add_the_repository_into_the_Zuul_configuration

July 31, 2020

  • A new job named rpm-install-test is run in case the distgit does not provide a tests/tests.yml. More details below.

June 22, 2020

January 24, 2020

January 14, 2020

  • Thanks to an update of the Zuul's Pagure driver on softwarefactory-project.io, the zuul user no longer needs to be in the collaborator **admin** group but only in the **commit** group. The project config helper tool has been updated.

November 13, 2019

October 22, 2019 - Service is ready to be beta tested

  • This wiki page contains the process to attach a repository from src.fedoraproject.org or pagure.io to Zuul.
  • Current known Issue: Sometime, src.fedoraproject.org, does not call Zuul event hooks (https://pagure.io/fedora-infrastructure/issue/8320). Then Zuul is unable to react and run jobs from Pagure events. This does not happen with pagure.io. This is a blocker for CI which needs to be reliable in order to gain confidence, I've pinged infra folks in that issue to help us debug it. UPDATE: "it was DNS" - resolved January 10, 2020

August 28, 2019 - Created Taiga EPIC

Flock 2019

What is Zuul/Nodepool

Zuul [1] is the CI and gating system from the Open Infrastructure Project [2]. It is able to scale and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [3].

Below is a list of features proposed by Zuul and its companion Nodepool:

  • Event-driven pipelines based on Code-Review or Pull-Request workflow: jobs can be triggered automatically when a PR is submitted, changed, approved, merged, or when the repository is tagged.
  • CI-as-code: jobs are defined as YAML + Ansible playbooks, pipeline definitions as YAML files. Zuul reads and loads those definitions directly from Git repositories.
  • Support for jobs inheritance, jobs dependencies, jobs chaining (with artifacts sharing).
  • Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
  • Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
  • Cross provider: a jobs' workspace can include unmerged patches from other projects even when hosted on different provider like Github and Pagure.
  • Parallel job run, only capped by resources available or predefined quotas
  • Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
  • Job resources support of OpenStack, OpenShift, K8S, Static nodes, AWS.
  • Well-defined, reproducible job environments to eliminate flakiness
  • Speculative testing before merging (gating): if several patches are about to land at the same time, they are tested on the repository's future state.

Until now, Zuul was only able to listen to Gerrit or Github events. Recently a new driver [4] allow Zuul to interface with Pagure as well. Pagure, Zuul and Nodepool could therefore combine into a very efficient CI/CD stack.

How to attach a Pagure repository on Zuul

Configure the repository for Zuul

If you want to activate Zuul check jobs for a distgit hosted on src.fedoraproject.org then no change is needed in project's settings so please jump to the section Add_the_repository_into_the_Zuul_configuration


In the project settings:

  • In Project Options
    • Web-hooks:
      • For a repository hosted on pagure.io add the following URL into the Web-hooks field: https://softwarefactory-project.io/zuul/api/connection/pagure.io/payload
    • In "Users & Groups" setting: add "zuul" as "ticket" collaborator (it allows Zuul to read the webhook token)
    • (For gating, optional)
      • Minimum score to merge pull-request: 0 or -1
      • Open metadata access to all: False
      • In "Users & Groups" setting: add "zuul" as "commit" collaborator
      • In "Tags" setting: add the "gateit" tag

Add the repository into the Zuul configuration

Open a Pull Request on https://pagure.io/fedora-project-config

  • For a repository hosted on pagure.io, edit resources/fedora-sources.yaml and add the repository such as:
resources:
  projects:
    Fedora-Sources:
      ...
      source-repositories:
        ...
        - repository-name
  • For a distgit repository hosted on src.fedoraproject.org, automatically generate the resources/fedora-distgits.yaml for a given FAS username or group. See section "How to add / maintain your packages list" in the fedora-project-config's README for details. Alternatively, manually edit resources/fedora-distgits.yaml and add the repository such as:
resources:
  projects:
    Fedora-Distgits:
      ...
      source-repositories:
        ...
        - repository-name:
            zuul/include: []

Once the Pull Request is accepted and merged, the repository should be available in the Zuul project list: https://fedora.softwarefactory-project.io/zuul/projects

For a repository hosted on pagure.io, the next sections are not relevant; skip ahead to the in-project configuration section.

Attach packaging jobs for a distgit repository on src.fedoraproject.org

Note that all repositories with a name that match the regexp '^rpms/.*' will get, by default, the jobs defined by the build, lint and test templates attached. Thus, if the default configuration is sufficient, then no need to edit the zuul.d/projects.yaml file as explained at the end of this section.

We have managed to provide some standard Pull Request's jobs and workflow. This idea is to reduce the amount of manual steps to propose packaging changes to Fedora by taking advantage of CI.

We propose a set of templates (a template can be seen as a workfow of jobs reacting to Pull Request events). Templates are available here https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml.

Templates configure jobs through three types of pipeline (not the same as Jenkins/Gitlab pipeline concept):

  • check: Jobs within that pipeline are triggered when a Pull Request is Opened or Updated
  • gate: Jobs within that pipeline are triggered when a Pull Request is approved (prior to be merged and closed)
  • promote: Jobs with that pipeline are triggered when a Pull Request is closed and merged

Conditions for a Pull Request to be approved:

  • All check pipelines's jobs succeeded
  • The Pull Request received the metadata tag: 'gateit'

Available jobs are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml):

  • rpm-scratch-build: Runs a scratch build on Koji (Koji target based on the PR's branch) and retrieves artifacts from the test node (rpms).
  • rpm-build: Runs a regular build on Koji (Koji target based on the PR's branch)
  • rpm-linter: Runs a rpmlint on artifacts (rpms) passed from the parent job
  • rpm-rpminspect: Runs rpminspect on artifacts (rpms) passed from the parent job. The job also finds the latest build on the related Koji tag and passes it to the rpminspect job so you get the rpminspect diff.
  • rpm-install-test: Install built packages (rpms passed from rpm-scratch-build) on the related Fedora node.
  • rpm-sti-test: Install built packages (rpms passed from rpm-scratch-build) and executes embedded STI tests on the related Fedora VM (rawhide VM for master branch, Fedora 34 VM for f34 branch, ...). The job is run only if STI tests are available in the repo, i.e. when files tests/tests.yaml* exists. Link to Testing Farm results can be found under Artifacts -> Testing Farm Artifacts in the Zuul web interface.
  • rpm-tmt-test: Install built packages (rpms passed from rpm-scratch-build) and executes embedded tmt tests on the related Fedora VM (rawhide VM for master branch, Fedora 34 VM for f34 branch, ...). The job is run only if tmt metadata is available in the repo, i.e. when file .fmf/version exists. Link to Testing Farm results can be found under Artifacts -> Testing Farm Artifacts in the Zuul web interface.


Available templates are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml):

  • build: In the check pipeline, runs the rpm-scratch-build job in the check pipeline.
  • lint: In the check pipeline, runs, once the parent job rpm-scratch-build is done, the rpm-linter + rpm-rpminspect jobs against the rpms built by the parent job.
  • test: In the check pipeline, runs, once the parent job rpm-scratch-build is done, the rpm-(sti|tmt)-test and rpm-install-test jobs against the rpms built by the parent job.
  • publish: In the gate pipeline, run a noop job (always succeed), to make the gate succeed. This makes Zuul to merge the Pull Request. Then in the promote pipeline, runs the rpm-build job (regular koji build).

You will notice the run of the check-for-(sti|fmf)-tests jobs. They use the zuul_return - child_jobs feature to control the execution of rpm-(sti|tmt)-test jobs.

To attach an additional template to a repository you need to open a Pull Request on https://pagure.io/fedora-zuul-jobs-config.

Edit zuul.d/projects.yaml to add the publication jobs: 

- project:
    name: repository
    templates:
      - publish

repository should be like "rpms/python-gear".

Once the Pull Request is merged, the jobs of the templates will be attached to the repository along with the default ones.

Currently f34, f33, epel8 and master branches are supported.

Supported build arch

Our architecture only provides x86_64 VMs to run the functional test job rpm-install-test. rpm-(sti|tmt)-test jobs relies on Testing Farm which run on x86_64 arch as well). Thus the default PR's jobs workflow runs a scratch build with the x86_64 arch override on Koji. Built artifacts (rpms) are validated on a x86_64 VM if the distgit provides a functional test.

However it might be useful to get feedback from the CI in case of build failure on other Fedora supported architecture. Thus we have created the following jobs.

  • rpm-scratch-build-aarch64
  • rpm-scratch-build-armv7hl
  • rpm-scratch-build-i686
  • rpm-scratch-build-ppc64le
  • rpm-scratch-build-s390x

Those new jobs are attached to the default CI workflow. In order to not trigger them in case of a pure noarch package we have created a conditional job called check-for-arches that tell Zuul to trigger or not those jobs.

Note that all of them have been configured as "non voting".

Here is example of PR that have triggered such a jobs: https://src.fedoraproject.org/rpms/mlocate/pull-request/6#comment-64239

Sequence diagrams of the PR workflow

These diagrams show interactions between components involved in a workflow where build, lint, test, publish templates are used.

Pull Request created/updated/rechecked workflow

Pull Request approved workflow

Example of PR managed with that workflow

Nodepool nodes

Jobs are configured to execute on Fedora VMs or containers. Default jobs we provide are configured to use the right VM based on the PR target branch. For instance the rpm-install-test job, for the master branch, executes on a rawhide VM. rpm-(sti|tmt)-test jobs relies on Testing Farm, Zuul passes the branch information to Testing Farm which run the functional tests on the right target system.

Here the container for fedora-33: https://softwarefactory-project.io/cgit/config/tree/containers/fedora-33 All of the jobs from the default templates we provide for distgit's PR execute on containers.

Fedora's projects using Zuul

Projects registered into the Fedora's Zuul are listed here: projects list

Advanced topics

In-project Zuul configuration (.zuul.yaml)

Zuul is very flexible about where configuration elements are defined. Each piece of a Zuul configuration is managed in a git repository and the full configuration can be spread across an unlimited amount of git repositories.

For distgit projects on src.fedoraproject.org

For every distgit projects we believe that the Zuul pipeline should be similar, thus, by default, we do not allow package maintainer to add custom Zuul configurations in-distgit. But for specific cases a maintainer may need custom and non generic jobs and project's pipeline.

If there is a need for such custom in-distgit Zuul configuration then Zuul must be told to read that configuration from the distgit. To do so:

Open a Pull Request on https://pagure.io/fedora-project-config

Edit resources/fedora-distgits.yaml and add modify such as: 

resources:
  projects:
    Fedora-Distgits:
      ...
      source-repositories:
        ...
        - repository-name:
            zuul/include:
              - project
              - job

This will let Zuul load jobs and project stanza from the .zuul.yaml stored in the distgit repository. Then in the dist-git repository, open a Pull Request with a .zuul.yaml file like this: 

 - project:
     templates:
       - build
       - lint
       - test
     check:
       jobs:
         - noop

This runs the noop special Zuul job (which simply returns SUCCESS immediately) along with the other jobs from the templates. This project pipeline allows to keep the default Zuul jobs for a distgit project but also to add custom jobs to the default pipeline. Now proceed to running jobs to run a real job instead of noop.

For a source project on pagure.io

The default configuration for pagure.io projects lets Zuul load all Zuul configuration elements, so you do not need to send a pull request to fedora-project-config in this case.

For a minimal configuration, in the project repository, open a Pull Request with a .zuul.yaml such as: 

 - project:
     check:
       jobs:
         - noop

This runs the noop special Zuul job (which simply returns SUCCESS immediately). Your pull request should get some output indicating that the tests ran successfully. Now proceed to the next section to learn how to run a real job instead of noop.

Running custom jobs

To run some real tests, you will have to define a job. Add a job to the top of the .zuul.yaml, like this: 

 - job:
     name: my-job
     description: My custom job
     run: ci/run.yaml

and change the job to run from noop to my-job: 

     check:
       jobs:
         - my-job

Finally, in a file called ci/run.yaml write the job Ansible playbook: 

 - hosts: all
   tasks:
     - name: List project directory on the test system
       command: ls -al {{ ansible_user_dir }}/{{ zuul.project.src_dir }}

That test simply lists out the project source files. For a sample of how a real test setup looks, ci/run.yaml might instead look like this: 

   - hosts: all
     tasks:
       - name: Ensure tox is installed
         include_role:
           name: ensure-tox
       - name: Install all Python versions to test
         package:
           name: ['python27', 'python36', 'python38', 'python39', 'python310']
           state: present
         become: yes
       - name: Run tox
         command: "{{ tox_executable }}"
         args:
           chdir: '{{ zuul.project.src_dir }}'

This is for a Python project which uses the tox automation project to run its test suite. The first task runs a built-in Zuul role which does everything necessary for tox to be available in the test environment. The second installs the Fedora packages for various Python interpreters, so the tests can be run on several different Python versions (a feature of tox). The third task runs tox (using a variable, tox_executable, defined by the ensure-tox task) in the project's source directory. tox returns zero if tests pass, non-zero if they fail. The CI system interprets a non-zero result of any task as a test failure. If your test system returns zero on failure you must somehow ensure a task in the job returns non-zero for the failure to be recognized.

Zuul has many useful built-in roles for various languages and ecosystems; you can read up on them in the Zuul docs. The job files are Ansible playbooks; see the extensive Ansible documentation if you are not familiar with these. If you don't specify an environment to run in, they will run in a Fedora VM environment. You can change this by setting the job's nodeset in .zuul.yaml. For instance, this: 

 - job:
     name: my-job
     description: My custom job
     run: ci/run.yaml
     nodeset: fedora-latest-container

will run the job in a container populated based on the most recent stable Fedora release.

Contacts

For any questions or issues you can contact fbo or tristanC on https://matrix.to/#/#sf-ops:matrix.org

Or create an issue on https://pagure.io/fedora-ci/general/issues.

Contributing

In order to propose any CI config update on the Fedora Zuul CI please follow the README.md on https://pagure.io/fedora-project-config.

FAQ

CI config repositories

Here is the list of repositories loaded by Zuul at runtime from where it loads its configuration.

Typical Zuul error messages

Here is a list of error messages that Zuul may return as a comment to a Pull-Request. Some of them might be difficult to interpret, so this section tries to give some clues about them.

  • Unable to freeze job graph: Job rpm-linter depends on rpm-scratch-build which was not run.

Unable to freeze job graph means that Zuul cannot run a job due to an invalid job context. In this specific case, the expected job execution context is not met to run the rpm-linter job. Such messages are usually due to CI config inconsistencies.

  • Merge Failed. - This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

The Merge Failed term does not relate to the final merge of the Pull-Request, but instead to an attempt from Zuul to merge in a sandbox the Pull-Request on the target branch.

The This change or one of its cross-repo dependencies relates to this Pull-Request or one of its dependent Pull-Requests. Dependent Pull-Requests are those specified by a Depends-on or behind the Merge Queue in a Gate pipeline.

How to re-run a CI job

Simply comment the Pull-Request with "recheck".

How to set custom rules for rpm-linter job

Define a <package>.rpmlintrc file like it has been done for this package: https://src.fedoraproject.org/rpms/zuul/blob/master/f/zuul.rpmlintrc

How to specify PR dependencies (runtime)

Sometime that might be needed, for instance when a distgit functional test job needs a not yet published package to succeed. In the case the needed dependent package is proposed as a PR, then you can tell Zuul to include the dependent PR built artifacts (so the rpms) into the testing environment.

To do so, add a "Depends-on" into the initial comment of the PR. 

Depends-on: <url-of-the-pr>
Depends-on: <url-of-another-pr>

As you can see you can also list multiple dependent PRs.

Note that this is only supported for the runtime dependencies especially for rpm-tests and rpm-install-tests jobs.

Request test node SSH access

Sometime, for debugging purposes you might want to request SSH access to a test node. Thanks to Zuul we can handle such requests quite easily by putting a test node on hold for given amount of time.

To request a node to be put on hold please ask us via IRC on the #software-factory or #fedora-ci channels on freenode.

rpminspect in-distgit configuration file

The rpm-rpminspect job is able to pass a in-repo rpminspect.conf file. Simply put the file at the root of the distgit repository

Skip built packages installation

In some cases the installation of all built packages is not expected during the rpm-test or rpm-install-test test. Some packages could be marked as excluded. To do so, first, Zuul must be configured to enable in-repo configuration (see here to include the project configuration element via zuul/include). Then a .zuul.yaml file must be created into the distgit repository such as: 

 - project:
   vars:
     install_repo_exclude:
       - <pkg1>
       - <pkg2>

How to opt-out a package from Zuul

  • Create a fork of https://pagure.io/fedora-project-config
  • Update the resources/fedora-distgits.yaml file to remove the package(s)
  • Commit and push the change on your fork
  • Use Open PR button on pagure.io (from your forked page) to Open a Pull Request to pagure.io/fedora-project-config

The Pull Request is then validated by the CI, then wait for someone to merge it.

Testing Farm issues

If you encounter issues with Testing Farm, please first check the status page for possible outages. If the issue persists, please file an issue to Fedora CI issue tracker.

Retrieved from "https://fedoraproject.org/w/index.php?title=Zuul-based-ci&oldid=702224"