How to run package-level tests in the Fedora build environment

You may not have heard about it yet, but a new feature was recently added to the Fedora build environment (i.e., Koji and Taskotron) that allows package-level tests to be added to dist-git that will be run when a build is kicked off. This is a quick how-to guide for doing so.

Preface

1. If you are trying this for the first time, it would be a good idea to do so using the staging instances of dist-git, Koji, etc. See How to configure your Fedora development system to access the Fedora staging infrastructure for  instructions.
2. There is currently (as of 29 March 2017) some oddness when triggering tests to run in staging dist-git. The Taskotron code goes through and tries to clone the dist-git repository with a branch matching the fXX in the NVR. If it can’t find that, it assumes rawhide and gets master. However, rawhide is still f26 in staging. Unfortunately, the f26 branch in staging dist-git may or may not contain the full package source files, as well has having improperly configured commit rights. Thus, for the time being, the f25 branch in staging dist-git should to be used for tests.
3. Currently (as of 29 March 2017), there may be some oddness triggering tests to run from a branch other than master in production dist-git. Thus, for the time being, the master branch in production dist-git should to be used for tests.

Steps

1. Install the necessary development packages and complete the Fedora development setup and Taskotron setup on your Fedora development machine:

   $ sudo dnf install fedora-packager
   $ fedora-packager-setup
   # configure ssh keys
   $ sudo dnf install libtaskotron
   $ sudo usermod -aG taskotron $USER
   # log out and back in for the new group assignment to take effect

2. Identify the Fedora package for which you will be adding tests. You will need commit rights to the package in order to actually add any tests. It would also be a good idea to get in contact with the current Fedora packagers/maintainers to let them know what you’re up to!
3. Check out the package from dist-git using the Fedora RPM packaging utility, fedpkg.
4. Create a test subdirectory in the Fedora dist-git repository for the package for which you are adding tests.
5. Within the RPM test directory, create a subdirectory for each test. Name it something useful that identifies the test. If is related to a BugZilla, consider starting the name of the subdirectory with the BugID and include a brief summary of the test. For example, “bz123456-foo-needs-to-support-bar”.
6. Inside the per-test subdirectory, create/copy test scripts, etc. to run the desired test. You can find some trivial examples in the packages libtaskotron and python-avocado.
7. Inside the per-test subdirectory, create a file named runtask.yml using the following as an example. It runs “make run” as a shell command (which happens to be the standard method by which beakerlib tests are run). See the Taskotron documentation for details on the format of this file and the directives that can be used. Note that this simple task does NOT reports its results to ResultsDB.

   ---
   name: bz123456-foo-needs-to-support-bar
   desc: "Tests BZ#123456 - Foo needs to support bar."
   maintainer: your_userid
   
   input:
       args:
       - koji_build
       - arch
   
   environment:
       rpm:
       - make
   
   actions:
       - name: download the build
       koji:
           action: download
           koji_build: ${koji_build}
           arch: $arch
           target_dir: ${workdir}/rpms
   
       - name: install the build
       shell:
           - dnf install -y ${workdir}/rpms/*.rpm
   
       - name: run test
       shell:
           - make run

8. The following example is slightly more complex. It runs python code that generates a specially formatted YAML results file that gets reported to ResultsDB.

   ---
   name: bz123456-foo-needs-to-support-bar
   desc: "Tests BZ#123456 - Foo needs to support bar."
   maintainer: your_userid
   
   input:
       args:
       - koji_build
       - arch
   
   actions:
       - name: install the build
       shell:
           - dnf install -y ${workdir}/rpms/*.rpm
       - name: run bz123456-foo-needs-to-support-bar
       python:
           file: run_test.py
           callable: run
           koji_build: ${koji_build}
           arch: ${arch}
           artifactsdir: ${artifactsdir}
       export: test_output
   
       - name: report results to resultsdb
       resultsdb:
           results: ${test_output}

9. Inside the per-test subdirectory, you should create a README file in reStructuredText (RST) or Markdown (MD) format that explains your test and how to run it. Following is an example README.rst.

   name: bz123456-foo-needs-to-support-bar
   =======================================
   
   This is an example task for `Taskotron <https://fedoraproject.org/wiki/Taskotron>`_ that implements a test that foo supports bar (BugZilla #123456).
   
   Standalone you can run it like this::
   
     $ make run
   
   Through taskotron runner you can run it like this::
   
     $ runtask -i foo-1.1-111.fc25 -t koji_build -a noarch runtask.yml

10. Test your test to make sure it runs locally directly from the shell, and through libtaskotron by running the runtask command similar to the README example shown in the previous step. You may need to run as root using sudo depending on what your task does.
11. Bump the package release number in the package spec file to get it to rebuild in Koji.
12. Commit your changes to the dist-git repository.
13. Submit a package build in Koji using fedpkg build. Note that Taskotron does NOT currently run tests from dist-git when performing a “scratch” build.
14. Results (if reported) can be found in ResultsDB (production/staging) after the build has finished if you query for it. Item would be NVR, testcase would be whatever you put into the formula (which should get standardized some day soon).
15. If the task crashes or fails to report results, you won’t see a result in ResultsDB. But, you should at least see it in the Tasoktron dist-git builders log (production/staging) and ExecDB (production/staging).

Good luck!