Integration test suite

At the time of writing this document, we have 60 integration tests which tests abrt, libreport and satyr functionality in real world scenarios.

These tests are run nightly for each of the currently supported releases of Fedora and Red Hat Enterprise Linux. You can find the results on our public mirror.

Running the test suite

Caution

Don’t run the test suite on your machine directly, use virtual machine or machine dedicated for the testing.

In your virtual machine or dedicated machine:
  • clone the ABRT repository: git clone git://git.fedorahosted.org/abrt.git
  • go to abrt/tests/runtest/
  • run ./run

This will run all the tests in order specified in ./aux/test_order. When the testing is done, the output can be found in /tmp/abrt-testsuite/.

To override the destination directory or the other options, you can edit ./aux/config.sh.

Running single test

To run only one of the tests, you have to run it via ./aux/runner.sh. This script prepares your system by restarting required services and cleaning up possible stale files.

For example, to run dbus-api test, you have to pass the whole path to runtest.sh to runner.sh script:

./aux/runner.sh dbus-api/runtest.sh

Running a subset of the tests

To run only part of the tests, edit ./aux/test_order file and comment out the tests you don’t want to run.

For example, when using the test suite to test already installed packages, you don’t need to build and install git versions. In this case, you can comment out abrt-nightly-build test and all the \*-make-check tests. These are now disabled by default.

Writing new integration test

To create a new test you have to:
  • create a directory reflecting its name
  • create two files in that directory:
    • executable runtest.sh file
    • PURPOSE file
  • add the test to ./aux/test_order

The tests are written in bash using BeakerLib library. BeakerLib Quick Reference [PDF] is what you need.

It’s always a good idea to copy one of the existing tests as a base for your test. Good candidates are:

  • run-abrtd which basically serves as an example,
  • ccpp-plugin if you need to test crash handling (if you require crash directory),
  • dbus-api if you need to test DBus functionality.

Complex tests

Please provide an explanation for complex tests. Either explain certain sections of the test in comments in runtest.sh or create README file inside the directory of the test.

Output directory structure

To avoid painful digging in several distinct files logging is now divided into several logical levels:

$OUTPUT_ROOT variable is defined in aux/config.sh which represents the root for all logs.

For each stage of the test suite run there is corresponding directory in root directory. It contains stage.log file which contains output of script governing the stage.

/tmp/abrt-testsuite
├── format/
├── post/
├── pre/
├── report/
├── results
└── test/

Stages:

  • PRE
    install required packages, collect log files, collect system information
  • TEST
    run all the tests
  • FORMAT
    format the results for REPORT stage
  • REPORT
    report the results
  • POST
    collect logs, cleanup

For TEST stage there is an additional subdirectory for each test case:

/tmp/abrt-testsuite/test/
├── abrt-make-check
├── abrt-nightly-build
├── abrt-should-return-rating-0-on-fail
├── blacklisted-package
...

Each directory contains several files:

/tmp/abrt-testsuite/test/systemd-init/
├── dmesg
├── avc
├── fail.log
├── full.log
├── messages
└── protocol.log

Only full.log is mandatory. It contains stdout and stderr of the test run. protocol.log only contains the protocol generated by BeakerLib. If the test fails with FATAL error, protocol.log is not generated. In case of other failures, these are extracted to fail.log along with line numbers pointing to lines in full.log.

dmesg, messages and avc each contains log file messages written during the test run.