.. _testsuite: Integration test suite ====================== At the time of writing this document, we have 60 integration tests which tests abrt, libreport and satyr functionality in real world scenarios. These tests are run nightly for each of the currently supported releases of Fedora and Red Hat Enterprise Linux. You can find the results on our `public mirror `_. Running the test suite ---------------------- .. caution:: Don't run the test suite on your machine **directly**, use virtual machine or machine dedicated for the testing. In your virtual machine or dedicated machine: - clone the ABRT repository: ``git clone git://git.fedorahosted.org/abrt.git`` - go to ``abrt/tests/runtest/`` - run ``./run`` This will run all the tests in order specified in ``./aux/test_order``. When the testing is done, the output can be found in ``/tmp/abrt-testsuite/``. To override the destination directory or the other options, you can edit ``./aux/config.sh``. Running single test ''''''''''''''''''' To run only one of the tests, you have to run it via ``./aux/runner.sh``. This script prepares your system by restarting required services and cleaning up possible stale files. For example, to run ``dbus-api`` test, you have to pass the whole path to ``runtest.sh`` to ``runner.sh`` script:: ./aux/runner.sh dbus-api/runtest.sh Running a subset of the tests ''''''''''''''''''''''''''''' To run only part of the tests, edit ``./aux/test_order`` file and comment out the tests you don't want to run. For example, when using the test suite to test already installed packages, you don't need to build and install git versions. In this case, you can comment out ``abrt-nightly-build`` test and all the ``\*-make-check`` tests. These are now disabled by default. .. _newinttest: Writing new integration test ---------------------------- To create a new test you have to: - create a directory reflecting its name - create two files in that directory: * executable ``runtest.sh`` file * ``PURPOSE`` file - add the test to ``./aux/test_order`` The tests are written in bash using `BeakerLib library `_. BeakerLib Quick Reference [`PDF `_] is what you need. It's always a good idea to copy one of the existing tests as a base for your test. Good candidates are: - ``run-abrtd`` which basically serves as an example, - ``ccpp-plugin`` if you need to test crash handling (if you require crash directory), - ``dbus-api`` if you need to test DBus functionality. Complex tests ''''''''''''' Please provide an explanation for complex tests. Either explain certain sections of the test in comments in ``runtest.sh`` or create ``README`` file inside the directory of the test. Output directory structure -------------------------- To avoid painful digging in several distinct files logging is now divided into several logical levels: ``$OUTPUT_ROOT`` variable is defined in ``aux/config.sh`` which represents the root for all logs. For each stage of the test suite run there is corresponding directory in root directory. It contains `stage.log` file which contains output of script governing the stage. :: /tmp/abrt-testsuite ├── format/ ├── post/ ├── pre/ ├── report/ ├── results └── test/ Stages: ''''''' - ``PRE`` install required packages, collect log files, collect system information - ``TEST`` run all the tests - ``FORMAT`` format the results for REPORT stage - ``REPORT`` report the results - ``POST`` collect logs, cleanup For ``TEST`` stage there is an additional subdirectory for each test case: :: /tmp/abrt-testsuite/test/ ├── abrt-make-check ├── abrt-nightly-build ├── abrt-should-return-rating-0-on-fail ├── blacklisted-package ... Each directory contains several files: :: /tmp/abrt-testsuite/test/systemd-init/ ├── dmesg ├── avc ├── fail.log ├── full.log ├── messages └── protocol.log Only ``full.log`` is mandatory. It contains `stdout` and `stderr` of the test run. ``protocol.log`` only contains the protocol generated by BeakerLib. If the test fails with FATAL error, ``protocol.log`` is not generated. In case of other failures, these are extracted to ``fail.log`` along with line numbers pointing to lines in ``full.log``. ``dmesg``, ``messages`` and ``avc`` each contains log file messages written during the test run.