Managing and Debugging Functional Tests

The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/meson.build`. Each test is a bash script. Functional tests are run during `installCheck` in the `nix` package build, as well as separately from the build, in VM tests. ### Running the whole test suite The whole test suite (functional and unit tests) can be run with: ```shell-session $ checkPhase ``` ### Grouping tests Sometimes it is useful to group related tests so they can be easily run together without running the entire test suite. Each test group is in a subdirectory of `tests`. For example, `tests/functional/ca/meson.build` defines a `ca` test group for content-addressing derivation outputs. That test group can be run like this: ```shell-session $ meson test --suite ca ninja: Entering directory `/home/jcericson/src/nix/master/build' ninja: no work to do. [1-20/20] 🌑 nix-functional-tests:ca / ca/why-depends 1/20 nix-functional-tests:ca / ca/nix-run OK 0.16s [2-20/20] 🌒 nix-functional-tests:ca / ca/why-depends 2/20 nix-functional-tests:ca / ca/import-derivation OK 0.17s ``` ### Running individual tests Individual tests can be run with `meson`: ```shell-session $ meson test --verbose ${testName} ninja: Entering directory `/home/jcericson/src/nix/master/build' ninja: no work to do. 1/1 nix-functional-tests:main / ${testName} OK 0.41s Ok: 1 Expected Fail: 0 Fail: 0 Unexpected Pass: 0 Skipped: 0 Timeout: 0 Full log written to /home/jcericson/src/nix/master/build/meson-logs/testlog.txt ``` The `--verbose` flag will make Meson also show the console output of each test for easier debugging. The test script will then be traced with `set -x` and the output displayed as it happens, regardless of whether the test succeeds or fails. Tests can be also run directly without `meson`: ```shell-session $ TEST_NAME=${testName} NIX_REMOTE='' PS4='+(${BASH_SOURCE[0]-$0}:$LINENO) tests/functional/${testName}.sh +(${testName}.sh:1) foo output from foo +(${testName}.sh:2) bar output from bar ... ``` ### Debugging failing functional tests When a functional test fails, it usually does so somewhere in the middle of the script. To figure out what's wrong, it is convenient to run the test regularly up to the failing `nix` command, and then run that command with a debugger like GDB. For example, if the script looks like: ```bash foo nix blah blub bar ``` edit it like so: ```diff foo -nix blah blub +gdb --args nix blah blub bar ``` Then, running the test with [`--interactive`](https://mesonbuild.com/Unit-tests.html#other-test-options) will prevent Meson from hijacking the terminal so you can drop you into GDB once the script reaches that point: ```shell-session $ meson test ${testName} --interactive ... + gdb blash blub GNU gdb (GDB) 12.1 ... (gdb) ``` One can debug the Nix invocation in all the usual ways. For example, enter `run` to start the Nix invocation. ### Troubleshooting Sometimes running tests in the development shell may leave artefacts in the local repository. To remove any traces of that: ```console git clean -x --force tests ``` ### Characterisation testing { #characterisation-testing-functional } Occasionally, Nix utilizes a technique called [Characterisation Testing](https://en.wikipedia.org/wiki/Characterization_test) as part of the functional tests. This technique is to include the exact output/behavior of a former version of Nix in a test in order to check that Nix continues to produce the same behavior going forward. For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered. It is frequently useful to regenerate the expected output. To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.

This document outlines how to manage, run, and debug functional tests, which are bash scripts located in the `tests/functional` directory. The entire test suite can be executed using `checkPhase`. Tests can be grouped and run selectively using `meson test --suite <group_name>`, or individual tests can be run with `meson test --verbose <testName>` for detailed output, or directly via their script. For debugging failing tests, users can insert `gdb --args` before `nix` commands in the script and run the test with `meson test --interactive`. Troubleshooting advice includes using `git clean -x --force tests` to remove test-related artifacts. Finally, it introduces characterization testing, a method to ensure consistent behavior, allowing regeneration of expected outputs with `_NIX_TEST_ACCEPT=1`.