The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/meson.build`.
Each test is a bash script.

Functional tests are run during `installCheck` in the `nix` package build, as well as separately from the build, in VM tests.

### Running the whole test suite

The whole test suite (functional and unit tests) can be run with:

```shell-session
$ checkPhase
```

### Grouping tests

Sometimes it is useful to group related tests so they can be easily run together without running the entire test suite.
Each test group is in a subdirectory of `tests`.
For example, `tests/functional/ca/meson.build` defines a `ca` test group for content-addressing derivation outputs.

That test group can be run like this:

```shell-session
$ meson test --suite ca
ninja: Entering directory `/home/jcericson/src/nix/master/build'
ninja: no work to do.
[1-20/20] 🌑 nix-functional-tests:ca / ca/why-depends                                1/20 nix-functional-tests:ca / ca/nix-run                                  OK               0.16s
[2-20/20] 🌒 nix-functional-tests:ca / ca/why-depends                                2/20 nix-functional-tests:ca / ca/import-derivation                        OK               0.17s
```

### Running individual tests

Individual tests can be run with `meson`:

```shell-session
$ meson test --verbose ${testName}
ninja: Entering directory `/home/jcericson/src/nix/master/build'
ninja: no work to do.
1/1 nix-functional-tests:main / ${testName}        OK               0.41s

Ok:                 1
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /home/jcericson/src/nix/master/build/meson-logs/testlog.txt
```

The `--verbose` flag will make Meson also show the console output of each test for easier debugging.
The test script will then be traced with `set -x` and the output displayed as it happens,
regardless of whether the test succeeds or fails.

Tests can be also run directly without `meson`:

```shell-session
$ TEST_NAME=${testName} NIX_REMOTE='' PS4='+(${BASH_SOURCE[0]-$0}:$LINENO) tests/functional/${testName}.sh
+(${testName}.sh:1) foo
output from foo
+(${testName}.sh:2) bar
output from bar
...
```

### Debugging failing functional tests

When a functional test fails, it usually does so somewhere in the middle of the script.

To figure out what's wrong, it is convenient to run the test regularly up to the failing `nix` command, and then run that command with a debugger like GDB.

For example, if the script looks like:

```bash
foo
nix blah blub
bar
```
edit it like so:

```diff
 foo
-nix blah blub
+gdb --args nix blah blub
 bar
```

Then, running the test with [`--interactive`](https://mesonbuild.com/Unit-tests.html#other-test-options) will prevent Meson from hijacking the terminal so you can drop you into GDB once the script reaches that point:

```shell-session
$ meson test ${testName} --interactive
...
+ gdb blash blub
GNU gdb (GDB) 12.1
...
(gdb)
```

One can debug the Nix invocation in all the usual ways.
For example, enter `run` to start the Nix invocation.

### Troubleshooting

Sometimes running tests in the development shell may leave artefacts in the local repository.
To remove any traces of that:

```console
git clean -x --force tests
```

### Characterisation testing { #characterisation-testing-functional }

Occasionally, Nix utilizes a technique called [Characterisation Testing](https://en.wikipedia.org/wiki/Characterization_test) as part of the functional tests.
This technique is to include the exact output/behavior of a former version of Nix in a test in order to check that Nix continues to produce the same behavior going forward.

For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.

It is frequently useful to regenerate the expected output.
To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.