Home Explore Blog Models CI



nixpkgs

4th chunk of `doc/build-helpers/testers.chapter.md`
7c07f6d0e275ce02fc188c57e04f7baef7b0dc98b92064290000000100000fa8
This example will run the command `hello --version`, and then check that the version of the `hello` package is in the output of the command.

```nix
{ passthru.tests.version = testers.testVersion { package = hello; }; }
```

:::

:::{.example #ex-testversion-different-commandversion}

# Check the program version using a specified command and expected version string

This example will run the command `leetcode -V`, and then check that `leetcode 0.4.2` is in the output of the command as a whole word (separated by whitespaces).
This means that an output like "leetcode 0.4.21" would fail the tests, and an output like "You're running leetcode 0.4.2" would pass the tests.

A common usage of the `version` attribute is to specify `version = "v${version}"`.

```nix
{
  version = "0.4.2";

  passthru.tests.version = testers.testVersion {
    package = leetcode-cli;
    command = "leetcode -V";
    version = "leetcode ${version}";
  };
}
```

:::

## `testBuildFailure` {#tester-testBuildFailure}

Make sure that a build does not succeed. This is useful for testing testers.

This returns a derivation with an override on the builder, with the following effects:

 - Fail the build when the original builder succeeds
 - Move `$out` to `$out/result`, if it exists (assuming `out` is the default output)
 - Save the build log to `$out/testBuildFailure.log` (same)

While `testBuildFailure` is designed to keep changes to the original builder's environment to a minimum, some small changes are inevitable:

 - The file `$TMPDIR/testBuildFailure.log` is present. It should not be deleted.
 - `stdout` and `stderr` are a pipe instead of a tty. This could be improved.
 - One or two extra processes are present in the sandbox during the original builder's execution.
 - The derivation and output hashes are different, but not unusual.
 - The derivation includes a dependency on `buildPackages.bash` and `expect-failure.sh`, which is built to include a transitive dependency on `buildPackages.coreutils` and possibly more.
   These are not added to `PATH` or any other environment variable, so they should be hard to observe.

:::{.example #ex-testBuildFailure-showingenvironmentchanges}

# Check that a build fails, and verify the changes made during build

```nix
runCommand "example"
  {
    failed = testers.testBuildFailure (
      runCommand "fail" { } ''
        echo ok-ish >$out
        echo failing though
        exit 3
      ''
    );
  }
  ''
    grep -F 'ok-ish' $failed/result
    grep -F 'failing though' $failed/testBuildFailure.log
    [[ 3 = $(cat $failed/testBuildFailure.exit) ]]
    touch $out
  ''
```

:::

## `testBuildFailure'` {#tester-testBuildFailurePrime}

This tester wraps the functionality provided by [`testers.testBuildFailure`](#tester-testBuildFailure) to make writing checks easier by simplifying checking the exit code of the builder and asserting the existence of entries in the builder's log.
Additionally, users may specify a script containing additional checks, accessing the result of applying `testers.testBuildFailure` through the variable `failed`.

NOTE: This tester will produce an empty output and exit with success if none of the checks fail; there is no need to `touch "$out"` in `script`.

:::{.example #ex-testBuildFailurePrime-doc-example}

# Check that a build fails, and verify the changes made during build

Re-using the example from [`testers.testBuildFailure`](#ex-testBuildFailure-showingenvironmentchanges), we can see how common checks are made easier and remove the need for `runCommand`:

```nix
testers.testBuildFailure' {
  drv = runCommand "doc-example" { } ''
    echo ok-ish >"$out"
    echo failing though
    exit 3
  '';
  expectedBuilderExitCode = 3;
  expectedBuilderLogEntries = [ "failing though" ];
  script = ''
    grep --silent -F 'ok-ish' "$failed/result"
  '';
}
```

:::

### Inputs {#tester-testBuildFailurePrime-inputs}

`drv` (derivation)

: The failing derivation to wrap with `testBuildFailure`.

`name` (string, optional)

Title: Nix Testers: `testBuildFailure` and `testBuildFailure'` for Validating Expected Build Failures
Summary
This chunk first concludes examples for `testers.testVersion`, demonstrating how to check program versions using default and custom commands/version strings. It then introduces `testers.testBuildFailure`, a utility for ensuring that a build *fails* as expected, which is particularly useful for testing other testers. This tester wraps the original builder, failing itself if the original succeeds, moving any `$out` to `$out/result`, and saving build logs to `$out/testBuildFailure.log`, while also detailing the minor environmental changes it introduces. Following this, `testers.testBuildFailure'` is presented as a simplified wrapper for `testBuildFailure`, making it easier to check expected builder exit codes and log entries, and allowing users to provide an additional script for more specific validations.