Home • Explore • Blog • Models • CI
☰



nixpkgs
nixos/doc/manual/development/developing-the-test-driver.chapter.md
e3e9297388e159aaac8ba7ee4679870bf2b014e809630de30000000300000ad0

# Developing the NixOS Test Driver {#chap-developing-the-test-driver}

The NixOS test framework is a project of its own.

It consists of roughly the following components:

 - `nixos/lib/test-driver`: The Python framework that sets up the test and runs the [`testScript`](#test-opt-testScript)
 - `nixos/lib/testing`: The Nix code responsible for the wiring, written using the (NixOS) Module System.

These components are exposed publicly through:

 - `nixos/lib/default.nix`: The public interface that exposes the `nixos/lib/testing` entrypoint.
 - `flake.nix`: Exposes the `lib.nixos`, including the public test interface.

Beyond the test driver itself, its integration into NixOS and Nixpkgs is important.

 - `pkgs/top-level/all-packages.nix`: Defines the `nixosTests` attribute, used
   by the package `tests` attributes and OfBorg.
 - `nixos/release.nix`: Defines the `tests` attribute built by Hydra, independently, but analogous to `nixosTests`
 - `nixos/release-combined.nix`: Defines which tests are channel blockers.

Finally, we have legacy entrypoints that users should move away from, but are cared for on a best effort basis.
These include `pkgs.nixosTest`, `testing-python.nix` and `make-test-python.nix`.

## Testing changes to the test framework {#sec-test-the-test-framework}

We currently have limited unit tests for the framework itself. You may run these with `nix-build -A nixosTests.nixos-test-driver`.

When making significant changes to the test framework, we run the tests on Hydra, to avoid disrupting the larger NixOS project.

For this, we use the `python-test-refactoring` branch in the `NixOS/nixpkgs` repository, and its [corresponding Hydra jobset](https://hydra.nixos.org/jobset/nixos/python-test-refactoring).
This branch is used as a pointer, and not as a feature branch.

1. Rebase the PR onto a recent, good evaluation of `nixos-unstable`
2. Create a baseline evaluation by force-pushing this revision of `nixos-unstable` to `python-test-refactoring`.
3. Note the evaluation number (we'll call it `<previous>`)
4. Push the PR to `python-test-refactoring` and evaluate the PR on Hydra
5. Create a comparison URL by navigating to the latest build of the PR and adding to the URL `?compare=<previous>`. This is not necessary for the evaluation that comes right after the baseline.

Review the removed tests and newly failed tests using the constructed URL; otherwise you will accidentally compare iterations of the PR instead of changes to the PR base.

As we currently have some flaky tests, newly failing tests are expected, but should be reviewed to make sure that
 - The number of failures did not increase significantly.
 - All failures that do occur can reasonably be assumed to fail for a different reason than the changes.
Chunks
f2d0d880 (1st chunk of `nixos/doc/manual/development/developing-the-test-driver.chapter.md`)
Title: Developing and Testing the NixOS Test Driver
Summary
This section details the architecture of the NixOS test framework, outlining its Python (`nixos/lib/test-driver`) and Nix (`nixos/lib/testing`) components, and how they are exposed publicly. It also describes the framework's integration into NixOS and Nixpkgs for defining and building tests, including channel blockers, and mentions legacy entrypoints. The latter part of the text explains the process for testing significant changes to the test framework itself, emphasizing the use of the `python-test-refactoring` branch on Hydra for comprehensive evaluation, including steps for creating baselines and comparing PR changes to avoid disrupting the larger NixOS project. It also advises on reviewing test failures, acknowledging flakiness, and ensuring new failures are not caused by the changes.