# Running tests

<!-- toc -->

You can run the entire test collection using `x`. But note that running the
*entire* test collection is almost never what you want to do during local
development because it takes a really long time. For local development, see the
subsection after on how to run a subset of tests.

<div class="warning">

Running plain `./x test` will build the stage 1 compiler and then run the whole
test suite. This not only include `tests/`, but also `library/`, `compiler/`,
`src/tools/` package tests and more.

You usually only want to run a subset of the test suites (or even a smaller set
of tests than that) which you expect will exercise your changes. PR CI exercises
a subset of test collections, and merge queue CI will exercise all of the test
collection.

</div>

```text
./x test
```

The test results are cached and previously successful tests are `ignored` during
testing. The stdout/stderr contents as well as a timestamp file for every test
can be found under `build/<target-tuple>/test/` for the given
`<target-tuple>`. To force-rerun a test (e.g. in case the test runner fails to
notice a change) you can use the `--force-rerun` CLI option.

> **Note on requirements of external dependencies**
>
> Some test suites may require external dependencies. This is especially true of
> debuginfo tests. Some debuginfo tests require a Python-enabled gdb. You can
> test if your gdb install supports Python by using the `python` command from
> within gdb. Once invoked you can type some Python code (e.g. `print("hi")`)
> followed by return and then `CTRL+D` to execute it. If you are building gdb
> from source, you will need to configure with
> `--with-python=<path-to-python-binary>`.

## Running a subset of the test suites

When working on a specific PR, you will usually want to run a smaller set of
tests. For example, a good "smoke test" that can be used after modifying rustc
to see if things are generally working correctly would be to exercise the `ui`
test suite ([`tests/ui`]):

```text
./x test tests/ui
```

Of course, the choice of test suites is
somewhat arbitrary, and may not suit the task you are doing. For example, if you
are hacking on debuginfo, you may be better off with the debuginfo test suite:

```text
./x test tests/debuginfo
```

If you only need to test a specific subdirectory of tests for any given test
suite, you can pass that directory as a filter to `./x test`:

```text
./x test tests/ui/const-generics
```

> **Note for MSYS2**
>
> On MSYS2 the paths seem to be strange and `./x test` neither recognizes
> `tests/ui/const-generics` nor `tests\ui\const-generics`. In that case, you can
> workaround it by using e.g. `./x test ui
> --test-args="tests/ui/const-generics"`.

Likewise, you can test a single file by passing its path:

```text
./x test tests/ui/const-generics/const-test.rs
```

`x` doesn't support running a single tool test by passing its path yet. You'll
have to use the `--test-args` argument as described
[below](#running-an-individual-test).

```text
./x test src/tools/miri --test-args tests/fail/uninit/padding-enum.rs
```

### Run only the tidy script

```text
./x test tidy
```

### Run tests on the standard library

```text
./x test --stage 0 library/std
```

Note that this only runs tests on `std`; if you want to test `core` or other
crates, you have to specify those explicitly.

### Run the tidy script and tests on the standard library

```text
./x test --stage 0 tidy library/std
```

### Run tests on the standard library using a stage 1 compiler

```text
./x test --stage 1 library/std
```

By listing which test suites you want to run,
you avoid having to run tests for components you did not change at all.

<div class="warning">

Note that bors only runs the tests with the full stage 2 build; therefore, while
the tests **usually** work fine with stage 1, there are some limitations.

</div>

### Run all tests using a stage 2 compiler

```text
./x test --stage 2
```

<div class="warning">
You almost never need to do this; CI will run these tests for you.
</div>

## Run unit tests on the compiler/library

You may want to run unit tests on a specific file with following:

```text
./x test compiler/rustc_data_structures/src/thin_vec/tests.rs
```

But unfortunately, it's impossible. You should invoke the following instead:

```text
./x test compiler/rustc_data_structures/ --test-args thin_vec
```

## Running an individual test

Another common thing that people want to do is to run an **individual test**,
often the test they are trying to fix. As mentioned earlier, you may pass the
full file path to achieve this, or alternatively one may invoke `x` with the
`--test-args` option:

```text
./x test tests/ui --test-args issue-1234
```

Under the hood, the test runner invokes the standard Rust test runner (the same
one you get with `#[test]`), so this command would wind up filtering for tests
that include "issue-1234" in the name. Thus, `--test-args` is a good way to run
a collection of related tests.

## Passing arguments to `rustc` when running tests

It can sometimes be useful to run some tests with specific compiler arguments,
without using `RUSTFLAGS` (during development of unstable features, with `-Z`
flags, for example).

This can be done with `./x test`'s `--compiletest-rustc-args` option, to pass
additional arguments to the compiler when building the tests.

## Editing and updating the reference files

If you have changed the compiler's output intentionally, or you are making a new
test, you can pass `--bless` to the test subcommand.

As an example,
if some tests in `tests/ui` are failing, you can run this command:

```text
./x test tests/ui --bless
```

It automatically adjusts the `.stderr`, `.stdout`, or `.fixed` files of all `test/ui` tests.
Of course you can also target just specific tests with the `--test-args your_test_name` flag,
just like when running the tests without the `--bless` flag.

## Configuring test running

There are a few options for running tests:

* `bootstrap.toml` has the `rust.verbose-tests` option. If `false`, each test will
  print a single dot (the default). If `true`, the name of every test will be
  printed. This is equivalent to the `--quiet` option in the [Rust test
  harness](https://doc.rust-lang.org/rustc/tests/).
* The environment variable `RUST_TEST_THREADS` can be set to the number of
  concurrent threads to use for testing.

## Passing `--pass $mode`

Pass UI tests now have three modes, `check-pass`, `build-pass` and `run-pass`.
When `--pass $mode` is passed, these tests will be forced to run under the given
`$mode` unless the directive `//@ ignore-pass` exists in the test file. For
example, you can run all the tests in `tests/ui` as `check-pass`:

```text
./x test tests/ui --pass check
```

By passing `--pass $mode`, you can reduce the testing time. For each mode,
please see [Controlling pass/fail
expectations](ui.md#controlling-passfail-expectations).

## Running tests with different "compare modes"

UI tests may have different output depending on certain "modes" that the
compiler is in. For example, when using the Polonius mode, a test `foo.rs` will
first look for expected output in `foo.polonius.stderr`, falling back to the
usual `foo.stderr` if not found. The following will run the UI test suite in
Polonius mode:

```text
./x test tests/ui --compare-mode=polonius
```

See [Compare modes](compiletest.md#compare-modes) for more details.

## Running tests manually

Sometimes it's easier and faster to just run the test by hand. Most tests are
just `.rs` files, so after [creating a rustup
toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain), you
can do something like:

```text
rustc +stage1 tests/ui/issue-1234.rs
```

This is much faster, but doesn't always work. For example, some tests include
directives that specify specific compiler flags, or which rely on other crates,
and they may not run the same without those options.

## Running tests on a remote machine

Tests may be run on a remote machine (e.g. to test builds for a different
architecture). This is done using `remote-test-client` on the build machine to
send test programs to `remote-test-server` running on the remote machine.
`remote-test-server` executes the test programs and sends the results back to
the build machine. `remote-test-server` provides *unauthenticated remote code
execution* so be careful where it is used.

To do this, first build `remote-test-server` for the remote machine, e.g. for
RISC-V

```text
./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
```

The binary will be created at
`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy this
over to the remote machine.

On the remote machine, run the `remote-test-server` with the `--bind
0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should look
like this:

```text
$ ./remote-test-server -v --bind 0.0.0.0:12345
starting test server
listening on 0.0.0.0:12345!
```

Note that binding the server to 0.0.0.0 will allow all hosts able to reach your
machine to execute arbitrary code on your machine. We strongly recommend either
setting up a firewall to block external access to port 12345, or to use a more
restrictive IP address when binding.

You can test if the `remote-test-server` is working by connecting to it and
sending `ping\n`. It should reply `pong`:

```text
$ nc $REMOTE_IP 12345
ping
pong
```

To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
variable then use `x` as usual. For example, to run `ui` tests for a RISC-V
machine with the IP address `1.2.3.4` use

```text
export TEST_DEVICE_ADDR="1.2.3.4:12345"
./x test tests/ui --target riscv64gc-unknown-linux-gnu
```

If `remote-test-server` was run with the verbose flag, output on the test
machine may look something like

```text
[...]
run "/tmp/work/test1007/a"
run "/tmp/work/test1008/a"
run "/tmp/work/test1009/a"
run "/tmp/work/test1010/a"
run "/tmp/work/test1011/a"
run "/tmp/work/test1012/a"
run "/tmp/work/test1013/a"
run "/tmp/work/test1014/a"
run "/tmp/work/test1015/a"
run "/tmp/work/test1016/a"
run "/tmp/work/test1017/a"
run "/tmp/work/test1018/a"
[...]
```

Tests are built on the machine running `x` not on the remote machine. Tests
which fail to build unexpectedly (or `ui` tests producing incorrect build
output) may fail without ever running on the remote machine.

## Testing on emulators

Some platforms are tested via an emulator for architectures that aren't readily
available. For architectures where the standard library is well supported and
the host operating system supports TCP/IP networking, see the above instructions
for testing on a remote machine (in this case the remote machine is emulated).

There is also a set of tools for orchestrating running the tests within the
emulator. Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are
set up to automatically run the tests under emulation on GitHub Actions. The
following will take a look at how a target's tests are run under emulation.

The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
architecture. Included in the Rust tree are the tools [remote-test-client] and
[remote-test-server] which are programs for sending test programs and libraries
to the emulator, and running the tests within the emulator, and reading the
results.  The Docker image is set up to launch `remote-test-server` and the
build tools use `remote-test-client` to communicate with the server to
coordinate running tests (see [src/bootstrap/src/core/build_steps/test.rs]).

> **TODO**
>
> - Is there any support for using an iOS emulator?
> - It's also unclear to me how the wasm or asm.js tests are run.


## Running rustc_codegen_gcc tests

First thing to know is that it only supports linux x86_64 at the moment. We will
extend its support later on.

You need to update `codegen-backends` value in your `bootstrap.toml` file in the
`[rust]` section and add "gcc" in the array:

```toml
codegen-backends = ["llvm", "gcc"]
```

Then you need to install libgccjit 12. For example with `apt`:

```text
apt install libgccjit-12-dev
```

Now you can run the following command:

```text
./x test compiler/rustc_codegen_gcc/
```

If it cannot find the `.so` library (if you installed it with `apt` for example), you
need to pass the library file path with `LIBRARY_PATH`:

```text
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/12/ ./x test compiler/rustc_codegen_gcc/
```

If you encounter bugs or problems, don't hesitate to open issues on the
[`rustc_codegen_gcc`
repository](https://github.com/rust-lang/rustc_codegen_gcc/).