Home Explore Blog CI



rustc

3rd chunk of `src/tests/best-practices.md`
e2b3ebc959e0ad28c76ad0c86b5841011f4c5fb115ed199c0000000100000ecf
//! Optional: Remarks on related tests/issues, external APIs/tools, crash
//!     mechanism, how it's fixed, FIXMEs, limitations, etc.
//! Example: This test is like `tests/attrs/linkage.rs`, but this test is
//!     specifically checking `#[coverage]` which exercises a different code
//!     path. The ICE was triggered during attribute validation when we tried
//!     to construct a `def_path_str` but only emitted the diagnostic when the
//!     platform is windows, causing an ICE on unix.
//!
//! Links to relevant issues and discussions. Examples below:
//! Regression test for <https://github.com/rust-lang/rust/issues/123456>.
//! See also <https://github.com/rust-lang/rust/issues/101345>.
//! See discussion at <https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/123456-example-topic>.
//! See [`clone(2)`].
//!
//! [`clone(2)`]: https://man7.org/linux/man-pages/man2/clone.2.html

//@ ignore-windows
// Reason: (why is this test ignored for windows? why not specifically
// windows-gnu or windows-msvc?)

// Optional: Summary of test cases: What positive cases are checked?
// What negative cases are checked? Any specific quirks?

fn main() {
    #[coverage]
    //~^ ERROR coverage attribute can only be applied to function items.
    let _ = {
        // Comment highlighting something that deserves reader attention.
        fn foo() {}
    };
}
```

For how much context/explanation is needed, it is up to the author and
reviewer's discretion. A good rule of thumb is non-trivial things exercised in
the test deserves some explanation to help other contributors to understand.
This may include remarks on:

- How an ICE can get triggered if it's quite elaborate.
- Related issues and tests (e.g. this test is like another test but is kept
  separate because...).
- Platform-specific behaviors.
- Behavior of external dependencies and APIs: syscalls, linkers, tools,
  environments and the likes.

## Test content

- Try to make sure the test is as minimal as possible.
- Minimize non-critical code and especially minimize unnecessary syntax and type
  errors which can clutter stderr snapshots.
- Where possible, use semantically meaningful names (e.g. `fn
  bare_coverage_attributes() {}`).

## Flaky tests

All tests need to strive to be reproducible and reliable. Flaky tests are the
worst kind of tests, arguably even worse than not having the test in the first
place.

- Flaky tests can fail in completely unrelated PRs which can confuse other
  contributors and waste their time trying to figure out if test failure is
  related.
- Flaky tests provide no useful information from its test results other than
  it's flaky and not reliable: if a test passed but it's flakey, did I just get
  lucky? if a test is flakey but it failed, was it just spurious?
- Flaky tests degrade confidence in the whole test suite. If a test suite can
  randomly spuriously fail due to flaky tests, did the whole test suite pass or
  did I just get lucky/unlucky?
- Flaky tests can randomly fail in full CI, wasting previous full CI resources.

## Compiletest directives

See [compiletest directives] for a listing of directives.

- For `ignore-*`/`needs-*`/`only-*` directives, unless extremely obvious,
  provide a brief remark on why the directive is needed. E.g. `"//@ ignore-wasi
  (wasi codegens the main symbol differently)"`.
- When using `//@ ignore-auxiliary`, specify the corresponding main test files,
  e.g. ``//@ ignore-auxiliary (used by `./foo.rs`)``.

## FileCheck best practices

See [LLVM FileCheck guide][FileCheck] for details.

- Avoid matching on specific register numbers or basic block numbers unless
  they're special or critical for the test. Consider using patterns to match
  them where suitable.

> **TODO**
>
> Pending concrete advice.


Title: Detailed Test Structure, Content, and Reliability
Summary
This section elaborates on structuring tests with descriptive comments that include the test's purpose, related issues, and platform-specific behaviors. It emphasizes minimizing code complexity in tests and using meaningful names. A significant portion is dedicated to addressing flaky tests, highlighting their detrimental impact on CI reliability and contributor confidence. The section also briefly touches on compiletest directives and FileCheck best practices, advocating for clear remarks explaining the need for directives and avoiding specific register or block number matching in FileCheck.