Toolstate, Public CI Dashboard, and Determining CI Configuration

The build logs for `rust-lang/rust` are huge, and it’s not practical to find what caused the build to fail by looking at the logs. To improve the developers’ experience we developed a bot called [Rust Log Analyzer][rla] (RLA) that receives the build logs on failure and extracts the error message automatically, posting it on the PR. The bot is not hardcoded to look for error strings, but was trained with a bunch of build failures to recognize which lines are common between builds and which are not. While the generated snippets can be weird sometimes, the bot is pretty good at identifying the relevant lines even if it’s an error we've never seen before. ### Toolstate to support allowed failures The `rust-lang/rust` repo doesn’t only test the compiler on its CI, but also a variety of tools and documentation. Some documentation is pulled in via git submodules. If we blocked merging rustc PRs on the documentation being fixed, we would be stuck in a chicken-and-egg problem, because the documentation's CI would not pass since updating it would need the not-yet-merged version of rustc to test against (and we usually require CI to be passing). To avoid the problem, submodules are allowed to fail, and their status is recorded in [rust-toolstate]. When a submodule breaks, a bot automatically pings the maintainers so they know about the breakage, and it records the failure on the toolstate repository. The release process will then ignore broken tools on nightly, removing them from the shipped nightlies. While tool failures are allowed most of the time, they’re automatically forbidden a week before a release: we don’t care if tools are broken on nightly but they must work on beta and stable, so they also need to work on nightly a few days before we promote nightly to beta. More information is available in the [toolstate documentation]. ## Public CI dashboard To monitor the Rust CI, you can have a look at the [public dashboard] maintained by the infra-team. These are some useful panels from the dashboard: - Pipeline duration: check how long the auto builds takes to run. - Top slowest jobs: check which jobs are taking the longest to run. - Change in median job duration: check what jobs are slowest than before. Useful to detect regressions. - Top failed jobs: check which jobs are failing the most. To learn more about the dashboard, see the [Datadog CI docs]. ## Determining the CI configuration If you want to determine which `bootstrap.toml` settings are used in CI for a particular job, it is probably easiest to just look at the build log. To do this: 1. Go to <https://github.com/rust-lang/rust/actions?query=branch%3Aauto+is%3Asuccess> to find the most recently successful build, and click on it. 2. Choose the job you are interested in on the left-hand side. 3. Click on the gear icon and choose "View raw logs" 4. Search for the string "Configure the build" 5. All of the build settings are listed below that starting with the `configure:` prefix.

This section describes the Toolstate system, which manages allowed failures for tools and documentation in the rust-lang/rust CI, and the public CI dashboard maintained by the infra-team, which provides insights into CI performance and failures. Finally, it explains how to determine the bootstrap.toml settings used for a specific CI job by examining the build logs.