Caching Mechanisms and Custom Tooling for CI

builders. - We can avoid reinstalling tools (like QEMU or the Android emulator) every time thanks to Docker image caching. - Users can run the same tests in the same environment locally by just running `cargo run --manifest-path src/ci/citool/Cargo.toml run-local <job-name>`, which is awesome to debug failures. Note that there are only linux docker images available locally due to licensing and other restrictions. The docker images prefixed with `dist-` are used for building artifacts while those without that prefix run tests and checks. We also run tests for less common architectures (mainly Tier 2 and Tier 3 platforms) in CI. Since those platforms are not x86 we either run everything inside QEMU or just cross-compile if we don’t want to run the tests for that platform. These builders are running on a special pool of builders set up and maintained for us by GitHub. ## Caching Our CI workflow uses various caching mechanisms, mainly for two things: ### Docker images caching The Docker images we use to run most of the Linux-based builders take a *long* time to fully build. To speed up the build, we cache them using [Docker registry caching], with the intermediate artifacts being stored on [ghcr.io]. We also push the built Docker images to ghcr, so that they can be reused by other tools (rustup) or by developers running the Docker build locally (to speed up their build). Since we test multiple, diverged branches (`master`, `beta` and `stable`), we can’t rely on a single cache for the images, otherwise builds on a branch would override the cache for the others. Instead, we store the images under different tags, identifying them with a custom hash made from the contents of all the Dockerfiles and related scripts. The CI calculates a hash key, so that the cache of a Docker image is invalidated if one of the following changes: - Dockerfile - Files copied into the Docker image in the Dockerfile - The architecture of the GitHub runner (x86 or ARM) ### LLVM caching with sccache We build some C/C++ stuff in various CI jobs, and we rely on [sccache] to cache the intermediate LLVM artifacts. Sccache is a distributed ccache developed by Mozilla, which can use an object storage bucket as the storage backend. With sccache there's no need to calculate the hash key ourselves. Sccache invalidates the cache automatically when it detects changes to relevant inputs, such as the source code, the version of the compiler, and important environment variables. So we just pass the sccache wrapper on top of cargo and sccache does the rest. We store the persistent artifacts on the S3 bucket `rust-lang-ci-sccache2`. So when the CI runs, if sccache sees that LLVM is being compiled with the same C/C++ compiler and the LLVM source code is the same, sccache retrieves the individual compiled translation units from S3. ## Custom tooling around CI During the years we developed some custom tooling to improve our CI experience. ### Rust Log Analyzer to show the error message in PRs

This section details caching strategies used to speed up CI processes: Docker image caching using Docker registry caching and LLVM caching with sccache. It also mentions the development of custom tools to improve the CI experience, specifically mentioning Rust Log Analyzer for displaying error messages in PRs.