CI services usually test the last commit of a branch merged with the last commit
in `master`, and while that’s great to check if the feature works in isolation,
it doesn’t provide any guarantee the code is going to work once it’s merged.
Breakages like these usually happen when another, incompatible PR is merged
after the build happened.

To ensure a `master` branch that works all the time, we forbid manual merges.
Instead, all PRs have to be approved through our bot, [bors] (the software
behind it is called [homu]). All the approved PRs are put in a [merge queue]
(sorted by priority and creation date) and are automatically tested one at the
time. If all the builders are green, the PR is merged, otherwise the failure is
recorded and the PR will have to be re-approved again.

Bors doesn’t interact with CI services directly, but it works by pushing the
merge commit it wants to test to specific branches (like `auto` or `try`), which
are configured to execute CI checks. Bors then detects the outcome of the build
by listening for either Commit Statuses or Check Runs. Since the merge commit is
based on the latest `master` and only one can be tested at the same time, when
the results are green, `master` is fast-forwarded to that merge commit.

Unfortunately testing a single PR at the time, combined with our long CI (~2
hours for a full run), means we can’t merge too many PRs in a single day, and a
single failure greatly impacts our throughput for the day. The maximum number of
PRs we can merge in a day is around ~10.

The large CI run times and requirement for a large builder pool is largely due
to the fact that full release artifacts are built in the `dist-` builders. This
is worth it because these release artifacts:

- Allow perf testing even at a later date.
- Allow bisection when bugs are discovered later.
- Ensure release quality since if we're always releasing, we can catch problems
  early.

### Rollups

Some PRs don’t need the full test suite to be executed: trivial changes like
typo fixes or README improvements *shouldn’t* break the build, and testing every
single one of them for 2+ hours is a big waste of time. To solve this, we
regularly create a "rollup", a PR where we merge several pending trivial PRs so
they can be tested together. Rollups are created manually by a team member using
the "create a rollup" button on the [merge queue]. The team member uses their
judgment to decide if a PR is risky or not, and are the best tool we have at the
moment to keep the queue in a manageable state.

## Docker

All CI jobs, except those on macOS and Windows, are executed inside that
platform’s custom [Docker container]. This has a lot of advantages for us:

- The build environment is consistent regardless of the changes of the
  underlying image (switching from the trusty image to xenial was painless for
  us).
- We can use ancient build environments to ensure maximum binary compatibility,
  for example [using older CentOS releases][dist-x86_64-linux] on our Linux
  builders.
- We can avoid reinstalling tools (like QEMU or the Android emulator) every time
  thanks to Docker image caching.
- Users can run the same tests in the same environment locally by just running
  `cargo run --manifest-path src/ci/citool/Cargo.toml run-local <job-name>`, which is awesome to debug failures. Note that there are only linux docker images available locally due to licensing and
  other restrictions.

The docker images prefixed with `dist-` are used for building artifacts while
those without that prefix run tests and checks.

We also run tests for less common architectures (mainly Tier 2 and Tier 3
platforms) in CI. Since those platforms are not x86 we either run everything
inside QEMU or just cross-compile if we don’t want to run the tests for that
platform.

These builders are running on a special pool of builders set up and maintained
for us by GitHub.


## Caching

Our CI workflow uses various caching mechanisms, mainly for two things:

### Docker images caching