Crate Loading and Pipelining in Rust Compilation

Crate Id](#stable-crate-id), and all CLI options marked with `[TRACKED]`). See [`compute_hir_hash`] for where the hash is actually computed. ### Stable Crate Id The [`StableCrateId`] is a 64-bit hash used to identify different crates with potentially the same name. It is a hash of the crate name and all the [`-C metadata`] CLI options computed in [`StableCrateId::new`]. It is used in a variety of places, such as symbol name mangling, crate loading, and much more. By default, all Rust symbols are mangled and incorporate the stable crate id. This allows multiple versions of the same crate to be included together. Cargo automatically generates `-C metadata` hashes based on a variety of factors, like the package version, source, and target kind (a lib and test can have the same crate name, so they need to be disambiguated). ## Crate loading Crate loading can have quite a few subtle complexities. During [name resolution], when an external crate is referenced (via an `extern crate` or path), the resolver uses the [`CrateLoader`] which is responsible for finding the crate libraries and loading the [metadata] for them. After the dependency is loaded, the `CrateLoader` will provide the information the resolver needs to perform its job (such as expanding macros, resolving paths, etc.). To load each external crate, the `CrateLoader` uses a [`CrateLocator`] to actually find the correct files for one specific crate. There is some great documentation in the [`locator`] module that goes into detail on how loading works, and I strongly suggest reading it to get the full picture. The location of a dependency can come from several different places. Direct dependencies are usually passed with `--extern` flags, and the loader can look at those directly. Direct dependencies often have references to their own dependencies, which need to be loaded, too. These are usually found by scanning the directories passed with the `-L` flag for any file whose metadata contains a matching crate name and [SVH](#strict-version-hash). The loader will also look at the [sysroot] to find dependencies. As crates are loaded, they are kept in the [`CStore`] with the crate metadata wrapped in the [`CrateMetadata`] struct. After resolution and expansion, the `CStore` will make its way into the [`GlobalCtxt`] for the rest of the compilation. ## Pipelining One trick to improve compile times is to start building a crate as soon as the metadata for its dependencies is available. For a library, there is no need to wait for the code generation of dependencies to finish. Cargo implements this technique by telling `rustc` to emit an [`rmeta`](#rmeta) file for each dependency as well as an [`rlib`](#rlib). As early as it can, `rustc` will save the `rmeta` file to disk before it continues to the code generation phase. The compiler sends a JSON message to let the build tool know that it can start building the next crate if possible. The [crate loading](#crate-loading) system is smart enough to know when it sees an `rmeta` file to use that if the `rlib` is not there (or has only been partially written). This pipelining isn't possible for binaries, because the linking phase will require the code generation of all its dependencies. In the future, it may be possible to further improve this scenario by splitting linking into a separate command (see [#64191]).

This section discusses the process of crate loading in Rust, where the `CrateLoader` finds crate libraries and loads metadata using the `CrateLocator`. Dependencies are located via `--extern` flags or by scanning directories specified with the `-L` flag, looking for matching crate names and SVHs. Loaded crates are stored in the `CStore` with metadata wrapped in `CrateMetadata`. To improve compile times, Rust uses pipelining. By emitting an `rmeta` file for each dependency, `rustc` can start building crates as soon as metadata is available. This is not possible for binaries, as the linking phase requires code generation of all dependencies.