Gradle Setup Hook and Dependency Management

# Gradle Setup Hook ## Introduction Gradle build scripts are written in a DSL, computing the list of Gradle dependencies is a Turing-complete task, not just in theory but also in practice. Fetching all of the dependencies often requires building some native code, running some commands to check the host platform, or just fetching some files using either JVM code or commands like `curl` or `wget`. This practice is widespread and isn't considered a bad practice in the Java world, so all we can do is run Gradle to check what dependencies end up being fetched, and allow derivation authors to apply workarounds so they can run the code necessary for fetching the dependencies our script doesn't fetch. "Run Gradle to check what dependencies end up being fetched" isn't a straightforward task. For example, Gradle usually uses Maven repositories, which have features such as "snapshots", a way to always use the latest version of a dependency as opposed to a fixed version. Obviously, this is horrible for reproducibility. Additionally, Gradle doesn't offer a way to export the list of dependency URLs and hashes (it does in a way, but it's far from being complete, and as such is useless for Nixpkgs). Even if it did, it would be annoying to use considering fetching non-Gradle dependencies in Gradle scripts is commonplace. That's why the setup hook uses mitm-cache, a program designed for intercepting all HTTP requests, recording all the files that were accessed, creating a Nix derivation with all of them, and then allowing the Gradle derivation to access these files. ## Maven Repositories (Reference: [Repository Layout](https://cwiki.apache.org/confluence/display/MAVENOLD/Repository+Layout+-+Final)) Most of Gradle dependencies are fetched from Maven repositories. For each dependency, Gradle finds the first repo where it can successfully fetch that dependency, and uses that repo for it. Different repos might actually return different files for the same artifact because of e.g. pom normalization. Different repos may be used for the same artifact even across a single package (for example, if two build scripts define repositories in a different order). The artifact metadata is specified in a .pom file, and the artifacts themselves are typically .jar files. The URL format is as follows: `<repo>/<group-id>/<artifact-id>/<base-version>/<artifact-id>-<version>[-<classifier>].<ext>` For example: - `https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/2.0.9/slf4j-api-2.0.9.pom` - `https://oss.sonatype.org/content/groups/public/com/tobiasdiez/easybind/2.2.1-SNAPSHOT/easybind-2.2.1-20230117.075740-16.pom` Where: - `<repo>` is the repo base (`https://repo.maven.apache.org/maven2`) - `<group-id>` is the group ID with dots replaced with slashes (`org.slf4j` -> `org/slf4j`) - `<artifact-id>` is the artifact ID (`slf4j-api`) - `<base-version>` is the artifact version (`2.0.9` for normal artifacts, `2.2.1-SNAPSHOT` for snapshots) - `<version>` is the artifact version - can be either `<base-version>` or `<version-base>-<timestamp>-<build-num>` (`2.0.9` for normal artifacts, and either `2.2.1-SNAPSHOT` or `2.2.1-20230117.075740-16` for snapshots) - `<version-base>` - `<base-version>` without the `-SNAPSHOT` suffix - `<timestamp>` - artifact build timestamp in the `YYYYMMDD.HHMMSS` format (UTC) - `<build-num>` - a counter that's incremented by 1 for each new snapshot build - `<classifier>` is an optional classifier for allowing a single .pom to refer to multiple .jar files. .pom files don't have classifiers, as they describe metadata. - `<ext>` is the extension. .pom Note that the artifact ID can contain `-`, so you can't extract the artifact ID and version from just the file name. Additionally, the files in the repository may have associated signature files, formed by appending `.asc` to the filename, and hashsum files, formed by appending `.md5` or `.sha1` to the filename. The signatures are harmless, but the `.md5`/`.sha1` files are rejected.

This document introduces a 'Gradle setup hook' for managing and reproducing Gradle dependencies, which are often non-deterministic due to practices like 'snapshots' and arbitrary code execution. As Gradle lacks robust dependency export mechanisms, the hook utilizes `mitm-cache`. This tool intercepts HTTP requests during Gradle execution, records accessed files, creates a Nix derivation, and allows the Gradle derivation to use these cached files, ensuring reproducibility. The text then details Maven repository structure, from which most Gradle dependencies are fetched. It explains the URL format for artifacts, including group ID, artifact ID, base version, and specific `version` formats for standard and 'snapshot' artifacts (with timestamps and build numbers). It also notes optional classifiers, file extensions, and that `.asc` signature files are accepted while `.md5` and `.sha1` hashsum files are rejected.