Scaling BuildKit with Kubernetes and Node Assignment

| `qemu.image` | String | `tonistiigi/binfmt:latest` | Sets the QEMU emulation image. See [QEMU][4]. | ## Scaling BuildKit One of the main advantages of the Kubernetes driver is that you can scale the number of builder replicas up and down to handle increased build load. Scaling is configurable using the following driver options: - `replicas=N` This scales the number of BuildKit pods to the desired size. By default, it only creates a single pod. Increasing the number of replicas lets you take advantage of multiple nodes in your cluster. - `requests.cpu`, `requests.memory`, `requests.ephemeral-storage`, `limits.cpu`, `limits.memory`, `limits.ephemeral-storage` These options allow requesting and limiting the resources available to each BuildKit pod according to the official Kubernetes documentation [here](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). For example, to create 4 replica BuildKit pods: ```console $ docker buildx create \ --bootstrap \ --name=kube \ --driver=kubernetes \ --driver-opt=namespace=buildkit,replicas=4 ``` Listing the pods, you get this: ```console $ kubectl -n buildkit get deployments NAME READY UP-TO-DATE AVAILABLE AGE kube0 4/4 4 4 8s $ kubectl -n buildkit get pods NAME READY STATUS RESTARTS AGE kube0-6977cdcb75-48ld2 1/1 Running 0 8s kube0-6977cdcb75-rkc6b 1/1 Running 0 8s kube0-6977cdcb75-vb4ks 1/1 Running 0 8s kube0-6977cdcb75-z4fzs 1/1 Running 0 8s ``` Additionally, you can use the `loadbalance=(sticky|random)` option to control the load-balancing behavior when there are multiple replicas. `random` selects random nodes from the node pool, providing an even workload distribution across replicas. `sticky` (the default) attempts to connect the same build performed multiple times to the same node each time, ensuring better use of local cache. For more information on scalability, see the options for [`docker buildx create`](/reference/cli/docker/buildx/create.md#driver-opt). ## Node assignment The Kubernetes driver allows you to control the scheduling of BuildKit pods using the `nodeSelector` and `tolerations` driver options. You can also set the `schedulername` option if you want to use a custom scheduler altogether. You can use the `annotations` and `labels` driver options to apply additional metadata to the deployments and pods that's hosting your builders. The value of the `nodeSelector` parameter is a comma-separated string of key-value pairs, where the key is the node label and the value is the label text. For example: `"nodeselector=kubernetes.io/arch=arm64"` The `tolerations` parameter is a semicolon-separated list of taints. It accepts the same values as the Kubernetes manifest. Each `tolerations` entry specifies a taint key and the value, operator, or effect. For example: `"tolerations=key=foo,value=bar;key=foo2,operator=exists;key=foo3,effect=NoSchedule"` These options accept CSV-delimited strings as values. Due to quoting rules for shell commands, you must wrap the values in single quotes. You can even wrap all of `--driver-opt` in single quotes, for example: ```console $ docker buildx create \ --bootstrap \ --name=kube \ --driver=kubernetes \ '--driver-opt="nodeselector=label1=value1,label2=value2","tolerations=key=key1,value=value1"' ``` ## Multi-platform builds The Kubernetes driver has support for creating [multi-platform images](/manuals/build/building/multi-platform.md), either using QEMU or by leveraging the native architecture of nodes. ### QEMU Like the `docker-container` driver, the Kubernetes driver also supports using [QEMU](https://www.qemu.org/) (user mode) to build images for non-native platforms. Include the `--platform` flag and specify which platforms you want to output to.

This section explains how to scale BuildKit using the Kubernetes driver by adjusting the number of replicas. It also covers load balancing options (sticky/random) and demonstrates how to create multiple BuildKit pods. Furthermore, it details node assignment options using `nodeSelector`, `tolerations`, `annotations`, and `labels` to control pod scheduling and apply metadata. Finally, the section touches upon multi-platform builds using QEMU, enabling building images for non-native platforms.