BTRFS Write Operations and Performance Considerations with Docker

Consider three scenarios where a container opens a file for write access with Btrfs. #### Writing new files Writing a new file to a container invokes an allocate-on-demand operation to allocate new data block to the container's snapshot. The file is then written to this new space. The allocate-on-demand operation is native to all writes with Btrfs and is the same as writing new data to a subvolume. As a result, writing new files to a container's snapshot operates at native Btrfs speeds. #### Modifying existing files Updating an existing file in a container is a copy-on-write operation (redirect-on-write is the Btrfs terminology). The original data is read from the layer where the file currently exists, and only the modified blocks are written into the container's writable layer. Next, the Btrfs driver updates the filesystem metadata in the snapshot to point to this new data. This behavior incurs minor overhead. #### Deleting files or directories If a container deletes a file or directory that exists in a lower layer, Btrfs masks the existence of the file or directory in the lower layer. If a container creates a file and then deletes it, this operation is performed in the Btrfs filesystem itself and the space is reclaimed. ## Btrfs and Docker performance There are several factors that influence Docker's performance under the `btrfs` storage driver. > [!NOTE] > > Many of these factors are mitigated by using Docker volumes for write-heavy > workloads, rather than relying on storing data in the container's writable > layer. However, in the case of Btrfs, Docker volumes still suffer from these > draw-backs unless `/var/lib/docker/volumes/` isn't backed by Btrfs. ### Page caching Btrfs doesn't support page cache sharing. This means that each process accessing the same file copies the file into the Docker host's memory. As a result, the `btrfs` driver may not be the best choice for high-density use cases such as PaaS. ### Small writes Containers performing lots of small writes (this usage pattern matches what happens when you start and stop many containers in a short period of time, as well) can lead to poor use of Btrfs chunks. This can prematurely fill the Btrfs filesystem and lead to out-of-space conditions on your Docker host. Use `btrfs filesys show` to closely monitor the amount of free space on your Btrfs device.

This section details how Btrfs handles write operations within Docker containers, covering writing new files (allocate-on-demand), modifying existing files (copy-on-write), and deleting files/directories (masking or reclaiming space). It also addresses performance aspects such as page caching limitations, which can impact high-density use cases, and the effects of small writes on Btrfs chunk usage, which can lead to space issues. The use of Docker volumes is suggested for write-heavy workloads to mitigate some of these performance concerns, unless `/var/lib/docker/volumes/` isn't backed by Btrfs.