Default GC settings can immediately discard work when working with large layers

[ag-TJNII]

Description

Yesterday I learned about the Build garbage collector, unfortunately because it was removing layers immediately after they were built.

Take the following Dockerfile, which is based on a unpolished, experimental Dockerfile I was iterating on:

FROM debian:trixie

RUN apt-get update
RUN apt-get install -y ca-certificates curl git git-lfs
RUN apt-get install -y python3 python3-pip

RUN pip3 install --break-system-packages torch

# Assume this is a real run command, but it has an error
RUN something with a mistake

Normally I'd expect the build to cache up to and including the pip3 torch install, so when I fix the error and re-run docker build it will not rebuild the torch layer, or any previous layers. However, due to layer size I found that the later cache was immediately being discarded. Note the layer sizes:

IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
6f0d75294b62   18 minutes ago   RUN /bin/sh -c pip3 install --break-system-p…   10.7GB    buildkit.dockerfile.v0
<missing>      15 hours ago     RUN /bin/sh -c apt-get install -y python3 py…   398MB     buildkit.dockerfile.v0
<missing>      20 hours ago     RUN /bin/sh -c apt-get install -y ca-certifi…   149MB     buildkit.dockerfile.v0
<missing>      20 hours ago     RUN /bin/sh -c apt-get update # buildkit        21.1MB    buildkit.dockerfile.v0
<missing>      9 days ago       # debian.sh --arch 'amd64' out/ 'trixie' '@1…   120MB     debuerreotype 0.16

As this build has a layer above the default 10GB cache size I noticed it was being GC'd immediately. I saw this behavior both with this big pip install, and another Dockerfile that was downloading a LLM model containing multiple 8GB safetensor files.

This 10GB threshold caught me by surprise. As I have relatively slow internet some of these layers took hours to download, having them immediately be discarded was frustrating. I think this behavior could use some improvement, such as:

• Do not enforce the build cache size, unless there is disk pressure.
• Do not discard layers completed within the last hour.

Basically, as a user, if there's no disk pressure I don't want to see the gc taking action against fresh cache layers. I had over 100GB free in /var/lib/docker when my layers were gc'd. These points are soft suggestions, I know each of the suggestion points will come with caveats. However, under certain circumstances the gc 10GB default threshold is a hidden sharp edge, so it would be nice to dull that.

[tonistiigi]

What is your GC configuration docker buildx inspect.

Recent BuildKit supports GC rules based on the minimum free space, which is recommended as the default starting point. https://docs.docker.com/build/cache/garbage-collection/

[ag-TJNII]

At the time I hit this I was running defaults for the gc. This was my Docker daemon.json at the time of the reported behavior:

{
    "max-concurrent-uploads": 1,
    "max-concurrent-downloads": 2,
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

I've since disabled the builder gc. This is current, as of this morning. I haven't performed any upgrades, only the config change to disable the gc:

$ docker buildx inspect
Name:          default
Driver:        docker
Last Activity: 2025-09-18 12:36:55 +0000 UTC

Nodes:
Name:             default
Endpoint:         default
Status:           running
BuildKit version: v0.24.0
Platforms:        linux/amd64, linux/amd64/v2, linux/386
Labels:
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1

This server was rebuilt this month so it should be quite current on versions. I'm running Docker CE on Debian sourced from deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable:

$ dpkg --list | grep -i docker
ii  docker-buildx-plugin               0.27.0-1~debian.13~trixie            amd64        Docker Buildx cli plugin.
ii  docker-ce                          5:28.4.0-1~debian.13~trixie          amd64        Docker: the open-source application container engine
ii  docker-ce-cli                      5:28.4.0-1~debian.13~trixie          amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras          5:28.4.0-1~debian.13~trixie          amd64        Rootless support for Docker.
ii  docker-compose-plugin              2.39.2-1~debian.13~trixie            amd64        Docker Compose (V2) plugin for the Docker CLI.

/var/lib/docker is on it's own partition, separate from /. Other than the daemon.json tuning I didn't perform any other Docker customization after installing Docker CE on a fresh OS install.

https://docs.docker.com/build/cache/garbage-collection/ says reservedSpace is "10% of total disk space or 10GB (whichever is lower)", so I think I was hitting the 10GB cap. The 10% calculation should work out to 25G, as my /var/lib/docker partition is 250G.

[tonistiigi]

says reservedSpace is "10% of total disk space or 10GB (whichever is lower)", so I think I was hitting the 10GB cap. The 10% calculation should work out to 25G, as my /var/lib/docker partition is 250G.

This is the minimum allowed space. If the other config variables allow bigger maximum space or free space condition isn't reached then you can use much more.