docker pull library/nginx:latest is pulling a 3 month old release instead of most current

[cah-michael-mcghee]

Describe the bug

I have attempted to perform a docker pull nginx:latest and have even tried nginx: on several different types of systems (Ubuntu (WSL), RaspberryPi, Windows/Docker Desktop). All of them pull an older version of nginx then the latest. Performing docker images show created 3 months ago and docker inspect show a different digest. Have tried nginx:stable and that one says created 9 months ago.

other docker pull of different repo appear to pull the correct release.

docker releases are 24.x and higher

[yosifkit]

The images are actually very new (built yesterday because of the Debian base image update: docker-library/official-images#16155).

To work toward more reproducible layers/images, the Docker Official Images builds are moving to be built with SOURCE_DATE_EPOCH set to the commit timestamp of the respective Dockerfile+context. This is where the old dates are coming from. You can see it has been built recently in a few ways.

• By looking at the org.opencontainers.image.created annotation in the manifest: https://explore.ggcr.dev/?image=nginx%3Alatest. This timestamp is saved just before the docker build happens.

• by inspecting the layers of it and its parent image (debian:bookworm-slim) and compare their timestamps. Condensed output showing their created date and the new debian layer diff_id in the layer diff_ids of the "old" nginx image:

$ docker buildx imagetools inspect debian:bookworm-slim --format '{{ json (index .Image "linux/amd64")}}'
{
  "created": "2024-01-31T22:35:18.796241083Z",
...
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:fb1bd2fc52827db4ce719cc1aafd4a035d68bc71183b3bc39014f23e9e5fa256"
    ]
...
$ docker buildx imagetools inspect nginx:latest --format '{{ json (index .Image "linux/amd64")}}'
{
  "created": "2023-10-24T22:44:45Z",
...
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:fb1bd2fc52827db4ce719cc1aafd4a035d68bc71183b3bc39014f23e9e5fa256",
      "sha256:83bdf27d9eaa4f0e8f11b95dc952e9ab92619792dbd3d72f775d54c67eae1248",
      "sha256:84e0c9ef07d70aff145d02f297c94d7d5cc0ff7d87449c20eac02f5408a48fd0",
      "sha256:06536efc503aa0d192e8a5232eb57cad57fce55a8a92dce48d45eaf284f0cc26",
      "sha256:9f21a390e3f6d9e5af2d02fdb6a622eded91c58c9fe82af76204ad4cd854da39",
      "sha256:2b28485849eadbf9f3066df698b1579f169a7a306ce754a9809349099a58d11d",
      "sha256:f205d290cd763a4d8dddb06bee8fbd813cea79b179515fde9dec1b9fd9525856"
    ]
...

full `docker buildx imagetools` output:

$ docker buildx imagetools inspect debian:bookworm-slim --format '{{ json (index .Image "linux/amd64")}}'
{
  "created": "2024-01-31T22:35:18.796241083Z",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "bash"
    ]
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:fb1bd2fc52827db4ce719cc1aafd4a035d68bc71183b3bc39014f23e9e5fa256"
    ]
  },
  "history": [
    {
      "created": "2024-01-31T22:35:18.499703981Z",
      "created_by": "/bin/sh -c #(nop) ADD file:af0f4e41d68b67ca88a1ce6297326159e18e27670d7bfc0bf5804a4e2b268cc8 in / "
    },
    {
      "created": "2024-01-31T22:35:18.796241083Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
      "empty_layer": true
    }
  ]
}
$ docker buildx imagetools inspect nginx:latest --format '{{ json (index .Image "linux/amd64")}}'
{
  "created": "2023-10-24T22:44:45Z",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "ExposedPorts": {
      "80/tcp": {}
    },
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "NGINX_VERSION=1.25.3",
      "NJS_VERSION=0.8.2",
      "PKG_RELEASE=1~bookworm"
    ],
    "Entrypoint": [
      "/docker-entrypoint.sh"
    ],
    "Cmd": [
      "nginx",
      "-g",
      "daemon off;"
    ],
    "Labels": {
      "maintainer": "NGINX Docker Maintainers \[email protected]\u003e"
    },
    "StopSignal": "SIGQUIT",
    "ArgsEscaped": true
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:fb1bd2fc52827db4ce719cc1aafd4a035d68bc71183b3bc39014f23e9e5fa256",
      "sha256:83bdf27d9eaa4f0e8f11b95dc952e9ab92619792dbd3d72f775d54c67eae1248",
      "sha256:84e0c9ef07d70aff145d02f297c94d7d5cc0ff7d87449c20eac02f5408a48fd0",
      "sha256:06536efc503aa0d192e8a5232eb57cad57fce55a8a92dce48d45eaf284f0cc26",
      "sha256:9f21a390e3f6d9e5af2d02fdb6a622eded91c58c9fe82af76204ad4cd854da39",
      "sha256:2b28485849eadbf9f3066df698b1579f169a7a306ce754a9809349099a58d11d",
      "sha256:f205d290cd763a4d8dddb06bee8fbd813cea79b179515fde9dec1b9fd9525856"
    ]
  },
  "history": [
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "/bin/sh -c #(nop) ADD file:af0f4e41d68b67ca88a1ce6297326159e18e27670d7bfc0bf5804a4e2b268cc8 in / "
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "LABEL maintainer=NGINX Docker Maintainers \[email protected]\u003e",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "ENV NGINX_VERSION=1.25.3",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "ENV NJS_VERSION=0.8.2",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "ENV PKG_RELEASE=1~bookworm",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "RUN /bin/sh -c set -x     \u0026\u0026 groupadd --system --gid 101 nginx     \u0026\u0026 useradd --system --gid nginx --no-create-home --home /nonexistent --comment \"nginx user\" --shell /bin/false --uid 101 nginx     \u0026\u0026 apt-get update     \u0026\u0026 apt-get install --no-install-recommends --no-install-suggests -y gnupg1 ca-certificates     \u0026\u0026     NGINX_GPGKEY=573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62;     NGINX_GPGKEY_PATH=/usr/share/keyrings/nginx-archive-keyring.gpg;     export GNUPGHOME=\"$(mktemp -d)\";     found='';     for server in         hkp://keyserver.ubuntu.com:80         pgp.mit.edu     ; do         echo \"Fetching GPG key $NGINX_GPGKEY from $server\";         gpg1 --keyserver \"$server\" --keyserver-options timeout=10 --recv-keys \"$NGINX_GPGKEY\" \u0026\u0026 found=yes \u0026\u0026 break;     done;     test -z \"$found\" \u0026\u0026 echo \u003e\u00262 \"error: failed to fetch GPG key $NGINX_GPGKEY\" \u0026\u0026 exit 1;     gpg1 --export \"$NGINX_GPGKEY\" \u003e \"$NGINX_GPGKEY_PATH\" ;     rm -rf \"$GNUPGHOME\";     apt-get remove --purge --auto-remove -y gnupg1 \u0026\u0026 rm -rf /var/lib/apt/lists/*     \u0026\u0026 dpkgArch=\"$(dpkg --print-architecture)\"     \u0026\u0026 nginxPackages=\"         nginx=${NGINX_VERSION}-${PKG_RELEASE}         nginx-module-xslt=${NGINX_VERSION}-${PKG_RELEASE}         nginx-module-geoip=${NGINX_VERSION}-${PKG_RELEASE}         nginx-module-image-filter=${NGINX_VERSION}-${PKG_RELEASE}         nginx-module-njs=${NGINX_VERSION}+${NJS_VERSION}-${PKG_RELEASE}     \"     \u0026\u0026 case \"$dpkgArch\" in         amd64|arm64)             echo \"deb [signed-by=$NGINX_GPGKEY_PATH] https://nginx.org/packages/mainline/debian/ bookworm nginx\" \u003e\u003e /etc/apt/sources.list.d/nginx.list             \u0026\u0026 apt-get update             ;;         *)             echo \"deb-src [signed-by=$NGINX_GPGKEY_PATH] https://nginx.org/packages/mainline/debian/ bookworm nginx\" \u003e\u003e /etc/apt/sources.list.d/nginx.list                         \u0026\u0026 tempDir=\"$(mktemp -d)\"             \u0026\u0026 chmod 777 \"$tempDir\"   \u0026\u0026 savedAptMark=\"$(apt-mark showmanual)\"                         \u0026\u0026 apt-get update             \u0026\u0026 apt-get build-dep -y $nginxPackages             \u0026\u0026 (                 cd \"$tempDir\"                 \u0026\u0026 DEB_BUILD_OPTIONS=\"nocheck parallel=$(nproc)\"                     apt-get source --compile $nginxPackages             )                         \u0026\u0026 apt-mark showmanual | xargs apt-mark auto \u003e /dev/null             \u0026\u0026 { [ -z \"$savedAptMark\" ] || apt-mark manual $savedAptMark; }                         \u0026\u0026 ls -lAFh \"$tempDir\"             \u0026\u0026 ( cd \"$tempDir\" \u0026\u0026 dpkg-scanpackages . \u003e Packages )             \u0026\u0026 grep '^Package: ' \"$tempDir/Packages\"             \u0026\u0026 echo \"deb [ trusted=yes ] file://$tempDir ./\" \u003e /etc/apt/sources.list.d/temp.list             \u0026\u0026 apt-get -o Acquire::GzipIndexes=false update             ;;     esac         \u0026\u0026 apt-get install --no-install-recommends --no-install-suggests -y                         $nginxPackages                         gettext-base                         curl     \u0026\u0026 apt-get remove --purge --auto-remove -y \u0026\u0026 rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx.list         \u0026\u0026 if [ -n \"$tempDir\" ]; then         apt-get purge -y --auto-remove         \u0026\u0026 rm -rf \"$tempDir\" /etc/apt/sources.list.d/temp.list;     fi     \u0026\u0026 ln -sf /dev/stdout /var/log/nginx/access.log     \u0026\u0026 ln -sf /dev/stderr /var/log/nginx/error.log     \u0026\u0026 mkdir /docker-entrypoint.d # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "COPY docker-entrypoint.sh / # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "COPY 10-listen-on-ipv6-by-default.sh /docker-entrypoint.d # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "COPY 15-local-resolvers.envsh /docker-entrypoint.d # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "COPY 20-envsubst-on-templates.sh /docker-entrypoint.d # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "COPY 30-tune-worker-processes.sh /docker-entrypoint.d # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "ENTRYPOINT [\"/docker-entrypoint.sh\"]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "EXPOSE map[80/tcp:{}]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "STOPSIGNAL SIGQUIT",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2023-10-24T22:44:45Z",
      "created_by": "CMD [\"nginx\" \"-g\" \"daemon off;\"]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    }
  ]
}

[cah-michael-mcghee]

I some what understand what you are say about the base being updated....in the case of nginx:stable date created shows 9 months ago, have validated the debian:bullseye-slim is the latest digest, but if I look at NJS version it shows 0.7.12 but the latest stable/debian/Dockerfile source show 0.8.0.

This make things very confusing when trying to make sure one has the latest version.

[thresheek]

Yeah, there's a discrepancy between what can be in the repository and what is there in the image. This happens because we pin specific commit hashes to build the images. See https://github.com/nginxinc/docker-nginx?tab=readme-ov-file#see-a-change-merged-here-that-doesnt-show-up-on-docker-hub-yet for more details of the workflow.