Images pushed to private repository not fetched lazily

[dmosdallas]

Background

I've installed the stargz snapshotter for docker following the instructions provided here.

I'm able to run one of the listed pre-converted images, doing the following:
$ docker run -it ghcr.io/stargz-containers/tomcat:10.1.0-jdk17-openjdk-bullseye-esgz /bin/bash

Running mount while the container is alive shows:

$ mount | grep 'stargz on'
stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/73/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/74/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
...

So, I believe the snapshotter is configured correctly and is lazily-loading this pre-built image

Issue

I'm attempting to build a new eStargz image with BuildKit, following the instructions here, I run:

docker buildx build -t $IMAGE_NAME \
    -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true \
    .

This builds & pushes to my registry successfully, but when attempt to run a container using this image, it appears to fallback to performing a full pull of the image (the entire image is downloaded locally & no fuse mounts are configured).

When I check the stargz-snapshotter logs, I see a bunch of:

ce: cannot resolve layer: failed to redirect (host \"$projectId.dkr.ecr.us-east-1.amazonaws.com\", ref:\"$projectId.dkr.ecr.us-east-1.amazonaws.com/$repo:$tag\", digest:\"sha256:43c55c8ef73d40846c5241f397f568f30bdb9021fbe8dcdf4ad>
43b0474f21309a206c23fdb620a0fee9937393cacfbe7fd","level":"info","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/91/fs","msg":"Received status code: 401 Unauthorized. Refreshing creds...","parent":"moby/133/sha256:77da24f33555fe3aeb4>
9e32bf27ab6635a13e0ad9daa8475e3c67bfc6e0b6d\" from \"$projectId.dkr.ecr.us-east-1.amazonaws.com/$repo:tag\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"$projectId.d>
256:a9ce33851abb469b59e609e32bf27ab6635a13e0ad9daa8475e3c67bfc6e0b6d\" from \"$projectId.dkr.ecr.us-east-1.amazonaws.com/$repo:$tag\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redire>
750f46f36dfc8ab93276958b","level":"debug","msg":"checking layer availability","time":"2023-09-19T15:25:09.921893818Z"}
750f46f36dfc8ab93276958b","level":"debug","mount-point":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/90/fs","msg":"layer is normal snapshot(overlayfs)","time":"2023-09-19T15:25:09.921998628Z"}

These 401s seem suspicious, but reading the overview doc, it claims the stargz snapshotter should search for credentials in ~/.docker/config.json, which contains the credentials for this registry.

I'm now unsure if this is an issue:

• Building the image or
• Fetching the image from my private repo

I've now verified that this does work when pushing/pulling to a public repo, which now confuses as it seems to contridict the auth section of the overview documentation.

Additional details

• I'm running docker installed from the official apt repo, at Docker version 24.0.6, build ed223bc
• I'm pushing these images to a private ECR repo on AWS

[dmosdallas]

And... after writing this out, I think this may be a dupe of #1107

[dmosdallas]

Reading closer, I'm not getting any of the gzip errors that #1107 had (e.g., gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header).

So I think this may still be a different issue.

[dmosdallas]

Okay, well... that was an easy experiment. I created a new public repo (still with AWS ECR) and pushed to that and lazy pulling worked without a hitch. So now the question is how to get the stargz-snapshotter to authenticate with a private repo.

[ktock]

@dmosdallas If you run stargz-snapshotter with the different user than your docker command (e.g. running stargz-snapshotter as root and docker as an unprivileged user), stargz-snapshotter maybe refer to different ~/.docker/config.json (e.g. /root/.docker/config.json). So maybe you need to explicitly add $DOCKER_CONFIG envvar to stargz-snapshotter.service.

[dmosdallas]

@ktock, yep, that makes sense to me. Let me verify that fixes things up and I'll close out the issue. Thanks for the quick response!

[dmosdallas]

And that did the trick!

To summarize, the startz-snapshotter is a systemd daemon which was started running as root. I authenticated with my private repo as a non-root user and somehow expected it to be able to share those credentials. Kinda obvious that that wouldn't work in hindsight.

For anyone coming across this in the future, I edited the stargz-snapshotter unit file (/etc/systemd/system/stargz-snapshotter.service) to look like this:

[Unit]
Description=stargz snapshotter
After=network.target
Before=containerd.service

[Service]
Type=notify
Environment=HOME=/root
Environment=DOCKER_CONFIG=/home/ubuntu/.docker
ExecStart=/usr/local/bin/containerd-stargz-grpc --log-level=debug --config=/etc/containerd-stargz-grpc/config.toml
Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target

and then ran:

sudo systemctl daemon-reload
sudo systemctl restart stargz-snapshotter

To pick up the changes