Support non-registry mount sources

[ktock]

Stargz snapshotter should support more mount sources other than registry.
They include the following (discussed at moby/buildkit#2685 (comment)).

• local OCI directory
• BuildKit Github Actions cache

[qi0523]

I want to implement P2P lazy pulling with containerd and stargz-snapshotter. Each node deploys a agent listening to 5001, which implements p2p distribution.

When i use ctr-remote image rpull docker.io/qi0523/ubuntu:latest to pull image lazily. I find that index.json, manifest.json, config.json can be downloaded from docker hub. The layer.tar doesn't exist in /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

However, when i use ctr-remote image rpull 192.168.56.101:5001/qi0523/ubuntu:latest to pull image lazily. I find that layer.tar also needs to be downloaded from agent.

I find that it starts to pull layer.tar after this line is executed.

[ktock]

@qi0523 Could you provide a reproducible step and the log for that issue?
There are several docs for IPFS + stargz-snapshotter. Have you followed some of these steps?

• https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/ipfs.md
• https://github.com/containerd/nerdctl/tree/v1.0.0/examples/nerdctl-ipfs-registry-kubernetes/ipfs-stargz-snapshotter

[qi0523]

@qi0523 Could you provide a reproducible step and the log for that issue? There are several docs for IPFS + stargz-snapshotter. Have you followed some of these steps?

• https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/ipfs.md
• https://github.com/containerd/nerdctl/tree/v1.0.0/examples/nerdctl-ipfs-registry-kubernetes/ipfs-stargz-snapshotter

Thanks.

The agent is a simple component implemented by myself.
It uses the image data under the directory(/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256) to serve HEAD and GET requests, such as http://xxx/v2/xxx/manifests/xxx and http://xxx/v2/xxx/blobs/xxx

[ktock]

@qi0523 So you're implementing your own registry API, right? Does that API fully implements OCI Distribution Spec? For successful lazy pulling, it needs to implement HTTP Range Request for blobs API at least.

Could you provide the log printed by stargz snapshotter (containerd-stargz-grpc) so that we can deeply investigate the issue?

You can also find the registry client logic at https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/fs/remote/resolver.go

[qi0523]

@qi0523 So you're implementing your own registry API, right? Does that API fully implements OCI Distribution Spec? For successful lazy pulling, it needs to implement HTTP Range Request for blobs API at least.

Could you provide the log printed by stargz snapshotter (containerd-stargz-grpc) so that we can deeply investigate the issue?

You can also find the registry client logic at https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/fs/remote/resolver.go

Yes, i read the code of distribution/distribution and use http.ServeContent(w, r, dgst.String(), time.Time{}, br) to implement HTTP Range Request for blobs API.

The logs show that stargz snapshotter can not extract sha256:73d944bd82

[ktock]

@qi0523 The error logs look truncated. Could you copy them and post here as text?

[qi0523]

Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"info","msg":"preparing filesy
stem mount at mountpoint=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs","parent":"","time":"2022-10-25T08:13:54.693558513Z"}
Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib
/containerd-stargz-grpc/snapshotter/snapshots/4/fs","msg":"resolving","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6
","time":"2022-10-25T08:13:54.703000767Z"}
Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"digest":"sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","error":null,"key":"default/6/extract-643806365-5X68 sha256:73d944bd
82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs","msg":"using default handler","parent":"","ref":"192.
168.56.106:5001/qi0523/ubuntu:latest","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","time":"2022-10-25T08:13:54.706477638Z"}
Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"error":"Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv
er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd
-stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t
ime":"2022-10-25T08:13:54.716664705Z"}
Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"error":"Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv
er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd
-stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t
ime":"2022-10-25T08:13:54.778420021Z"}
Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"error":"Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv
er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd
-stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t
ime":"2022-10-25T08:13:54.899760506Z"}
Oct 25 08:13:55 xv6 containerd-stargz-grpc[645]: {"error":"Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv
er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd
-stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t
ime":"2022-10-25T08:13:55.037032914Z"}
Oct 25 08:13:55 xv6 containerd-stargz-grpc[645]: {"error":"Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv
er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd
-stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t
ime":"2022-10-25T08:13:55.518904153Z"}
Oct 25 08:13:56 xv6 containerd-stargz-grpc[645]: {"error":"Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv
er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd
-stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t
ime":"2022-10-25T08:13:56.344004361Z"}
Oct 25 08:13:56 xv6 containerd-stargz-grpc[645]: {"error":"failed to resolve layer "sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6" from "192.168.56.106:5001/qi0523/ubuntu:l
atest": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host "192.168.56.106:5001", ref:"192.168.56.106:5001/qi0523/ubuntu:latest", digest:"sha
256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6"): failed to request: GET https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88
c3a3afb27c60939c6 giving up after 6 attempt(s): Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: server gave HTT
P response to HTTPS client: failed to resolve: failed to resolve target","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug"
,"mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs","msg":"failed to resolve layer","parent":"","time":"2022-10-25T08:13:56.347004192Z"}
Oct 25 08:13:56 xv6 containerd-stargz-grpc[645]: {"error":"failed to resolve layer: failed to resolve layer "sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6" from "192.168.56
.106:5001/qi0523/ubuntu:latest": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host "192.168.56.106:5001", ref:"192.168.56.106:5001/qi0523/ubun
tu:latest", digest:"sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6"): failed to request: GET https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb
23d447c74ff234d01da7b6d88c3a3afb27c60939c6 giving up after 6 attempt(s): Get "https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6
": http: server gave HTTP response to HTTPS client: failed to resolve: failed to resolve target","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c26815010
7fab2058","level":"warning","msg":"failed to prepare remote snapshot","parent":"","remote-snapshot-prepared":"false","time":"2022-10-25T08:13:56.348244797Z"}

[ktock]

@qi0523

http: server gave HTTP response to HTTPS client

The error message says the server gave HTTP response to HTTPS client. So could you configure stargz snapshotter (/etc/containerd-stargz-grpc/config.toml) to use plain HTTP for connecting to your registry?

https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/overview.md#registry-mirrors-and-insecure-connection

[qi0523]

@qi0523

http: server gave HTTP response to HTTPS client

The error message says the server gave HTTP response to HTTPS client. So could you configure stargz snapshotter (/etc/containerd-stargz-grpc/config.toml) to use plain HTTP for connecting to your registry?

https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/overview.md#registry-mirrors-and-insecure-connection

Thanks.

Another question is that when we use stargz-snaphotter, layer.tar doesn't exist in directory /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

The data of /var/lib/containerd-stargz-grpc/ can be used to serve blob GET request?

[ktock]

@qi0523

Another question is that when we use stargz-snaphotter, layer.tar doesn't exist in directory /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

This is because of lazy pulling. containerd doesn't pull layer blobs but stargz-snapshotter lazily pulls chunks of layer blobs.

The data of /var/lib/containerd-stargz-grpc/ can be used to serve blob GET request?

Yes. Blobs are stored under that directory and they are chunked. Note that some chunks of blobs can be lack until stargz-snapshotter completes downloading the entire blob contents to the node.

The followings are some of packages managing them:

• https://github.com/containerd/stargz-snapshotter/tree/v0.12.1/cache
• https://github.com/containerd/stargz-snapshotter/tree/v0.12.1/fs/reader
• https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/fs/layer/layer.go

[qi0523]

@qi0523

Another question is that when we use stargz-snaphotter, layer.tar doesn't exist in directory /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

This is because of lazy pulling. containerd doesn't pull layer blobs but stargz-snapshotter lazily pulls chunks of layer blobs.

The data of /var/lib/containerd-stargz-grpc/ can be used to serve blob GET request?

Yes. Blobs are stored under that directory and they are chunked. Note that some chunks of blobs can be lack until stargz-snapshotter completes downloading the entire blob contents to the node.

The followings are some of packages managing them:

• https://github.com/containerd/stargz-snapshotter/tree/v0.12.1/cache
• https://github.com/containerd/stargz-snapshotter/tree/v0.12.1/fs/reader
• https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/fs/layer/layer.go

Hi, i try to use the data of /var/lib/containerd-stargz-grpc/ to serve blob GET request.
Generally, the URL of GET request is http://xxxxx/blobs/sha256:xxx. So, we can only know the digest of the blob.
Although the above packages can manage the stargz layer, the mapping of blobs‘ digest and stargz layer is not maintained. How can i find out stargz layer by using the digest of the blob(sha256:xxx)?