Support multi-arch image/artifact push #1053

sajayantony · 2023-08-08T00:47:41Z

What is the version of your ORAS CLI

1.0.0

What would you like to be added?

As artifacts become increasingly accepted, it would be useful for ORAS to support multi-arch artifacts as well.
There are already some standard conventions of tools like homebrew implementing this.

Multi-arch manifest below - https://explore.ggcr.dev/?image=ghcr.io%2Fhomebrew%2Fcore%2Foras:1.0.1

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d0e42e6f6d564a9d1e178e72b42ff12b7e2e3c36cd32b44b150f7465c171db41",
      "size": 1914,
      "platform": {
        "architecture": "arm64",
        "os": "darwin",
        "os.version": "macOS 12.6"
      },
      "annotations": {
        "org.opencontainers.image.ref.name": "1.0.1.arm64_monterey",
        "sh.brew.bottle.digest": "8690763d0a9d0be6574ee4cfdb8edbab302f232c8056bae2d0083f170719afed",
        "sh.brew.bottle.size": "3436422",
        "sh.brew.tab": "{\"homebrew_version\":\"4.1.4-6-ga8022e6\",\"changed_files\":[],\"source_modified_time\":1691075089,\"compiler\":\"clang\",\"runtime_dependencies\":[],\"arch\":\"arm64\",\"built_on\":{\"os\":\"Macintosh\",\"os_version\":\"macOS 12.6\",\"cpu_family\":\"dunno\",\"xcode\":\"14.2\",\"clt\":\"14.2.0.0.1.1668646533\",\"preferred_perl\":\"5.30\"}}"
      }
    },

Each binary is uploaded with the following manifest

 oras manifest get --pretty ghcr.io/homebrew/core/oras@sha256:d0e42e6f6d564a9d1e178e72b42ff12b7e2e3c36cd32b44b150f7465c171db41
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:0e7e738a321110bc395c82868cd93c6dbd0e4afa7b8931f08a6039ed59544249",
    "size": 222
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8690763d0a9d0be6574ee4cfdb8edbab302f232c8056bae2d0083f170719afed",
      "size": 3436422,
      "annotations": {
        "org.opencontainers.image.title": "oras--1.0.1.arm64_monterey.bottle.tar.gz"
      }
    }
  ],
  "annotations": {
    "com.github.package.type": "homebrew_bottle",
    "org.opencontainers.image.created": "2023-08-07",
    "org.opencontainers.image.description": "OCI Registry As Storage",
    "org.opencontainers.image.documentation": "https://formulae.brew.sh/formula/oras",
    "org.opencontainers.image.license": "Apache-2.0",
    "org.opencontainers.image.ref.name": "1.0.1.arm64_monterey",
    "org.opencontainers.image.revision": "7716dfd65ccfe0502f5acfc35b3e8c6aa4cc52b1",
    "org.opencontainers.image.source": "https://github.com/homebrew/homebrew-core/blob/7716dfd65ccfe0502f5acfc35b3e8c6aa4cc52b1/Formula/oras.rb",
    "org.opencontainers.image.title": "oras 1.0.1.arm64_monterey",
    "org.opencontainers.image.url": "https://github.com/oras-project/oras",
    "org.opencontainers.image.vendor": "homebrew",
    "org.opencontainers.image.version": "1.0.1",
    "sh.brew.bottle.digest": "8690763d0a9d0be6574ee4cfdb8edbab302f232c8056bae2d0083f170719afed",
    "sh.brew.bottle.size": "3436422",
    "sh.brew.tab": "{\"homebrew_version\":\"4.1.4-6-ga8022e6\",\"changed_files\":[],\"source_modified_time\":1691075089,\"compiler\":\"clang\",\"runtime_dependencies\":[],\"arch\":\"arm64\",\"built_on\":{\"os\":\"Macintosh\",\"os_version\":\"macOS 12.6\",\"cpu_family\":\"dunno\",\"xcode\":\"14.2\",\"clt\":\"14.2.0.0.1.1668646533\",\"preferred_perl\":\"5.30\"}}"
  }
}

Why is this needed for ORAS?

ORAS does not have CLI commands to author a manifest. Being able to author this multi arch manifest or index-based manifest would enable artifact authors to use ORAS CLI for scenarios like above.

Things to keep in mind

Multi-arch platform property is one way for clients to determine the artifact. This is already supported in ORAS throught the --platform flag.
Clients can choose to use annotations or any custom filter from index.
There could be an ability for clients to compose the manifest using a tag and add more artifacts in a multi step script and finally copy the full index and related manifests in one shot.

Composotion

sequenceDiagram
    participant User
    participant LocalRegistry as Local Store
    participant RemoteRegistry as Remote Registry

    User->>LocalRegistry: Begin creation of artifact with tag mybinary:v1 for amd64 linux
    Note right of LocalRegistry: Artifact: mybinary:v1 (linux/amd64) initiated

    User->>LocalRegistry: Add binary for a different platform (e.g. arm64)
    Note right of LocalRegistry: Artifact: mybinary:v1 (linux/amd64, linux/arm64) updated

    User->>LocalRegistry: Add binary with a different annotation
    Note right of LocalRegistry: Artifact: mybinary:v1 (linux/amd64, linux/arm64, annotations)

    User->>RemoteRegistry: Copy artifact mybinary:v1 from Local Registry
    Note right of RemoteRegistry: Artifact: mybinary:v1 (linux/amd64, linux/arm64, annotations)

Appending

Consider appending a remote manifest and add a new item to the index

sequenceDiagram
    participant User
    participant RemoteRegistry1 as Local Store
    participant RemoteRegistry2 as Remote Registry

    User->>RemoteRegistry1: Inspect artifact with tag mybinary:v1
    Note right of RemoteRegistry1: GET manifest details of mybinary:v1

    User->>User: Determine modifications without pulling full artifact

    User->>RemoteRegistry1: Add binary for a different platform (e.g. linux/arm64) directly
    Note right of RemoteRegistry1: Artifact: mybinary:v1 (linux/amd64, linux/arm64)  updated

    User->>RemoteRegistry: Copy artifact mybinary:v1 from Local Registry
    Note right of RemoteRegistry2: Artifact: mybinary:v1 (linux/amd64, linux/arm64) updated

Note

Currently Index implementations don't support artifactType in some registries and hence we should consider if tools need this even through it will be a part of the image spec.
Index support subject and we should consider how we will support subject properties to link and index to another artifact.

/cc @siggy

The text was updated successfully, but these errors were encountered:

shizhMSFT · 2023-08-08T03:44:03Z

/cc @FeynmanZhou @yizha1

FeynmanZhou · 2023-08-08T05:59:50Z

There is another request for attaching artifacts to a multi-arch image with ORAS raised by an ORAS user @SamirPS described in #813. Designing a full user story for multi-arch image distribution (pull, push, discover, attach) in ORAS sounds reasonable to me.

qweeah · 2023-08-10T07:49:36Z

There is another request for attaching artifacts to a multi-arch image with ORAS raised by an ORAS user @SamirPS described in #813. Designing a full user story for multi-arch image distribution (pull, push, discover, attach) in ORAS sounds reasonable to me.

#813 requires ability to find the correct subject, this issue is proposing ability of packing(composing or authoring) an index, which is different.

qweeah · 2023-08-10T08:20:42Z

Composing an index is easy and the composed index can be output to stdout for uploading like:

oras index my.reg/repo tag1 tag2 dgst2 --subject tag3 | oras manifest push -

Platform info can be easily obtained from the predecessor manifest's config, the tricky part is customizing descriptor annotations in manifests. e.g. we can found that the annotations of oras mac build doesn't all presented in the referring index.

I don't have any idea to accommodate the need for selectively add annotation into the index, I suggest we can introduce the index composing feature first.

qweeah · 2023-08-18T02:44:40Z

For composition scenario, oras should be able to let user specify platform info when it's not in the artifact.
oras push and oras attach should support --platform to allow user bake artifacts with platform metadata, see support --platform in oras push and oras attach #1066
For appending scenario, oras should not do garbage collection.

Currently OS and arch is stored in the tag however once oras CLI starts to support pushing multi-arch artifacts, we'll use that. Ref - oras-project/oras#1053 Signed-off-by: Rudraksh Pareek <[email protected]>

sajayantony · 2024-06-26T00:40:19Z

Adding 2 scripts to show the challenges
https://gist.github.com/sajayantony/0c9d6436c03d531b1bbebe43249381cf
and from @theory https://gist.github.com/theory/7dc164e5772cae652d838a1c508972ae#file-push_trunk

TerryHowe · 2024-06-26T22:31:50Z

Blog post https://justatheory.com/2024/06/trunk-oci-poc/

qweeah · 2024-07-24T09:27:38Z

This special case of composition scenario might need a index merge command:

The tool chain I used is listed as below

✗ docker version
Client: Docker Engine - Community
 Version:           27.1.0
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Fri Jul 19 17:43:11 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.1.0
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       a21b1a2
  Built:            Fri Jul 19 17:43:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.1.1
  GitCommit:        v1.1.0-20-g52de29d7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I run docker build and docker save and got below manifest:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f4c8fbb580a9de23437857b686a96394451e5bd66c464ba025d09c8aee3de9d6",
      "size": 665,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:cea0a71c4a8214c54e655c587607687a112ab1ed852a68add9863fbbb1b93774",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:f4c8fbb580a9de23437857b686a96394451e5bd66c464ba025d09c8aee3de9d6",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

Worth pointing out that even though I was trying to build a single arch image, still the builder(buildx) generates me an index, whose manifests field points to the image manifest and attestation manifest. The experience should apply to all users building from non-overlayfs image store (in my case, containerd image store)

To better help such user compose multi-arch image from those indexes. We can add a new command to accept index references and flatten the input into one index. Comparing to index create, the advantages are

Performance is better: as it requires less network request to fetch platform information from the config
Annotations added by building tools (like vnd.docker.reference.digest and vnd.docker.reference.type) will be kept.

Currently OS and arch is stored in the tag however once oras CLI starts to support pushing multi-arch artifacts, we'll use that. Ref - oras-project/oras#1053 Signed-off-by: Rudraksh Pareek <[email protected]> Signed-off-by: Navin Chandra <[email protected]>

sajayantony added the enhancement New feature or request label Aug 8, 2023

shizhMSFT added this to the future milestone Aug 8, 2023

sajayantony changed the title ~~Support multi-arch image push~~ Support multi-arch image/artifact push Aug 30, 2023

qweeah mentioned this issue Dec 18, 2023

oras pull should work against OCI layouts that have no tags but have no ambiguous manifests #1202

Closed

1 task

yizha1 modified the milestones: future, v1.3.0 Mar 4, 2024

FeynmanZhou added the spec required Issues that require specifications label Mar 14, 2024

DelusionalOptimist mentioned this issue May 23, 2024

chore(CI): publish KubeArmor tars to dockerhub kubearmor/KubeArmor#1764

Merged

7 tasks

shizhMSFT assigned FeynmanZhou and wangxiaoxuan273 Jul 9, 2024

wangxiaoxuan273 mentioned this issue Jul 22, 2024

feat: PoC for multi arch image create/update #1450

Closed

FeynmanZhou added the multi-arch label Jul 23, 2024

This was referenced Aug 12, 2024

feat: create index from existing manifests (index create) #1475

Merged

feat: update existing index (index update) #1476

Merged

wangxiaoxuan273 mentioned this issue Aug 29, 2024

feat: --output for oras manifest index create #1490

Merged

4 tasks

This was referenced Sep 13, 2024

feat: --annotation for index create #1499

Merged

feat: --output for oras manifest index update #1502

Merged

TerryHowe mentioned this issue Sep 30, 2024

feat: add platform option to push command #1500

Merged

4 tasks

FeynmanZhou linked a pull request Oct 15, 2024 that will close this issue

spec: Multi-arch image management with ORAS #1514

Open

Wwwsylvia mentioned this issue Oct 31, 2024

Identifying Platform-Specific OCI Artifacts opencontainers/image-spec#1216

Open

FeynmanZhou modified the milestones: v1.3.0, v1.3.0-beta.1 Nov 27, 2024

FeynmanZhou modified the milestones: v1.3.0-beta.1, v1.3.0-beta.2 Dec 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support multi-arch image/artifact push #1053

Support multi-arch image/artifact push #1053

sajayantony commented Aug 8, 2023 •

edited

Loading

shizhMSFT commented Aug 8, 2023

FeynmanZhou commented Aug 8, 2023

qweeah commented Aug 10, 2023 •

edited

Loading

qweeah commented Aug 10, 2023 •

edited

Loading

qweeah commented Aug 18, 2023 •

edited

Loading

sajayantony commented Jun 26, 2024

TerryHowe commented Jun 26, 2024

qweeah commented Jul 24, 2024 •

edited

Loading

Support multi-arch image/artifact push #1053

Support multi-arch image/artifact push #1053

Comments

sajayantony commented Aug 8, 2023 • edited Loading

What is the version of your ORAS CLI

What would you like to be added?

Why is this needed for ORAS?

Composotion

Appending

Note

shizhMSFT commented Aug 8, 2023

FeynmanZhou commented Aug 8, 2023

qweeah commented Aug 10, 2023 • edited Loading

qweeah commented Aug 10, 2023 • edited Loading

qweeah commented Aug 18, 2023 • edited Loading

sajayantony commented Jun 26, 2024

TerryHowe commented Jun 26, 2024

qweeah commented Jul 24, 2024 • edited Loading

sajayantony commented Aug 8, 2023 •

edited

Loading

qweeah commented Aug 10, 2023 •

edited

Loading

qweeah commented Aug 10, 2023 •

edited

Loading

qweeah commented Aug 18, 2023 •

edited

Loading

qweeah commented Jul 24, 2024 •

edited

Loading