Catalog design

[dhiguero]

Following on the last community call items, the concept of having a catalog for OAM applications is an interesting topic that is being actively developed by some of the members. In order to improve the reusability of catalogs among different solutions, we believe that it will be good to approach its definition as a community proposal so that there exists a basic definition of its layout and purpose.

Note This issue has been edited to address the different comments and questions.

Changelog

• May, 12th, 2021
  • Use keywords instead of tags as tags may be confused with the image tags.
• Feb, 15th, 2021
  • Clarify purpose of the catalog
  • Extend support for external dependencies
  • Add Example
  • Rename kubernetes specific requirements in ApplicationMetadata.

Purpose

The purpose of the catalog is to provide a standarized entity to store OAM applications. Different platforms and implementations could then leverage one or more of such catalogs to provide an easy method for users to deploy OAM applications. Moreover, user can also decide to upload their applications to that catalog to improve reusability. It could be expected that there could be both private and public catalogs, but we believe that having a common structure will improve reusability and compatibility between different tools and vendors.

Examples of similar catalogs in purpose the are:

• Google Cloud Marketplace for K8s
• Helm Artifact Hub

Basic layout

We have approached the catalog trying to go for its most basic form so that an application can define all the elements that are needed considering those as (preferiably OAM) YAML entities. We think that the catalog should not contain other entities such as Dockerfiles or becoming a code repository.

Based on this assumptions, we are working on a catalog with the following layout:

/<application>
  |- metadata.yaml
  |- dependencies.yaml
  |- README.md
  |- app_cfg.yaml
  |
  |- <opt_directory>
  |  |- OAM_entity.yaml
  |  \- K8s_entity.yaml
  |- OAM_entity.yaml
  \- K8s_entity.yaml

Where the YAML files are OAM entities plus basic K8s entities that follow the current OAM suggestions/workarounds for items such as secrets and configmaps. The metadata.yaml file contains information about the application as whole to facilitate its discovery on the catalog. The README.md provides a textual description on how to use the application as found in other systems such as a GitHub repo, or the Docker Hub readme for an image. Directory structure is left for the application designer/developer to decide as the catalog engine has enough information (YAML headers containing kind and apiVersion) to identify entities.

metatadata.yaml

In terms of the metadata file we believe its purpose is to facilitate the discovery on the catalog by means of the different tools that may appear for exploring/integrating catalogs in OAM environments. Other similar entities are both defined on APIServer + Catalog Architecture Design and Application metadata descriptor CRD and share similar attributes.

apiVersion: core.oam.dev/XYZ
kind: ApplicationMetadata
# Name of the application, not necessarily a valid k8s name.
name: "My App Name"
version: 1.0
description: Short description for searchs. Long one plus how to goes into the README.md
# Keywords facilitate searches on the catalog
keywords:
  - "keyword1"
  - "keyword2"
  - "keyword3" 
license: "Apache License Version 2.0"
url: "https://..."
doc: "https://..."
# Requires gives a list of entities that are needed to launch the application.
requires:
  traits:
    - my.custom.trait
  scopes:
    - my.custom.scope
  # K8s lists Kubernetes specific entities. This provides a separation between OAM entities in an orchestration-agnostic environment, and applications that specifically require Kubernetes.
  k8s:
    - apiVersion: my.custom.package
      kind: CustomEntityKind
      name: name
# The logo can be used as visual information when listing the catalog so the user recognizes more easily the application.
logo:
  - src: "https://my.domain/path/logo.png"
    type: "image/png"
    size: "120x120"

Some differences with the aforementioned proposals are:

• Labels vs keywords: We think that keywords are more simpler to use for catalog searches.
• Emails: While some of the approaches include contact information (emails) on the metadata, we believe having two URLs for going to the main web page, plus the documentation and having a README.md is enough to point the user into contact methods depending on their needs.
• Requirements: As traits, scopes or in general other CRD an application may need have a collateral impact or requiring installing the appropriate operators, we think that is important to facilitate identifying whether an application can be deployed in a given cluster or a new element must be installed at cluster level. Installing traits and scope is another topic that may need further discussion as it has many collateral impacts in a multi-tenant cluster.

dependencies.yaml

The dependencies file lists dependencies of an application. In this context, a dependency could be any subset of YAML entities that are required to deploy the application. Optionally, dependencies may specify which entities they provide enabling complex cases where an application requires a particular trait, and the dependency defines where to get that trait from. As Traits and Scopes are not namespaced, the catalog consumer will decide how to resolve conflicts. Other methods to define dependencies as reference are: Helm dependencies, Kubevela Catalog Structure.

apiVersion: core.oam.dev/XYZ
kind: ApplicationDependencies
name: "My App Name"
dependencies:
- name: "dependency name"
  # PackageType defines how the entity is distributed so that it can be easily installed if needed.
  packageType: chart
  # Version is optional and could be use to retrieve specific versions of a given dependency
  version: "1.2.3"
  repository: "https://example.com/charts"
  # Provides is optional and can provide more information in situations where the dependency is related with cluster level components such as new Traits, or operators.
  provides:
    - apiVersion: my.custom.package
      kind: EntityKind
      name: name

Example entry

As an example of how an application may look like on this type of catalog, let's consider the containerized-workload example from the runtime.

/containerized-workload
  |- metadata.yaml
  |- README.md
  |- sample_application_config.yaml
  |- sample_component.yaml
  |- sample_scope_definition.yaml
  \- sample_secret.yaml

where metadata.yaml will contain:

apiVersion: core.oam.dev/XYZ
kind: ApplicationMetadata
name: "Containerized Workload"
version: 1.0
description: An example of a containerized workload using wordpress
keywords:
  - "example"
  - "wordpress"
license: "Apache License Version 2.0"
url: "https://github.com/crossplane/oam-kubernetes-runtime"
doc: "https://github.com/crossplane/oam-kubernetes-runtime"
# We are using traits and scope to illustrate the requirements even if those are by default on the runtime.
requires:
  traits:
    - manualscalertraits.core.oam.dev
  scopes:
    - healthscope.core.oam.dev
logo:
  - src: "https://my.domain/path/logo.png"
    type: "image/png"
    size: "120x120"

[resouer]

@dhiguero It's great to see this idea!

A quick question is what will be the main purpose of this catalog? Is it for GitOps layout to deploy the app? Or just for app store display?

If the purpose is for app store, any reason we do not reuse Helm to package the OAM yamls?

[hongchaodeng]

@dhiguero Could you provide more examples? These files seem too vague in high level: app_cfg.yaml, OAM_entity.yaml, k8s_entity.yaml

[dhiguero]

Thanks @resouer , @hongchaodeng for the comments. We have been working on an updated proposal and I have updated the issue to reflect the changes.

[h7kanna]

Can we use https://backstage.io/docs/features/software-catalog/software-catalog-overview for the Catalog?

[h7kanna]

Backstage + Kubevela will be super combo.
Backstage also has form builder based on reactjs forms and jsonschema
https://backstage.io/docs/features/software-templates/writing-templates#specparameters---formstep--formstep

[hongchaodeng]

Hi @h7kanna
We are already building a catalog UI + design for KubeVela. See the design and example and UI

We have looked at Backstage before. It looks awesome. But it doesn't match our requirements well so we have to build our own.

[wonderflow]

@h7kanna Hi, we've already build some combination for KubeVela and backstage, have a try and give some feed back. https://github.com/kubevela-contrib/backstage-plugin-kubevela

Backstage + Kubevela will be super combo. Backstage also has form builder based on reactjs forms and jsonschema https://backstage.io/docs/features/software-templates/writing-templates#specparameters---formstep--formstep

[wonderflow]

After more than two years evolution, I think KubeVela has become mature, I think it's time for KubeVela to have an application registry and discuss this issue deeper. \cc @dhiguero @barnettZQG

[dhiguero]

I think it is also a good time to start discussing the catalog/application registry again. Given that there have been some on and off discussions over this time, I would suggest to create a shared document where we can start discussing high-level requirements before going into the implementation discussions. I can start a document and shared it, but I do not know if you prefer to have the document hosted in the OAM/Kubevela gmail account.

[wonderflow]

@dhiguero Thanks Daniel, I think it's fine to create a document and share it. We can finally merge that content inside the repo here after discussion.

[dhiguero]

Ok, I will create an initial draft so we can jump-start the discussion.

[dhiguero]

Hi all, I have created an initial document to facilitate the discussion.

https://docs.google.com/document/d/1Acvl2DQYKj0R0aQMEb4czDfsE6oYTPslgbN8P4ZsQfM/edit?usp=sharing

[zt810950783]

邮件已收到，谢谢

[dh20156]

Great proposal, which aligns with our idea of choosing OAM and also expanding OAM. We are Workflos, and in the user stories we have encountered, the flow of applications is a very important scenario.

It involves a complete definition of an application, such as: the basic metadata information of the application, such as the name, version, industry domain, developer, etc., the open capability information of the application, such as which OpenAPIs, Events, etc. are included, the pricing and specification information of the application, such as the developer version, which available features and resource usage quotas, whether it is free or at what price, the authentication integration information of the application, such as which authentication and authorization protocols the application supports, and the deployment information of the application, such as whether the application supports SaaS or private deployment, and which components the private deployment depends on, etc.

As an open application model, OAM should not only define the basic deployment information model, but also support other nodes in the application flow lifecycle. We are looking forward to contributing to OAM better together.

[dhiguero]

Thanks everyone for the comments. I will update the document to reflect the discussion from the community call and the different questions and proposals raised in the document.