Headlamp Plugins CLI: Bulk installations support

[Faakhir30]

Refs #2787, first PR towards original propossed solution.

Scope and Tasks of this PR:

• Add parsing and validation of yaml config(Used ajv for now, would update if maintainers suggests so.)
• Add batch installation module (reuses PluginManager for installation, URL and version validations ...)
• Install specific versions pinned, else latest
• Add tests for batch installation module
• Update CLI arguments to support both yaml config file files and direct artifact hub source URLs
• Update Docs (document the plugin yml config)
• Add concurrency control for parallel installations
• add the ability to define DAG using dependencies relations in plugins

The plugin-level config option was removed for now, as it might cause unnecessary confusion, though saved the TODO-based implementation that I removed in git stash, can be redone if required.

---

Testing:

individual Plugin with URL:
bin/headlamp-plugin.js install --source https://artifacthub.io/packages/headlamp/test-123/appcatalog_headlamp_plugin --folderName tmp
Bulk installations:

• Create a file plugin-config.yml:

plugins:
  - name: test-app-catalog
    source: https://artifacthub.io/packages/headlamp/test-123/appcatalog_headlamp_plugin
    version: 0.0.3
  - name: ai-plugin
    source: https://artifacthub.io/packages/headlamp/test-123/ai_plugin
    version: 0.0.2
  - name: prometheus
    source: https://artifacthub.io/packages/headlamp/test-123/prometheus_headlamp_plugin
    version: 0.0.3
    dependencies:
      - test-app-catalog

installOptions:
  parallel: true
  maxConcurrent: 2

• Then try out: bin/headlamp-plugin.js install --config tmp/plugin-config.yml --folderName tmp Make sure tmp directory is present.

Unit Tests:
For jest based tests:
f@f:~/w/oss/headlamp/plugins/headlamp-plugin$ npx jest plugin-management/*

---

Would refine commit messages at end(once reviews and changes required implemented) to avoid force pushing commits again n again.

[knrt10]

The src/ directory we use is based on typescript for better type safety, consistency and better documentation.

So, I would request you to please address this and change the manager in ts as well to avoid inconsistency.

[knrt10]

the name of the folder plugin-management seems redundant. It can be something simpler 1 word like manager or something like that.

[knrt10]

Saw the previous files I think it should be fine to keep it in JS.

[knrt10]

Some suggestions

[knrt10]

we should have validation for version format, also there is no constraints on plugin names (could have spaces/special chars).

Maybe something like

# for name
pattern: '^[a-z0-9-]+$'  // Only allow lowercase, numbers, hyphens

# for version
pattern: '^\\d+\\.\\d+\\.\\d+' // Semver format

[Faakhir30]

For name, I've also allowed underscore (except at the start and end), because headlamp plugin names are using underscores somewhere IMO. e.g., https://artifacthub.io/packages/headlamp/test-123/ai_plugin.

So used ^[a-z0-9][a-z0-9-_]*[a-z0-9-]$

[Faakhir30]

I've implemented the above suggestions, thanks for the initial review.

Some additional things:

• Regarding Plugin sources validation, I've currently set them to https://artifacthub.io/packages/headlamp/*, as this validation is being used previously also, reference: https://github.com/headlamp-k8s/headlamp/blob/08e1e298989284ddc91bf4e6e82785c71657f132/plugins/headlamp-plugin/plugin-management/plugin-management.js#L449-L453
• I've updated the template in this PR's description to test concurrency, and dependency DAG type flow.
• Regarding improving logging further, I've used progressCallback to its potential current logs based on installations from the template in this PR's description are following, (any further suggestions for logging styling appreciated):

f@f:~/w/oss/lfx/headlamp/plugins/headlamp-plugin$ bin/headlamp-plugin.js install --config
 tmp/config/plugin.yml --folderName tmp
Installing plugins from config { configPath: 'tmp/config/plugin.yml' }
Installation chunks: [ [ 'test-app-catalog', 'ai-plugin' ], [ 'prometheus' ] ]
1 of 3 (test-app-catalog): info: Installing plugin test-app-catalog
1 of 3 (test-app-catalog): info: Fetching Plugin Metadata
2 of 3 (ai-plugin): info: Installing plugin ai-plugin
2 of 3 (ai-plugin): info: Fetching Plugin Metadata
2 of 3 (ai-plugin): info: Plugin Metadata Fetched
2 of 3 (ai-plugin): info: Downloading Plugin
1 of 3 (test-app-catalog): info: Plugin Metadata Fetched
1 of 3 (test-app-catalog): info: Downloading Plugin
2 of 3 (ai-plugin): info: Plugin Downloaded
1 of 3 (test-app-catalog): info: Plugin Downloaded
2 of 3 (ai-plugin): error: Checksum mismatch.
1 of 3 (test-app-catalog): info: Extracting Plugin
1 of 3 (test-app-catalog): info: Plugin Extracted
Moved directory from /tmp/headlamp-plugin-temp-WyoyJw/appcatalog_headlamp_plugin to tmp/appcatalog_headlamp_plugin
1 of 3 (test-app-catalog): success: Plugin Installed
1 of 3 (test-app-catalog): success: Plugin installed successfully
3 of 3 (prometheus): info: Installing plugin prometheus
3 of 3 (prometheus): info: Fetching Plugin Metadata
3 of 3 (prometheus): info: Plugin Metadata Fetched
3 of 3 (prometheus): info: Downloading Plugin
3 of 3 (prometheus): info: Plugin Downloaded
3 of 3 (prometheus): info: Extracting Plugin
3 of 3 (prometheus): info: Plugin Extracted
Moved directory from /tmp/headlamp-plugin-temp-tmM1w4/prometheus_headlamp_plugin to tmp/prometheus_headlamp_plugin
3 of 3 (prometheus): success: Plugin Installed
3 of 3 (prometheus): success: Plugin installed successfully
Bulk installation completed { total: 3, failed: 1, skipped: 0, successful: 2 }
f@f:~/w/oss/lfx/headlamp/plugins/headlamp-plugin$

• New considered scenarios:

  • should reject duplicate plugin names
  • should reject duplicate plugin sources
  • should detect circular dependencies
  • should skip plugins with failed/skipped dependencies
  • should respect the maxConcurrent setting
  • should not install dependent plugins in parallel, create optimal chunks within maxConcurrency size limit

• Regarding the folder name plugin-management, needed confirmation once more from your end cause that was originally plugin-management, can/should I update that to management?

[k8s-triage-robot]

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-contributor-experience at kubernetes/community.

/check-cla
/easycla

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Faakhir30
Once this PR has been reviewed and has the lgtm label, please assign joaquimrocha for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[chkpwd]

Will support for Github and other hosting sites be added or is ArtifactHub the defacto right now?