Support custom certificate configuration while interacting with central repository

[prkalle]

What this PR does / why we need it

This PR adds support for custom certificate configuration for a given host.

• Also, this PR implement changes to use the custom certs for central repository(e.g air-gapped env where user has their own registry with self-signed)
• Add HTTPS support for test local central repository
• Add cosign signatures for test local central repository
• e2e tests now validates using the custom certs for test local central repo

Which issue(s) this PR fixes

Fixes #12, 133

Describe testing done for PR

• Updated the default central repo to test local central repo

❯ ./bin/tanzu plugin source update default -u localhost:9876/tanzu-cli/plugins/central:small
[ok] updated discovery source default

• running tanzu plugin search should show error that test central repo SSL cert is not signed by known authority

❯ ./bin/tanzu plugin search
[!] Unable to resolve the plugin discovery image: error getting the image digest: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://localhost:9876/v2/": x509: “localhost.local” certificate is not standards compliant; GET http://localhost:9876/v2/: unexpected status code 400 Bad Request: Client sent an HTTP request to an HTTPS server.
[x] Fatal: plugins discovery image resolution failed. Please check that the repository image URL "localhost:9876/tanzu-cli/plugins/central:small" is correct

• Check the Cert configuration and it should show empty

❯ ./bin/tanzu config cert list
  HOSTNAME  CA-CERTIFICATE  SKIP-CERT-VERIFICATION  INSECURE

• Add the ca cert for the test local central repository and show the list

❯ ./bin/tanzu config cert add --hostname localhost:9876 --ca-certificate /Users/pkalle/projects/tanzu-cli/hack/central-repo/certs/localhost.crt
[ok] successfully added certificate data for host localhost:9876

❯ ./bin/tanzu config cert list
  HOSTNAME        CA-CERTIFICATE  SKIP-CERT-VERIFICATION  INSECURE
  localhost:9876  <REDACTED>      Not configured          Not configured

• Now the plugin search should show the error as the CLI is using it's embedded public-key where as local test central repo is cosign signature is done using the test key-pairs available in the repository

❯ ./bin/tanzu plugin search
[i] Reading plugin inventory for "localhost:9876/tanzu-cli/plugins/central:small", this will take a few seconds.
[!] Unable to verify the plugins discovery image signature: failed validating the signature of the image localhost:9876/tanzu-cli/plugins/central:small :no matching signatures:
invalid signature when validating ASN.1 encoded signature
[x] Fatal, plugins discovery image signature verification failed. The `tanzu` CLI can not ensure the integrity of the plugins to be installed. To ignore this validation please append "localhost:9876/tanzu-cli/plugins/central:small" to the comma-separated list in the environment variable "TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST".  This is NOT RECOMMENDED and could put your environment at risk!

• Now export the custom public key for the test local repository DB image cosign signature and tanzu plugin search should be successful and should show the available plugins and also tanzu plugin install should install the plugin successfully

❯ export TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_PUBLIC_KEY_PATH=/Users/pkalle/projects/tanzu-cli/hack/central-repo/cosign-key-pair/cosign.pub


❯ ./bin/tanzu plugin search
[i] Reading plugin inventory for "localhost:9876/tanzu-cli/plugins/central:small", this will take a few seconds.
  NAME                DESCRIPTION                  TARGET           LATEST
  isolated-cluster    Desc for isolated-cluster    global           v9.9.9
  pinniped-auth       Desc for pinniped-auth       global           v9.9.9
  cluster             Desc for cluster             kubernetes       v9.9.9
  feature             Desc for feature             kubernetes       v9.9.9
  kubernetes-release  Desc for kubernetes-release  kubernetes       v9.9.9
  management-cluster  Desc for management-cluster  kubernetes       v9.9.9
  package             Desc for package             kubernetes       v9.9.9
  secret              Desc for secret              kubernetes       v9.9.9
  telemetry           Desc for telemetry           kubernetes       v9.9.9
  account             Desc for account             mission-control  v9.9.9
  apply               Desc for apply               mission-control  v9.9.9
  audit               Desc for audit               mission-control  v9.9.9
  cluster             Desc for cluster             mission-control  v9.9.9
  clustergroup        Desc for clustergroup        mission-control  v9.9.9
  continuousdelivery  Desc for continuousdelivery  mission-control  v9.9.9
  data-protection     Desc for data-protection     mission-control  v9.9.9
  ekscluster          Desc for ekscluster          mission-control  v9.9.9
  events              Desc for events              mission-control  v9.9.9
  helm                Desc for helm                mission-control  v9.9.9
  iam                 Desc for iam                 mission-control  v9.9.9
  inspection          Desc for inspection          mission-control  v9.9.9
  integration         Desc for integration         mission-control  v9.9.9
  management-cluster  Desc for management-cluster  mission-control  v9.9.9
  policy              Desc for policy              mission-control  v9.9.9
  secret              Desc for secret              mission-control  v9.9.9
  tanzupackage        Desc for tanzupackage        mission-control  v9.9.9
  workspace           Desc for workspace           mission-control  v9.9.9


❯ ./bin/tanzu plugin install isolated-cluster
[i] Installing plugin 'isolated-cluster:v9.9.9' with target 'global'
[ok] successfully installed 'isolated-cluster' plugin

• Add another cert with --skip-cert-verify and --insecure

❯ ./bin/tanzu config cert add --hostname test.host.com --skip-cert-verify true --insecure false
[ok] successfully added certificate data for host test.host.com


❯ ./bin/tanzu config cert list
  HOSTNAME        CA-CERTIFICATE  SKIP-CERT-VERIFICATION  INSECURE
  localhost:9876  <REDACTED>      Not configured          Not configured
  test.host.com   Not configured  true                    false

• Update cert config (--skip-cert-verify to true) for hostname localhost:9876

❯ ./bin/tanzu config cert update localhost:9876 --skip-cert-verify true
[ok] updated certificate data for host localhost:9876

❯ ./bin/tanzu config cert list
  HOSTNAME        CA-CERTIFICATE  SKIP-CERT-VERIFICATION  INSECURE
  localhost:9876  <REDACTED>      true                    Not configured
  test.host.com   Not configured  true                    false

• Delete cert config

❯ ./bin/tanzu config cert delete test.host.com
[ok] deleted certificate data for host test.host.com

❯ ./bin/tanzu config cert list
  HOSTNAME        CA-CERTIFICATE  SKIP-CERT-VERIFICATION  INSECURE
  localhost:9876  <REDACTED>      true                    Not configured

Release note

Add support for custom certificate configuration to interact with custom registry with self-signed certificate or with expired cert

Additional information

May have to raise another PR if checking in the test certs flags any security vulnerabilities.

Special notes for your reviewer

[marckhouzam]

Continuing the review, but here are a few comments to start.
Thanks @prkalle for this PR.

[marckhouzam]

I wonder if a user will understand they can use a hostname:port form when we use the term hostname all the time? Would having --port flag (which we would then join to the hostname before storing it in the config file)?

[prkalle]

I wonder if a user will understand they can use a hostname:port form when we use the term hostname all the time? Would having --port flag (which we would then join to the hostname before storing it in the config file)?

We can use host to specify the combination of both hostname:port. I can rename it host instead of hostname and provide examples(both with and without port).
Ref:

1. https://github.com/golang/go/blob/go1.19.2/src/net/url/url.go#L363
2. https://developer.mozilla.org/en-US/docs/Web/API/URL/host

[marckhouzam]

This was really great @prkalle !

Two questions and some final in-line comments. Thanks!

1. I'm not really sure, but do we need any special handling for a Proxy? Could there be a proxy in between the CLI and the central repo?
2. The builder plugin uses imgpkg directly. Does it also need to respect the new cert configuration? See: https://github.com/vmware-tanzu/tanzu-cli/blob/main/cmd/plugin/builder/imgpkg/imgpkg.go

[prkalle]

Two questions and some final in-line comments. Thanks!

1. I'm not really sure, but do we need any special handling for a Proxy? Could there be a proxy in between the CLI and the central repo?

Go by default handles the Proxy (User can set environment variables, HTTP_PROXY,HTTPS_PROXY,NO_PROXY). So we should be good. I did double check the imgpkg library, it does uses the DefaultTransport which handles the proxy.

2.The builder plugin uses imgpkg directly. Does it also need to respect the new cert configuration? See: https://github.com/vmware-tanzu/tanzu-cli/blob/main/cmd/plugin/builder/imgpkg/imgpkg.go

This is a good point. I see currently, developer can use these plugins to test locally, where the local repository is instantiated without HTTPS port. The other usecase would be , Core CLI would use these plugin publish command to publish to other harbor repositories (which are currently signed with well known CA). I think we should respect the new cert configuration for these as well, but we can discuss before updating the same.

[anujc25]

Is it possible to generate the hack/central-repo/cert and hack/central-repo/cosign-key-pair directories dynamically with the added make target setup-custom-cert-for-test-central-repo and put both directories into .gitignore to avoid adding those test keys/certs to the GitHub repository?

[marckhouzam]

This all look good to me @prkalle .

As for the new co-existence tests failing, we may need to set the TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_PUBLIC_KEY_PATH as you did for the E2E tests.

[anujc25]

non-blocking: But something to enhance in future if we decide to do so.

Generally, Convention is to use the 9443 port to run this type of server for testing. Basically 9443 port gives users the idea that the server is backed by https and not http.

[prkalle]

I know of 8443 as substitute for 443(https port). Would like to know if there is any reference document for the 9443 port convention. I checked the IANA, but I couldn't get more information about the convention.

[prkalle]

Is it possible to generate the hack/central-repo/cert and hack/central-repo/cosign-key-pair directories dynamically with the added make target setup-custom-cert-for-test-central-repo and put both directories into .gitignore to avoid adding those test keys/certs to the GitHub repository?

Filed an issue :#249 for the same. Generating certs for test local central registry endpoint CA cert shouldn't be an issue. However co-sign keys generation and especially signing the local test central repository with cosign key would be tricky( since it would be time consuming, because currently we are using pregenerated registry contents with cosign signatures) but not impossible.

[Update]: private keys (for both local test central repo cert private key and test central repo plugin inventory image co-sign key) would not be added to git repository until we work on the issue filed.

[anujc25]

Added a comment regarding the recent pipeline failure. Otherwise looks good.

Thanks for being flexible and fixing all the comments.

[anujc25]

This looks great. 👏