Commonupdate (#94)

* Updated namespaces template to include labels and annotations functionality

* Added schema validation to support additional formal for labels and annotations

* Updated the values-example.yaml to include new format for namespaces

* Updated Changes.md to include new namespaces functionality.

* Updating CI tests

* Fixed Markdown errors

* - Added functionality to support the following format for labels and annotations:
      labels:
        openshift.io/node-selector: ""
      annotations:
        openshift.io/cluster-monitoring: "true"

* Fixed CI Issues

* Avoid exited containers proliferation

When running the `pattern.sh` script multiple times, a lot of
podman exited containers will be left on the machine, adding
`--rm` parameter to `podman run` makes podman automatically
delete the exited containers leaving the machine cleaner.

* Handling of pre-release builds is too complex for a helm chart

Generating the ICSP and allowing insecure registries is best done prior
to helm upgrade, and requires VPN access to registry-proxy.engineering.redhat.com

* Fixing issues with operator groups

* Adding CI test

* Updated operator group template

* Updating CI issues

* Removed duplicate code for operatorgroup by using multiple conditions

* Allow overriding the pattern's name

This is especially useful when multiple people are working on a pattern
an have been using different names:

    $ make help |grep Pattern:
    Pattern: multicloud-gitops
    $ make NAME=foobar help |grep Pattern:
    Pattern: foobar

* Add precise instruction to upgrade the vault subchart

* Upgrade vault-helm to v0.24.1

* Add an item to README.md

* Fix up common/ tests

* Fix super linter

* Set gitOpsSpec.operatorSource

After merging validatedpatterns/patterns-operator@235b303
it is now effectively possible to pick a different catalogSource for
the gitops operator. This is needed in order to allow CI to install
the gitops operator from an IIB

* Introduce EXTRA_HELM_OPTS

This variable can be set in order to pass additional helm arguments from the
the command line. I.e. we can set things without having to tweak values files
So it is now possible to run something like the following:

  ./pattern.sh make install \
  EXTRA_HELM_OPTS="--set main.gitops.operatorSource=iib-49232"

* Disable var-naming[no-role-prefix] in ansible lint

* Add new ansible role to deal with IIBs

* Simplify load-iib target

* Add templates folder

* Fix a couple of linting warnings

* Fix some super-linter complaints

* Skip the iib-ci playbook

* Drop var-naming[no-role-prefix] linter

* Allow for multiple images when calling load-iib

* Add help for load-iib

* Output index_image in make

* Output index_image in make (2)

* Set facts later in the playbook not in defaults/

* Fix how we export vars in make load-iib

* Fix how we export vars in make load-iib (2)

* Use machineCount to register the number of nodes that need to be ready

* Add helpful debug messages

* Add | on shell now that we call pipefail

* Test dropping nevercontact source

* Skip insecure tls when logging in

* Also allow gchr.io

* Revert "Test dropping nevercontact source"

This reverts commit d8746a37fce2663018f52203c892f00b825e32a7.

* Fix typo

* Clarify instructions in the README file

* Automate the channel example

* Find out KUBEADMINAPI programmatically

* Use command instead of shell

* Do not grep for operator bundle unless it is the gitops operator

* Also whitelist ghcr.io

* Fetch the operator bundle itself in a more robust way

It seems that the operator bundle image itself is nowhere to be found
inside any OCP cluster object (it's not in packagemanifests nor
catalogsource). Resorting to parsing the IIB via opm alpha commands
to fetch the exact image.

* Add more mirrors

* Some more work to support MCE

* Cleanup spacing

* Fix super-linter

* Move task in right folder

* Drop last mention of operator instead of item

* Improve the grepping for the operator bundle

Without also grepping for the default_channel we can end up getting
multiple results, which breaks everything.

Tested this and it fixed the issue I was seeing with the
openshift-gitops-operator this morning

* Drop display_skipped_hosts

display_skipped_hosts=False has a horrible side-effect:
When a task takes a long time, it is always the *next* task and not the
one printed on the screen/log. That is because ansible has to wait for
the task to finish before printing it as it does not know before hand if
the host will be skipped and hence the task should not be displayed at
all

* Be more specific about the steps in the README

* Upgrade ESO to v0.8.2

* Update README.md

* Update tests after eso 0.8.2 upgrade

* Move to new spec format for dex/sso

Via https://issues.redhat.com/browse/GITOPS-2761 we are told that the
dex configuration has a new format.
Old format:

    spec:
      dex:
        openShiftOAuth: true
        resources:
        ...

New format:

    spec:
      sso:
        provider: dex
        dex:
          openShiftOAuth: true
          resources:
          ...

This format is only supported starting with gitops-1.8.0, so we should
merge this only when we are absolutely sure that no pattern in no
situation needs an older gitops version.

Tested on MCG with gitops-1.8.2

Note: with this change gitops < 1.8 is not supported. Starting with
gitops-1.9 the old format will be unsupported.

* Disable ArgoCD from kubeconform

The reason is that most of the tools we used to generate the json
schema, seem to be unmaintained, so it is getting hard to update
our schemas in our GH org. We'll need to revisit this in the future.

* Add a short line about username/token for the iib role on OCP <= 4.12

* Drop https:// from podman login

Seems we hit https://www.github.com/containers/podman/issues/13691 at
least with older podman versions.

If this turns out to break podman 4.5.0 I will special case it later

* Set the mce-subscription-spec annotation

We set it by default to "redhat-operators" and if defined to .Values.clusterGroup.subscriptions.acm.source
The reason we do this is the following:
1. In a default deployment scenario MCE has to be deployed as normal
   from the redhat-operators catalogSource just as ACM is
2. When we deploy gitops-operator from an IIB instead, MCE would be
   installed trying to get it from the IIB because
   https://www.github.com/stolostron/multiclusterhub-operator/pull/975
   made it so that it picks the latest version looking at all catalog
   sources. But since we only mirrored the gitops operator in the
   cluster, this breaks as the images for MCE from the IIB are not there
   By setting the default to "redhat-operators" we fix this case
3. Now in the case where we want to install ACM from an IIB we need to
   be able to override this and we will pick whatever value is set in
   .Values.clusterGroup.subscriptions.acm.source, which will need to be
   defined for this to work when testing ACM+MCE from an IIB

Note: Currently point 3. works only if you set it in a values file.
Setting .Values.clusterGroup.subscriptions.acm.source via extraParams
won't be passed down from the clusterGroup app to the applications.
It's a bug that we need to fix.

Note(2): We surround this with an 'if kindIs "map" .Values.clusterGroup.subscriptions'
because we do not want to break things if subscription is a list and not
a map. If we ever manage to drop subscriptions as list, then we can
remove that if

* Fix typo in README for iib

* Simplify the README a bit

* Add support for extraParams being passed down to all applications

Via validatedpatterns/patterns-operator#74
we add the extraParams in an extraParametersNested dictionary that holds
the extraParams key/value pairs. If they exist, let's add them as
parameters.

This allows them to end up in the applications.

* Add a lookup playbook to figure out IIB numbers

* Allow overriding channel and source when installing the patterns-operator

This will allow us to test the patterns-operator using a different
catalogsource (potentially installed via an IIB). So we can run:

make EXTRA_HELM_OPTS="\
  --set main.extraParameters[0].name=main.patternsOperator.channel --set main.extraParameters[0].value=slow \
  --set main.extraParameters[1].name=main.patternsOperator.source --set main.extraParameters[1].value=patten-index" install

* Fix small typo in iib instructions

* Drop a redirect and up retries when pushing the IIB to the internal registry

* Update ESO to v0.8.3

* WIP add presync for eso that waits for vault to be up

* Add tests

* Fix image and comment

* Adding rbac to support the vault sa checking on the vault-0 pod status.

* Make Test

* Removed previous version of common to convert to subtree from https://github.com/hybrid-cloud-patterns/common.git main

* Make test

---------

Co-authored-by: Lester Claudio <[email protected]>
Co-authored-by: Lorenzo Dalrio <[email protected]>
Co-authored-by: Michele Baldessari <[email protected]>
Co-authored-by: Andrew Beekhof <[email protected]>
Co-authored-by: Martin Jackson <[email protected]>

common/.ansible-lint

@@ -6,9 +6,12 @@ skip_list:
   - template-instead-of-copy # Templated files should use template instead of copy
   - yaml[line-length] # too long lines
   - yaml[indentation] # Forcing lists to be always indented by 2 chars is silly IMO
+  - var-naming[no-role-prefix] # This would be too much churn for very little gain
+  - no-changed-when
+  - var-naming[no-role-prefix] # There are too many changes now and it would be too risky
 
 # ansible-lint gh workflow cannot find ansible.cfg hence fails to import vault_utils role
 exclude_paths:
   - ./ansible/playbooks/vault/vault.yaml
+  - ./ansible/playbooks/iib-ci/iib-ci.yaml
   - ./ansible/roles/vault_utils/tests/test.yml
-

common/Changes.md

@@ -1,5 +1,18 @@
 # Changes
 
+## May 22, 2023
+
+* Upgraded ESO to 0.8.2
+* *Important* we now use the newly blessed sso config for argo. This means that gitops < 1.8 are *unsupported*
+
+## May 18, 2023
+
+* Introduce a EXTRA_HELM_OPTS env variable that will be passed to the helm invocations
+
+## April 21, 2023
+
+* Added labels and annotation support to namespaces.yaml template
+
 ## Apr 11, 2023
 
 * Apply the ACM ocp-gitops-policy everywhere but the hub

common/Makefile

@@ -1,11 +1,16 @@
-NAME=$(shell basename "`pwd`")
+NAME ?= $(shell basename "`pwd`")
 ifneq ($(origin TARGET_SITE), undefined)
   TARGET_SITE_OPT=--set main.clusterGroupName=$(TARGET_SITE)
 endif
 
+# This variable can be set in order to pass additional helm arguments from the
+# the command line. I.e. we can set things without having to tweak values files
+EXTRA_HELM_OPTS ?=
+
 # INDEX_IMAGES=registry-proxy.engineering.redhat.com/rh-osbs/iib:394248
-INDEX_IMAGES ?= 
-INDEX_OPTIONS=$(shell echo $(INDEX_IMAGES) | tr ',' '\n' | awk -F: 'match($$1,"/"){print "--set main.extraParameters."NR".name=clusterGroup.indexImages."NR".image --set main.extraParameters."NR".value="$$1":"$$2}')
+# or
+# INDEX_IMAGES=registry-proxy.engineering.redhat.com/rh-osbs/iib:394248,registry-proxy.engineering.redhat.com/rh-osbs/iib:394249
+INDEX_IMAGES ?=
 
 TARGET_ORIGIN ?= origin
 # This is to ensure that whether we start with a git@ or https:// URL, we end up with an https:// URL
@@ -15,12 +20,13 @@ TARGET_REPO=$(shell git ls-remote --get-url --symref $(TARGET_ORIGIN) | sed -e '
 TARGET_BRANCH=$(shell git rev-parse --abbrev-ref HEAD)
 
 # --set values always take precedence over the contents of -f
-HELM_OPTS=-f values-global.yaml --set main.git.repoURL="$(TARGET_REPO)" --set main.git.revision=$(TARGET_BRANCH) $(TARGET_SITE_OPT) $(INDEX_OPTIONS)
+HELM_OPTS=-f values-global.yaml --set main.git.repoURL="$(TARGET_REPO)" --set main.git.revision=$(TARGET_BRANCH) $(TARGET_SITE_OPT) $(EXTRA_HELM_OPTS)
 
 ##@ Pattern Common Tasks
 
 .PHONY: help
 help: ## This help message
+	@echo "Pattern: $(NAME)"
 	@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n"} /^(\s|[a-zA-Z_0-9-])+:.*?##/ { printf "  \033[36m%-35s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
 
 #  Makefiles in the individual patterns should call these targets explicitly
@@ -55,6 +61,15 @@ uninstall: ## runs helm uninstall
 load-secrets: ## loads the secrets into the vault
 	common/scripts/vault-utils.sh push_secrets $(NAME)
 
+.PHONY: load-iib
+load-iib: ## CI target to install Index Image Bundles
+	@set -e; if [ x$(INDEX_IMAGES) != x ]; then \
+		for IIB in $(shell echo $(INDEX_IMAGES) | tr ',' '\n'); do \
+			INDEX_IMAGE="$${IIB}" ansible-playbook common/ansible/playbooks/iib-ci/iib-ci.yaml; \
+		done; \
+	fi
+
+
 ##@ Validation Tasks
 
 # We only check the remote ssh git branch's existance if we're not running inside a container
@@ -110,7 +125,8 @@ helmlint: ## run helm lint
 	@for t in $(CHARTS); do common/scripts/lint.sh $$t $(TEST_OPTS); if [ $$? != 0 ]; then exit 1; fi; done
 
 API_URL ?= https://raw.githubusercontent.com/hybrid-cloud-patterns/ocp-schemas/main/openshift/4.10/
-KUBECONFORM_SKIP ?= -skip 'CustomResourceDefinition,ClusterIssuer,CertManager,Certificate'
+KUBECONFORM_SKIP ?= -skip 'CustomResourceDefinition,ClusterIssuer,CertManager,Certificate,ArgoCD'
+
 # We need to skip 'CustomResourceDefinition' as openapi2jsonschema seems to be unable to generate them ATM
 .PHONY: kubeconform
 kubeconform: ## run helm kubeconform

common/acm/templates/multiclusterhub.yaml

@@ -5,4 +5,7 @@ metadata:
   namespace: open-cluster-management
   annotations:
     argocd.argoproj.io/sync-wave: "-1"
+    {{- if kindIs "map" .Values.clusterGroup.subscriptions }}
+    installer.open-cluster-management.io/mce-subscription-spec: '{"source": "{{ default "redhat-operators" .Values.clusterGroup.subscriptions.acm.source }}" }'
+    {{- end }}
 spec: {}

common/acm/values.yaml

@@ -9,6 +9,9 @@ global:
 
 
 clusterGroup:
+  subscriptions:
+    acm:
+      source: redhat-operators
   managedClusterGroups:
 #    testRegion:
 #      name: region-one

common/ansible/ansible.cfg

@@ -1,5 +1,4 @@
 [defaults]
-display_skipped_hosts=False
 localhost_warning=False
 library=./plugins/modules:~/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
 roles_path=./roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles

common/ansible/playbooks/iib-ci/iib-ci.yaml

@@ -0,0 +1,8 @@
+# This playbook invokes the iib_ci role
+---
+- name: IIB CI playbook
+  hosts: localhost
+  connection: local
+  gather_facts: false
+  roles:
+    - iib_ci

common/ansible/playbooks/iib-ci/lookup.yml

@@ -0,0 +1,46 @@
+---
+- name: IIB CI playbook
+  hosts: localhost
+  connection: local
+  gather_facts: false
+  vars:
+    rh_url: "https://datagrepper.engineering.redhat.com/raw?topic=/topic/VirtualTopic.eng.ci.redhat-container-image.index.built&contains=%s&rows_per_page=20"
+    operator: "openshift-gitops-1-gitops-operator-bundle"
+    ocp_versions: {}
+  tasks:
+    - name: Set url fact
+      ansible.builtin.set_fact:
+        url: "{{ rh_url | format(operator + ':v') }}"
+
+    - name: Fetch URI
+      ansible.builtin.uri:
+        url: "{{ url }}"
+        return_content: true
+      register: jsoncontent
+
+    - name: Setting content
+      ansible.builtin.set_fact:
+        content: "{{ jsoncontent['content'] | from_json }}"
+
+    - name: Set messages fact
+      ansible.builtin.set_fact:
+        raw_messages: "{{ content.raw_messages }}"
+
+    # The when clause is because if we already have an IIB for an ocp version we do not
+    # want to override it (combine will always override existing keys)
+    # Reason for this is that the messages are sorted last first and we only want the
+    # last entries
+    - name: Set output
+      ansible.builtin.set_fact:
+        ocp_versions: "{{ ocp_versions | combine({item['msg']['index']['ocp_version']: {'indeximage': item['msg']['index']['index_image'], 'bundleimage': item['msg']['index']['added_bundle_images'][0]}}) }}"
+      loop: "{{ raw_messages }}"
+      when: item['msg']['index']['ocp_version'] is not in ocp_versions
+      loop_control:
+        label: "{{ item['msg']['index']['ocp_version'] }}"
+
+    - name: Print OCP versions for "{{ operator }}"
+      ansible.builtin.debug:
+        msg: "{{ item.key }} -> {{ item.value }}"
+      loop: "{{ ocp_versions | dict2items }}"
+      loop_control:
+        label: "{{ item.key }}"

common/ansible/roles/iib_ci/README.md

@@ -0,0 +1,85 @@
+# IIB Utilities
+
+A set of ansible plays to fetch an IIB (Image Index Bundle, aka a container created by the operator sdk
+that contains a bunch of references to operators that can be installed in an OpenShift cluster)
+
+Run `make lookup` to see which IIBs are available.
+
+Typically IIB are prerelease stuff that lives on some internal boxes. What these scripts do is fetch
+the IIB internally, mirror it to the registry inside the cluster, parse all the needed images and mirror
+those to the internal cluster registry and then set up the registries.conf files on all nodes so
+that the images used are the ones pointing to the internal cluster.
+
+## Usage
+
+By default the operator to be installed from the IIB is `openshift-gitops-operator`. You can override this through the `OPERATOR` env variable.
+For example, to mirror an operator into an existing cluster you would do the following:
+
+```sh
+export KUBECONFIG=/tmp/foo/kubeconfig
+export OPERATOR=openshift-gitops-operator
+export IIB=492329
+export INDEX_IMAGES=registry-proxy.engineering.redhat.com/rh-osbs/iib:${IIB}
+export KUBEADMINPASS="11111-22222-33333-44444"
+# This will push the IIB and all the needed images for the default openshift-gitops-operator into the cluster
+make load-iib
+# This will install the pattern using the gitops operator from the IIB
+```
+
+Then in case of the `openshift-gitops-operator` we would install with:
+
+```sh
+export CHANNEL=$(oc get -n openshift-marketplace packagemanifests -l "catalog=iib-${IIB}" --field-selector "metadata.name=${OPERATOR}" -o jsonpath='{.items[0].status.defaultChannel}')
+make EXTRA_HELM_OPTS="--set main.gitops.operatorSource=iib-${IIB} --set main.gitops.channel=${CHANNEL}" install
+```
+
+To install ACM (`export OPERATOR=advanced-cluster-management`) or any other
+operator (except the gitops one) from an IIB we would call the following as a
+final step:
+
+```sh
+export CHANNEL=$(oc get -n openshift-marketplace packagemanifests -l "catalog=iib-${IIB}" --field-selector "metadata.name=${OPERATOR}" -o jsonpath='{.items[0].status.defaultChannel}')
+make EXTRA_HELM_OPTS="--set main.extraParameters[0].name=clusterGroup.subscriptions.acm.source --set main.extraParameters[0].value=iib-${IIB} --set main.extraParameters[1].name=clusterGroup.subscriptions.acm.channel --set main.extraParameters[1].value=${CHANNEL}" install
+```
+
+*Note*: This needs VP operator version >= 0.0.14
+
+### OCP 4.13 and onwards
+
+Since 4.13 supports an internal registry that can cope with v2 docker manifests, we
+use that. Run `make iib` with the following environment variables set:
+
+* `INDEX_IMAGES=registry-proxy.engineering.redhat.com/rh-osbs/iib:492329`
+* `KUBEADMINPASS="11111-22222-33333-44444"`
+
+### OCP 4.12 and previous versions
+
+Due to the lack of v2 manifest support on the internal registry, we use an external
+registry. Run `make iib` with the following environment variables set:
+
+* `INDEX_IMAGES=registry-proxy.engineering.redhat.com/rh-osbs/iib:492329`
+* `REGISTRY=quay.io/rhn_support_mbaldess/iib`
+* `REGISTRY_TOKEN=<username>:<token>`
+
+*Note*: For the REGISTRY_TOKEN go to your quay repository, add a robot with "Write" permissions. The robot created will have a "username" and "password" fields. Set the REGISTRY_TOKEN environment variable to that value.
+
+## Useful commands
+
+* List IIBs for an operator:
+
+```sh
+ansible-playbook common/ansible/playbooks/iib-ci/lookup.yml
+...
+ok: [localhost] => (item=v4.13) => {
+    "msg": "v4.13 -> {'indeximage': 'registry-proxy.engineering.redhat.com/rh-osbs/iib:509435', 'bundleimage': 'registry-proxy.engineering.redhat.com/rh-osbs/openshift-gitops-1-gitops-operator-bundle:v99.9.0-106'}"
+}
+...
+```
+
+Override the `operator` value with the desired bundle name to figure out the last IIBs for it.
+
+* List all images uploaded to the internal registry:
+
+```sh
+oc exec -it -n openshift-image-registry $(oc get pods -n openshift-image-registry -o json | jq -r '.items[].metadata.name | select(. | test("^image-registry-"))' | head -n1) -- bash -c "curl -k -u kubeadmin:$(oc whoami -t) https://localhost:5000/v2/_catalog"
+```

common/ansible/roles/iib_ci/defaults/main.yml

@@ -0,0 +1,17 @@
+rh_internal_registry: registry-proxy.engineering.redhat.com
+iib_image: "{{ lookup('env', 'INDEX_IMAGE') }}"
+
+external_registry: "{{ lookup('env', 'REGISTRY') }}"
+external_registry_token: "{{ lookup('env', 'REGISTRY_TOKEN') }}"
+external_registry_email: noemail@localhost
+
+kubeadminpass: "{{ lookup('env', 'KUBEADMINPASS') }}"
+
+internal_registry_ns: openshift-marketplace
+internal_registry_email: noemail@localhost
+internal_registry_user: registry-custom-user
+internal_registry_pass: "{{ lookup('env', 'INTERNAL_REGISTRY_USER') }}"
+
+# We can use default(, true) below because OPERATOR is a string and not
+# a boolean
+operator: "{{ lookup('env', 'OPERATOR') | default('openshift-gitops-operator', true) }}"

common/ansible/roles/iib_ci/handlers/main.yml

@@ -0,0 +1,2 @@
+---
+# handlers file for vault_utils

common/ansible/roles/iib_ci/meta/main.yml

@@ -0,0 +1,29 @@
+galaxy_info:
+  author: Validated Patterns Team https://github.com/hybrid-cloud-patterns/
+  description: Internal module to work with IIBs (Image Index Bundles)
+
+  issue_tracker_url: https://github.com/hybrid-cloud-patterns/common/issues
+  license: Apache-2.0
+  min_ansible_version: "2.1"
+
+  # If this a Container Enabled role, provide the minimum Ansible Container version.
+  # min_ansible_container_version:
+
+  platforms:
+    - name: Fedora
+      versions:
+        - all
+    - name: Ubuntu
+      versions:
+        - all
+    - name: Debian
+      versions:
+        - all
+    - name: EL
+      versions:
+        - "8"
+        - "9"
+
+  galaxy_tags: []
+
+dependencies: []