add policy enforcement for bundle metadata format based on OCP versions

Signed-off-by: Jordan Keister <[email protected]>

task/fbc-validation/0.1/README.md

@@ -1,11 +1,9 @@
 # fbc-validation task
 
 ## Description:
-The fbc-validation task will ensure FBC (File based catalog) components uniquely linted to ensure they're properly
-constructed as part of the build pipeline. To validate the image in build pipeline, Skopeo is used to extract
-information from the image itself and then contents are checked using the OpenShift Operator Framework.  The binary
-used to run the validation is extracted from the base image for the component being tested.  Because of this, the
-base image must come from a trusted source.  Trusted sources are declared in `ALLOWED_BASE_IMAGES` in fbc-validation.yaml.
+Ensures file-based catalog (FBC) components are uniquely linted for proper construction as part of build pipeline.
+For details on specific checks, see the USAGE.md file.
+For troubleshooting assistance, see the TROUBLESHOOTING.md file.
 
 ## Params:
 

task/fbc-validation/0.1/TROUBLESHOOTING.md

@@ -0,0 +1,21 @@
+
+## Bundle properties are not permitted in a FBC fragment for OCP version
+
+Tasks may fail with an error message containing the string `bundle properties are not permitted in a FBC fragment for OCP version`.  This means that your fragment needs to utilize the appropriate FBC bundle metadata format which aligns with your target catalog. Failure to do so will result in your package not being displayed in the OpenShift Console.
+
+For OCP versions:
+- _4.16 or earlier_, bundle metadata must use the `olm.bundle.object` format
+- _4.17 or later_, bundle metadata must use the `olm.csv.metadata` format
+
+### If you use `opm` tooling to generate your fragment
+
+Note: This assumes that opm is version v1.46.0 or later.
+
+If you generate your FBC using catalog template expansion or migration of existing catalogs, then by default, the tool will output `olm.bundle.object` metadata format.
+You can choose to produce `olm.csv.metadata` format by using the `--migrate-level=bundle-object-to-csv-metadata` flag.  
+
+### If you use other tooling to generate your fragment
+
+Bundle data in `olm.csv.metadata` format contains only information that the OpenShift Console needs which is derived from the package's Cluster Standard Version(CSV).  Since the previous `olm.bundle.object` format would include bundle CSV metadata as well as other properties it is possible to convert from `olm.bundle.object` to `olm.csv.metadata`, but not the reverse. 
+
+If you rely on other tooling/processes to produce your fragment and currently use the `olm.bundle.object` bundle metadata format, then you may either adjust your tooling to generate `olm.csv.metadata` format or you may use `opm` to migrate your fragment's bundle metadata by using `opm render --migrate-level=bundle-object-to-csv-metadata [fragment-ref]` (where `fragment-ref` is a pullspec to the fragment or a path to a directory containing the fragment).

task/fbc-validation/0.1/USAGE.md

@@ -0,0 +1,22 @@
+# fbc-validation task
+
+## Checks:
+### Valid base image
+To validate the image in build pipeline, Skopeo is used to extract
+information from the image itself and then contents are checked using the OpenShift Operator Framework.  The binary
+used to run the validation is extracted from the base image for the component being tested.  Because of this, the
+base image must come from a trusted source.  Trusted sources are declared in `ALLOWED_BASE_IMAGES` in fbc-validation.yaml.
+
+### Valid FBC schema
+To validate the schema format of the FBC fragment, the test
+1. validates that the `operators.operatoframework.io.index.configs.v1` label is present on the image to identify the fragment path
+2. extracts the `opm` binary from the base image for the fragment
+3. executes `opm validate` over the fragment
+
+### Only one package in fragment
+To validate that only a single package is included in the fragment, the test renders the FBC using `opm` and uses `jq` to count instances of `olm.package` and fail if there is more than one.
+
+### Bundle metadata in the appropriate format
+To validate bundle metadata, the test evaluates bundle metadata usage against the target OCP version:
+- for 4.16 and earlier, fragments must use `olm.bundle.object` (and not use `olm.csv.metadata`)
+- for 4.17 and later, fragments must use `olm.csv.metadata` (and not use `olm.bundle.object`)

task/fbc-validation/0.1/fbc-validation.yaml

@@ -10,8 +10,8 @@ metadata:
 spec:
   description: >-
     Ensures file-based catalog (FBC) components are uniquely linted for proper construction as part of build pipeline.
-    The manifest data of container images obtained previously (via Skopeo) from inspect-image task is checked using OpenShift Operator Framework's opm CLI tool.
-    The opm binary is extracted from the container's base image, which must come from a trusted source.
+    For details on specific checks, see the USAGE.md file.
+    For troubleshooting assistance, see the TROUBLESHOOTING.md file.
   params:
     - name: IMAGE_URL
       description: Fully qualified image name.
@@ -62,7 +62,7 @@ spec:
 
         ### FBC base image check
         if [ -z "${BASE_IMAGE}" ]; then
-          echo "Base image is uknown. The file-based catalog must have base image defined. Check inspect-image task log."
+          echo "Base image is unknown. The file-based catalog must have base image defined. Check inspect-image task log."
           note="Task $(context.task.name) failed: The file-based catalog must have base image defined. For details, check Tekton task result TEST_OUTPUT in task inspect-image."
           TEST_OUTPUT=$(make_result_json -r ERROR -t "$note")
           echo "${TEST_OUTPUT}" | tee $(results.TEST_OUTPUT.path)
@@ -88,7 +88,6 @@ spec:
           exit 0
         fi
 
-
         ### Try to extract binaries with configs > check binaries functionality > check opm validate ###
         if [ ! -s ../inspect-image/image_inspect.json ]; then
           echo "File $(workspaces.workspace.path)/hacbs/inspect-image/image_inspect.json did not generate correctly. Check inspect-image task log."
@@ -152,8 +151,8 @@ spec:
         echo "OPM_BINARY: '${OPM_BINARY}'"
         chmod 775 "$OPM_BINARY"
 
-        # We have totally 3 checks here currently
-        check_num=3
+        # We have 7 total checks
+        check_num=7
         failure_num=0
         TESTPASSED=true
 
@@ -162,16 +161,78 @@ spec:
           failure_num=`expr $failure_num + 1`
           TESTPASSED=false
         fi
+
         if ! ${OPM_BINARY} validate ."${conffolder}"; then
           echo "!FAILURE! - opm validate check failed."
           failure_num=`expr $failure_num + 1`
           TESTPASSED=false
         fi
-        if ${OPM_BINARY} render ."${conffolder}" | jq -en 'reduce (inputs | select(.schema == "olm.package")) as $obj (0; .+1) < 1'; then
+
+        OPM_RENDERED_CATALOG=/tmp/catalog.json
+        ${OPM_BINARY} render ."${conffolder}" > ${OPM_RENDERED_CATALOG}
+        if [ ! -f ${OPM_RENDERED_CATALOG} ]; then
+          echo "!FAILURE! - unable to render the fragment FBC."
+          failure_num=`expr $failure_num + 1`
+          TESTPASSED=false
+        fi
+
+        if jq -en 'reduce (inputs | select(.schema == "olm.package")) as $obj (0; .+1) < 1' ${OPM_RENDERED_CATALOG}; then
           echo "!FAILURE! - There are no olm package entries defined in this FBC fragment."
           failure_num=`expr $failure_num + 1`
           TESTPASSED=false
         fi
+
+        if ! jq -en 'reduce (inputs | select(.schema == "olm.package")) as $obj (0; .+1) == 1' /tmp/ ${OPM_RENDERED_CATALOG}; then
+          echo "!FAILURE! - More than one olm.package is not permitted in a FBC fragment."
+          failure_num=`expr $failure_num + 1`
+          TESTPASSED=false
+        fi
+
+        OCP_VER_FROM_BASE=$(echo "${BASE_IMAGE}" | sed "s/@.*$//" | sed "s/^.*://")   # strips hash first due to greedy match
+        OCP_VER_MAJOR=$(echo "${OCP_VER_FROM_BASE}" | cut -d '.' -f 1)
+        OCP_VER_MINOR=$(echo "${OCP_VER_FROM_BASE}" | cut -d '.' -f 2)
+        OCP_BUNDLE_METADATA_THRESHOLD_MAJOR=4
+        OCP_BUNDLE_METADATA_THRESHOLD_MINOR=17
+        OCP_BUNDLE_METADATA_FORMAT="olm.bundle.object"
+
+        if [[ "${OCP_VER_MAJOR}" -ge "${OCP_BUNDLE_METADATA_THRESHOLD_MAJOR}" ]] && [[ "${OCP_VER_MINOR}" -ge "${OCP_BUNDLE_METADATA_THRESHOLD_MINOR}" ]]; then
+           OCP_BUNDLE_METADATA_FORMAT="olm.csv.metadata"
+        fi
+
+        # enforce the presence of either olm.csv.metadata or olm.bundle.object based on OCP version
+        if [[ "${OCP_BUNDLE_METADATA_FORMAT}" = "olm.csv.metadata" ]]; then
+          if ! jq -en 'reduce( inputs | select(.schema == "olm.bundle" and .properties[].type == "olm.bundle.object")) as $_ (0;.+1) == 0' ${OPM_RENDERED_CATALOG}; then
+            echo "!FAILURE! - olm.bundle.object bundle properties are not permitted in a FBC fragment for OCP version ${OCP_VER_MAJOR}.${OCP_VER_MINOR}. Fragments must move to olm.csv.metadata bundle metadata."
+            failure_num=`expr $failure_num + 1`
+            TESTPASSED=false
+          fi
+        else
+          if ! jq -en 'reduce( inputs | select(.schema == "olm.bundle" and .properties[].type == "olm.csv.metadata")) as $_ (0;.+1) == 0' ${OPM_RENDERED_CATALOG}; then
+            echo "!FAILURE! - olm.csv.metadata bundle properties are not permitted in a FBC fragment for OCP version ${OCP_VER_MAJOR}.${OCP_VER_MINOR}. Fragments must only use olm.bundle.object bundle metadata."
+            failure_num=`expr $failure_num + 1`
+            TESTPASSED=false
+          fi
+        fi
+
+        # enforce that each bundle has the OCP-version-appropriate bundle metadata.
+        BUNDLE_COUNT=$(jq -en 'def count(stream): reduce stream as $i (0; .+1); count(inputs|select(.schema=="olm.bundle"))' ${OPM_RENDERED_CATALOG})
+        BUNDLE_BO_COUNT=$(jq -en 'def count(stream): reduce stream as $i (0; .+1); count(inputs|select(.schema == "olm.bundle" and .properties[].type == "olm.bundle.object"))' ${OPM_RENDERED_CATALOG})
+        BUNDLE_CM_COUNT=$(jq -en 'def count(stream): reduce stream as $i (0; .+1); count(inputs|select(.schema == "olm.bundle" and .properties[].type == "olm.csv.metadata"))' ${OPM_RENDERED_CATALOG})
+
+        if [[ "${OCP_BUNDLE_METADATA_FORMAT}" = "olm.csv.metadata" ]]; then
+          if [[ "${BUNDLE_COUNT}" -ne "${BUNDLE_CM_COUNT}" ]]; then
+            echo "!FAILURE! - every olm.bundle object in the fragment must have a corresponding olm.csv.metadata bundle property"
+            failure_num=`expr $failure_num + 1`
+            TESTPASSED=false
+          fi
+        else
+          if [[ "${BUNDLE_BO_COUNT}" -lt "${BUNDLE_COUNT}" ]]; then
+            echo "!FAILURE! - every olm.bundle object in the fragment must have at least one olm.bundle.object bundle property"
+            failure_num=`expr $failure_num + 1`
+            TESTPASSED=false
+          fi
+        fi
+
         note="Task $(context.task.name) completed: Check result for task result."
         if [ $TESTPASSED == false ]; then
           ERROR_OUTPUT=$(make_result_json -r FAILURE -f $failure_num -s `expr $check_num - $failure_num` -t "$note")