From bc396a637079b82d2a2f2a7de82f8ab3e5d1b120 Mon Sep 17 00:00:00 2001 From: Patrick Thiem Date: Thu, 18 Jan 2024 10:22:06 +0100 Subject: [PATCH 01/11] Add general guidelines for writing test scripts regarding RFC 2119 Keywords (standards/issues/#447) Signed-off-by: Patrick Thiem --- .../zuul-ci-cd-quickstart-user-guide.md | 0 .../identity-federation-in-scs.md | 0 .../openstack-federation-via-oidc.md | 0 .../03-tests/RFC2119-keyword-test-guide.md | 65 +++++++++++++++++++ 4 files changed, 65 insertions(+) rename dev-docs/operations/{operations => 01-operations}/zuul-ci-cd-quickstart-user-guide.md (100%) rename dev-docs/operations/{iam => 02-iam}/identity-federation-in-scs.md (100%) rename dev-docs/operations/{iam => 02-iam}/openstack-federation-via-oidc.md (100%) create mode 100644 dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md diff --git a/dev-docs/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/dev-docs/operations/01-operations/zuul-ci-cd-quickstart-user-guide.md similarity index 100% rename from dev-docs/operations/operations/zuul-ci-cd-quickstart-user-guide.md rename to dev-docs/operations/01-operations/zuul-ci-cd-quickstart-user-guide.md diff --git a/dev-docs/operations/iam/identity-federation-in-scs.md b/dev-docs/operations/02-iam/identity-federation-in-scs.md similarity index 100% rename from dev-docs/operations/iam/identity-federation-in-scs.md rename to dev-docs/operations/02-iam/identity-federation-in-scs.md diff --git a/dev-docs/operations/iam/openstack-federation-via-oidc.md b/dev-docs/operations/02-iam/openstack-federation-via-oidc.md similarity index 100% rename from dev-docs/operations/iam/openstack-federation-via-oidc.md rename to dev-docs/operations/02-iam/openstack-federation-via-oidc.md diff --git a/dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md b/dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md new file mode 100644 index 0000000000..e068ddb33f --- /dev/null +++ b/dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md @@ -0,0 +1,65 @@ +--- +title: SCS RFC2119 Keyword Test Guide +type: +status: Draft +track: Global +--- + +The process of test creation in Sovereign Cloud Stack (SCS) involves developing and +validating tests that align with the specific requirements and standards set by SCS. +This entails a comprehensive understanding of the SCS framework, which is built on +open-source components and aims to provide interoperable and sovereign cloud services. +In creating tests for SCS, it's essential to consider the various aspects of cloud +technology, including infrastructure, services, and the integration of these components. +The tests should be designed to ensure that the SCS standards are met, which includes +adhering to specified requirement levels as indicated by RFC2119 keywords such as "MUST, +" "SHOULD," and "MAY." + +A systematic approach to test creation involves several steps: + +## 1. Understanding SCS Standards and RFC2119 Keywords: + +* SCS standards use RFC2119 keywords like "MUST," "MUST NOT," "SHOULD," "SHOULD NOT" and +"MAY" to define requirements for SCS-compatible IaaS and KaaS. These +standards are crucial for ensuring interoperability and sovereignty in cloud services, +as they are built on open-source components such as Kubernetes. + +## 2. Interpreting RFC2119 Keywords in the Context of SCS: + +* **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or +non-compliance directly affects the standard meeting. For visualization, fulfilled +requirements could be marked in green, and unfulfilled ones in red. + +* **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do +not directly affect standard compliance, it is advised for future-proofing against +potential standard evolutions. Non-compliance could be visualized in yellow, possibly +with a valid reason for non-fulfillment. + +* **MAY**: This is more flexible and could be used to provide additional +information or best practices. Their impact on compliance is not as direct as "MUST" or +"SHOULD" categories. These can be marked as blue. + +## 3. Channels for Output in Test Scripts: + +* Align the RFC2119 keywords with specific channels in test scripts: + + * "MUST" and "MUST NOT" should lead to errors + + * "SHOULD" and "SHOULD NOT" to warnings + + * "MAY" to notes or hints. + +## 4. Compliance and Test Passing Criteria: + +* Emphasize that for a test to pass, it must meet all the "MUST" requirements without +errors. The presence of warnings or informational items from "SHOULD" and "MAY" +categories does not directly impact the pass/fail status but is important for overall +quality and compatibility with future standards. + +## 5. Conclusion and Continuous Improvement: + +* Summarize the importance of adhering to these guidelines for maintaining compliance +and adaptability within the SCS framework. Highlight the need for ongoing updates and +revisions to the document to align with evolving standards and best practices in cloud +infrastructure and services. + From 1cb9b13dfbc4b446b9163660716ffbba33d9b4dd Mon Sep 17 00:00:00 2001 From: Patrick Thiem Date: Mon, 11 Mar 2024 12:55:08 +0100 Subject: [PATCH 02/11] fixup! Add general guidelines for writing test scripts regarding RFC 2119 Keywords (standards/issues/#447) Signed-off-by: cah-patrickthiem --- {dev-docs => contributor-docs}/index.md | 0 .../iam}/identity-federation-in-scs.md | 0 .../iam}/openstack-federation-via-oidc.md | 0 .../zuul-ci-cd-quickstart-user-guide.md | 0 .../tests/rfc2119-keyword-test-guide.md | 48 ++++++++----------- 5 files changed, 20 insertions(+), 28 deletions(-) rename {dev-docs => contributor-docs}/index.md (100%) rename {dev-docs/operations/02-iam => contributor-docs/operations/iam}/identity-federation-in-scs.md (100%) rename {dev-docs/operations/02-iam => contributor-docs/operations/iam}/openstack-federation-via-oidc.md (100%) rename {dev-docs/operations/01-operations => contributor-docs/operations/operations}/zuul-ci-cd-quickstart-user-guide.md (100%) rename dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md => contributor-docs/operations/tests/rfc2119-keyword-test-guide.md (81%) diff --git a/dev-docs/index.md b/contributor-docs/index.md similarity index 100% rename from dev-docs/index.md rename to contributor-docs/index.md diff --git a/dev-docs/operations/02-iam/identity-federation-in-scs.md b/contributor-docs/operations/iam/identity-federation-in-scs.md similarity index 100% rename from dev-docs/operations/02-iam/identity-federation-in-scs.md rename to contributor-docs/operations/iam/identity-federation-in-scs.md diff --git a/dev-docs/operations/02-iam/openstack-federation-via-oidc.md b/contributor-docs/operations/iam/openstack-federation-via-oidc.md similarity index 100% rename from dev-docs/operations/02-iam/openstack-federation-via-oidc.md rename to contributor-docs/operations/iam/openstack-federation-via-oidc.md diff --git a/dev-docs/operations/01-operations/zuul-ci-cd-quickstart-user-guide.md b/contributor-docs/operations/operations/zuul-ci-cd-quickstart-user-guide.md similarity index 100% rename from dev-docs/operations/01-operations/zuul-ci-cd-quickstart-user-guide.md rename to contributor-docs/operations/operations/zuul-ci-cd-quickstart-user-guide.md diff --git a/dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md similarity index 81% rename from dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md rename to contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index e068ddb33f..06a93330b1 100644 --- a/dev-docs/operations/03-tests/RFC2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -4,62 +4,54 @@ type: status: Draft track: Global --- - -The process of test creation in Sovereign Cloud Stack (SCS) involves developing and -validating tests that align with the specific requirements and standards set by SCS. -This entails a comprehensive understanding of the SCS framework, which is built on +The process of test creation in Sovereign Cloud Stack (SCS) involves developing and +validating tests that align with the specific requirements and standards set by SCS. +This entails a comprehensive understanding of the SCS framework, which is built on open-source components and aims to provide interoperable and sovereign cloud services. -In creating tests for SCS, it's essential to consider the various aspects of cloud +In creating tests for SCS, it's essential to consider the various aspects of cloud technology, including infrastructure, services, and the integration of these components. -The tests should be designed to ensure that the SCS standards are met, which includes +The tests should be designed to ensure that the SCS standards are met, which includes adhering to specified requirement levels as indicated by RFC2119 keywords such as "MUST, " "SHOULD," and "MAY." A systematic approach to test creation involves several steps: - -## 1. Understanding SCS Standards and RFC2119 Keywords: +## 1. Understanding SCS Standards and RFC2119 Keywords * SCS standards use RFC2119 keywords like "MUST," "MUST NOT," "SHOULD," "SHOULD NOT" and -"MAY" to define requirements for SCS-compatible IaaS and KaaS. These -standards are crucial for ensuring interoperability and sovereignty in cloud services, +"MAY" to define requirements for SCS-compatible IaaS and KaaS. These +standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. -## 2. Interpreting RFC2119 Keywords in the Context of SCS: +## 2. Interpreting RFC2119 Keywords in the Context of SCS -* **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or -non-compliance directly affects the standard meeting. For visualization, fulfilled +* **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or +non-compliance directly affects the standard meeting. For visualization, fulfilled requirements could be marked in green, and unfulfilled ones in red. - -* **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do -not directly affect standard compliance, it is advised for future-proofing against -potential standard evolutions. Non-compliance could be visualized in yellow, possibly +* **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do +not directly affect standard compliance, it is advised for future-proofing against +potential standard evolutions. Non-compliance could be visualized in yellow, possibly with a valid reason for non-fulfillment. - -* **MAY**: This is more flexible and could be used to provide additional -information or best practices. Their impact on compliance is not as direct as "MUST" or +* **MAY**: This is more flexible and could be used to provide additional +information or best practices. Their impact on compliance is not as direct as "MUST" or "SHOULD" categories. These can be marked as blue. -## 3. Channels for Output in Test Scripts: - -* Align the RFC2119 keywords with specific channels in test scripts: +## 3. Channels for Output in Test Scripts +* Align the RFC2119 keywords with specific channels in test scripts: * "MUST" and "MUST NOT" should lead to errors - * "SHOULD" and "SHOULD NOT" to warnings - * "MAY" to notes or hints. -## 4. Compliance and Test Passing Criteria: +## 4. Compliance and Test Passing Criteria * Emphasize that for a test to pass, it must meet all the "MUST" requirements without errors. The presence of warnings or informational items from "SHOULD" and "MAY" categories does not directly impact the pass/fail status but is important for overall quality and compatibility with future standards. -## 5. Conclusion and Continuous Improvement: +## 5. Conclusion and Continuous Improvement * Summarize the importance of adhering to these guidelines for maintaining compliance and adaptability within the SCS framework. Highlight the need for ongoing updates and revisions to the document to align with evolving standards and best practices in cloud infrastructure and services. - From 68778d8c8c63e881734acf2d3f5a896295747f04 Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Fri, 15 Mar 2024 17:20:34 +0100 Subject: [PATCH 03/11] Excluded the "Conclusion" section for now, done some rewording and added examples for RFC2119 keyword use for standard documents and tests. --- .../operations/tests/rfc2119-keyword-test-guide.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index 06a93330b1..87a748002e 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -17,7 +17,7 @@ adhering to specified requirement levels as indicated by RFC2119 keywords such a A systematic approach to test creation involves several steps: ## 1. Understanding SCS Standards and RFC2119 Keywords -* SCS standards use RFC2119 keywords like "MUST," "MUST NOT," "SHOULD," "SHOULD NOT" and +SCS standards use RFC2119 keywords like "MUST," "MUST NOT," "SHOULD," "SHOULD NOT" and "MAY" to define requirements for SCS-compatible IaaS and KaaS. These standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. @@ -44,14 +44,16 @@ information or best practices. Their impact on compliance is not as direct as "M ## 4. Compliance and Test Passing Criteria -* Emphasize that for a test to pass, it must meet all the "MUST" requirements without -errors. The presence of warnings or informational items from "SHOULD" and "MAY" -categories does not directly impact the pass/fail status but is important for overall -quality and compatibility with future standards. +A test is considered to pass if and only if it doesn't produce any messages on the +ERROR nor the CRITICAL channel. The presence of warnings or informational items from +"SHOULD" and "MAY" categories do not directly impact the pass/fail status but is +important for overall quality and compatibility with future standards. ## 5. Conclusion and Continuous Improvement -* Summarize the importance of adhering to these guidelines for maintaining compliance +Summarize the importance of adhering to these guidelines for maintaining compliance and adaptability within the SCS framework. Highlight the need for ongoing updates and revisions to the document to align with evolving standards and best practices in cloud infrastructure and services. + +In summary, in the context of the SCS framework it's From b8a705901d612b194c9ed24bc7b141182ade45b3 Mon Sep 17 00:00:00 2001 From: Patrick Thiem Date: Fri, 15 Mar 2024 17:30:29 +0100 Subject: [PATCH 04/11] fixup! Excluded the "Conclusion" section for now, done some rewording and added examples for RFC2119 keyword use for standard documents and tests. Signed-off-by: cah-patrickthiem --- .../tests/rfc2119-keyword-test-guide.md | 108 +++++++++++++++--- 1 file changed, 91 insertions(+), 17 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index 87a748002e..e05e0dbf73 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -11,14 +11,14 @@ open-source components and aims to provide interoperable and sovereign cloud ser In creating tests for SCS, it's essential to consider the various aspects of cloud technology, including infrastructure, services, and the integration of these components. The tests should be designed to ensure that the SCS standards are met, which includes -adhering to specified requirement levels as indicated by RFC2119 keywords such as "MUST, -" "SHOULD," and "MAY." +adhering to specified requirement levels as indicated by RFC2119 keywords such as **MUST**, +**SHOULD**, and **MAY**. A systematic approach to test creation involves several steps: ## 1. Understanding SCS Standards and RFC2119 Keywords -SCS standards use RFC2119 keywords like "MUST," "MUST NOT," "SHOULD," "SHOULD NOT" and -"MAY" to define requirements for SCS-compatible IaaS and KaaS. These +SCS standards use RFC2119 keywords like **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT** and +**MAY** to define requirements for SCS-compatible IaaS and KaaS. These standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. @@ -26,34 +26,108 @@ as they are built on open-source components such as Kubernetes. * **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or non-compliance directly affects the standard meeting. For visualization, fulfilled -requirements could be marked in green, and unfulfilled ones in red. +requirements could be marked in green, unfulfilled ones in red. * **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do not directly affect standard compliance, it is advised for future-proofing against potential standard evolutions. Non-compliance could be visualized in yellow, possibly with a valid reason for non-fulfillment. * **MAY**: This is more flexible and could be used to provide additional -information or best practices. Their impact on compliance is not as direct as "MUST" or -"SHOULD" categories. These can be marked as blue. +information or best practices. Their impact on compliance is not as direct as **MUST** or +**SHOULD** categories. These can be marked as blue. ## 3. Channels for Output in Test Scripts * Align the RFC2119 keywords with specific channels in test scripts: - * "MUST" and "MUST NOT" should lead to errors - * "SHOULD" and "SHOULD NOT" to warnings - * "MAY" to notes or hints. + * **MUST** and **MUST NOT** lead to the ERROR channel for failed tests + * **SHOULD** and **SHOULD NOT** lead to the WARNING channel, the test is still passed + * **MAY** to leads to the INFO channel, the test is still passed ## 4. Compliance and Test Passing Criteria A test is considered to pass if and only if it doesn't produce any messages on the ERROR nor the CRITICAL channel. The presence of warnings or informational items from -"SHOULD" and "MAY" categories do not directly impact the pass/fail status but is +**SHOULD** and **MAY** categories do not directly impact the pass/fail status but is important for overall quality and compatibility with future standards. -## 5. Conclusion and Continuous Improvement +## 5. Example: Standards Document using RFC2119 Keywords -Summarize the importance of adhering to these guidelines for maintaining compliance -and adaptability within the SCS framework. Highlight the need for ongoing updates and -revisions to the document to align with evolving standards and best practices in cloud -infrastructure and services. +The following example of a fictitious document for a "Web Server Installation Standard" +is intended to illustrate the use of RFC2119 keywords in connection with the creation of +SCS-compliant standard documents. -In summary, in the context of the SCS framework it's +### Web Server Installation Standard +#### Purpose +This document provides a concise set of requirements for installing a secure web server. + +#### Requirements +* Web Server Software: Apache HTTP Server 2.4.x or Nginx 1.18.x **MUST** be installed. No +other web server software versions or types are permitted. + +* Operating System: The server **MUST** run on an OS that receives current security updates. +Unsupported versions **MUST NOT** be used. + +* TLS Configuration: TLS 1.2 or higher **MUST** be enabled for all connections. SSL and +earlier TLS versions **MUST NOT** be used. + +* Security Patches: The web server and OS **MUST** be kept up-to-date with security patches. +Critical patches **SHOULD** be applied within 48 hours of release. + +* Firewall Setup: The firewall **SHOULD** be configured to allow only necessary ports, +such as 80 (HTTP) and 443 (HTTPS). All unnecessary ports **SHOULD NOT** be open. + +* Backup Plan: Regular backups of the web server data **MAY** be performed, and it is +recommended that backup integrity checks are conducted to ensure recoverability. + +#### Compliance +Failure to comply with these requirements results in the web server being +considered non-compliant with the organization's security policies. + +## 6. Example: Test script using RFC2119 Keywords + +The following example of a fictitious code for a test of "process_requirements" +is intended to illustrate the use of RFC2119 keywords in connection with the +creation of SCS-compliant standard tests. + +```python +def process_requirements(var1, var2): + try: + # debug log for input variables + logger.debug(f"Received input var1: {var1}, var2: {var2}") + + # example of an error case (equivalent to MUST and MUST NOT) + if var1 < 0: + logger.error("var1 must be positive") + elif var1 > 100: + logger.error("var1 must not exceed 100") + + # add debug log to confirm evaluation of var1 + logger.debug(f"var1 evaluated: {var1 < 0 or var1 > 100}") + + # example of a warning case (equivalent to SHOULD and SHOULD NOT) + if var2 < 10: + logger.warning("var2 should be at least 10 for optimal performance") + elif var2 > 50: + logger.warning("var2 should not exceed 50 for optimal performance") + + # add debug log to confirm evaluation of var2 + logger.debug(f"var2 evaluated: {var2 < 10 or var2 > 50}") + + # example of an informative case (equivalent to MAY) + if var1 + var2 < 100: + logger.info("Combination is within recommended range") + else: + logger.info("Combination exceeds the recommended range but may still proceed") + + # debug log before completing the process + logger.debug(f"Final evaluation before completion: var1 + var2 = {var1 + var2}") + + # success log + logger.debug("Process completed successfully") + + except ValueError as e: + # debug log before raising a critical error + logger.debug("Encountered a critical error about to log", exc_info=True) + logger.critical(f"Process failed: {e}", exc_info=True) +``` +Note: The DEBUG channel is used for additional information for developers to better understand a +process and does not apply to RFC2119 keywords. From a4c342a9029842d67308423630c8dfd0361bef6b Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Tue, 26 Mar 2024 18:40:49 +0100 Subject: [PATCH 05/11] Clarified explaination of RFC2119 keywords --- .../tests/rfc2119-keyword-test-guide.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index e05e0dbf73..0e7b3edc78 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -11,22 +11,26 @@ open-source components and aims to provide interoperable and sovereign cloud ser In creating tests for SCS, it's essential to consider the various aspects of cloud technology, including infrastructure, services, and the integration of these components. The tests should be designed to ensure that the SCS standards are met, which includes -adhering to specified requirement levels as indicated by RFC2119 keywords such as **MUST**, -**SHOULD**, and **MAY**. +adhering to specified requirement levels as indicated by RFC2119 keywords such as **MUST**, **SHOULD**, and **MAY**. -A systematic approach to test creation involves several steps: ## 1. Understanding SCS Standards and RFC2119 Keywords -SCS standards use RFC2119 keywords like **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT** and -**MAY** to define requirements for SCS-compatible IaaS and KaaS. These -standards are crucial for ensuring interoperability and sovereignty in cloud services, -as they are built on open-source components such as Kubernetes. +SCS standards use RFC2119 keywords like: +* **MUST** (same as **REQUIRED**, **SHALL**), +* **MUST NOT** (same as **SHALL NOT**), +* **SHOULD** (same as **RECOMMENDED**), +* **SHOULD NOT** (same as **NOT RECOMMENDED**), +* **MAY** (same as **OPTIONAL**) + +to define requirements for SCS-compatible IaaS and KaaS. These standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. + +To ensure readability and comprehensibility, only the main keywords **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT** and **MAY** are referred to below. ## 2. Interpreting RFC2119 Keywords in the Context of SCS * **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or non-compliance directly affects the standard meeting. For visualization, fulfilled -requirements could be marked in green, unfulfilled ones in red. +requirements could be marked in green, unfulfilled ones in red. * **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do not directly affect standard compliance, it is advised for future-proofing against potential standard evolutions. Non-compliance could be visualized in yellow, possibly @@ -92,7 +96,7 @@ creation of SCS-compliant standard tests. def process_requirements(var1, var2): try: # debug log for input variables - logger.debug(f"Received input var1: {var1}, var2: {var2}") + logger.debug(f"Received input var1: {var1}, var2: {var2}") # example of an error case (equivalent to MUST and MUST NOT) if var1 < 0: From 9cae2a4c9ff8a9e6751dd7bbc44eb64597ea82b2 Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Thu, 28 Mar 2024 14:14:21 +0100 Subject: [PATCH 06/11] fixup! Clarified explaination of RFC2119 keywords --- .../tests/rfc2119-keyword-test-guide.md | 35 +++++++++++++------ 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index 0e7b3edc78..af816b29ae 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -4,14 +4,17 @@ type: status: Draft track: Global --- -The process of test creation in Sovereign Cloud Stack (SCS) involves developing and -validating tests that align with the specific requirements and standards set by SCS. -This entails a comprehensive understanding of the SCS framework, which is built on -open-source components and aims to provide interoperable and sovereign cloud services. -In creating tests for SCS, it's essential to consider the various aspects of cloud -technology, including infrastructure, services, and the integration of these components. -The tests should be designed to ensure that the SCS standards are met, which includes -adhering to specified requirement levels as indicated by RFC2119 keywords such as **MUST**, **SHOULD**, and **MAY**. + +## Introduction + +The process of test creation and standards definition in Sovereign Cloud Stack (SCS) +involves developing and validating tests that align with the specific requirements and +standards set by SCS. This entails a comprehensive understanding of the SCS framework, +which is built on open-source components and aims to provide interoperable and sovereign +cloud services. In creating tests for SCS, it's essential to consider the various aspects +of cloud technology, including infrastructure, services, and the integration of these +components. The tests should be designed to ensure that the SCS standards are met, which +includes adhering to specified requirement levels as indicated by [RFC2119 keywords](https://datatracker.ietf.org/doc/html/rfc2119). ## 1. Understanding SCS Standards and RFC2119 Keywords @@ -22,7 +25,7 @@ SCS standards use RFC2119 keywords like: * **SHOULD NOT** (same as **NOT RECOMMENDED**), * **MAY** (same as **OPTIONAL**) -to define requirements for SCS-compatible IaaS and KaaS. These standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. +to define requirements for SCS-compatible IaaS and KaaS resources. These standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. To ensure readability and comprehensibility, only the main keywords **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT** and **MAY** are referred to below. @@ -53,6 +56,12 @@ ERROR nor the CRITICAL channel. The presence of warnings or informational items **SHOULD** and **MAY** categories do not directly impact the pass/fail status but is important for overall quality and compatibility with future standards. +A test can have one of three results: + +* fail: if it has at least one message on ERROR +* DNF (did not finish): if it has no message on ERROR, but at least one on CRITICAL +* pass: otherwise + ## 5. Example: Standards Document using RFC2119 Keywords The following example of a fictitious document for a "Web Server Installation Standard" @@ -93,6 +102,10 @@ is intended to illustrate the use of RFC2119 keywords in connection with the creation of SCS-compliant standard tests. ```python +import logging +logging.basicConfig(level=logging.DEBUG) +logger = logging.getLogger(__name__) + def process_requirements(var1, var2): try: # debug log for input variables @@ -105,7 +118,7 @@ def process_requirements(var1, var2): logger.error("var1 must not exceed 100") # add debug log to confirm evaluation of var1 - logger.debug(f"var1 evaluated: {var1 < 0 or var1 > 100}") + logger.debug(f"var1 evaluated: {var1 > 0 and var1 < 100}") # example of a warning case (equivalent to SHOULD and SHOULD NOT) if var2 < 10: @@ -114,7 +127,7 @@ def process_requirements(var1, var2): logger.warning("var2 should not exceed 50 for optimal performance") # add debug log to confirm evaluation of var2 - logger.debug(f"var2 evaluated: {var2 < 10 or var2 > 50}") + logger.debug(f"var2 evaluated: {var2 > 10 and var2 < 50}") # example of an informative case (equivalent to MAY) if var1 + var2 < 100: From a903f2bb9c10ff107609259952d8564a07f7ebb5 Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Thu, 28 Mar 2024 15:18:56 +0100 Subject: [PATCH 07/11] fixup! Merge branch 'main' into General-guidelines-for-writing-test-scripts/#447 --- .../operations/tests/rfc2119-keyword-test-guide.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index af816b29ae..1cb2ba04e2 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -19,6 +19,7 @@ includes adhering to specified requirement levels as indicated by [RFC2119 keywo ## 1. Understanding SCS Standards and RFC2119 Keywords SCS standards use RFC2119 keywords like: + * **MUST** (same as **REQUIRED**, **SHALL**), * **MUST NOT** (same as **SHALL NOT**), * **SHOULD** (same as **RECOMMENDED**), @@ -69,10 +70,13 @@ is intended to illustrate the use of RFC2119 keywords in connection with the cre SCS-compliant standard documents. ### Web Server Installation Standard + #### Purpose + This document provides a concise set of requirements for installing a secure web server. #### Requirements + * Web Server Software: Apache HTTP Server 2.4.x or Nginx 1.18.x **MUST** be installed. No other web server software versions or types are permitted. @@ -92,16 +96,18 @@ such as 80 (HTTP) and 443 (HTTPS). All unnecessary ports **SHOULD NOT** be open. recommended that backup integrity checks are conducted to ensure recoverability. #### Compliance + Failure to comply with these requirements results in the web server being considered non-compliant with the organization's security policies. -## 6. Example: Test script using RFC2119 Keywords +## 6. Example: Test script using RFC2119 Keywords The following example of a fictitious code for a test of "process_requirements" is intended to illustrate the use of RFC2119 keywords in connection with the creation of SCS-compliant standard tests. ```python + import logging logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger(__name__) @@ -145,6 +151,6 @@ def process_requirements(var1, var2): # debug log before raising a critical error logger.debug("Encountered a critical error about to log", exc_info=True) logger.critical(f"Process failed: {e}", exc_info=True) + ``` -Note: The DEBUG channel is used for additional information for developers to better understand a -process and does not apply to RFC2119 keywords. +Note: The DEBUG channel is used for additional information for developers to better understand a process and does not apply to RFC2119 keywords. \ No newline at end of file From 38c91cbf44a63d2a0551d6f73d440e6678bcbbc4 Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Thu, 28 Mar 2024 15:27:12 +0100 Subject: [PATCH 08/11] fixup! Merge branch 'main' into General-guidelines-for-writing-test-scripts/#447 --- contributor-docs/index.md | 2 +- .../tests/rfc2119-keyword-test-guide.md | 79 ++++++++++--------- 2 files changed, 41 insertions(+), 40 deletions(-) diff --git a/contributor-docs/index.md b/contributor-docs/index.md index e5a1381661..4d48f15b57 100644 --- a/contributor-docs/index.md +++ b/contributor-docs/index.md @@ -2,4 +2,4 @@ Welcome to the Contributor Docs. This section is primarily for SCS Contributors and will contain documentation regarding the Development and Architecture of the Sovereign Cloud Stack and its components. You will find documents, explanations and guides regarding the tooling necessary for the development of SCS. -If you want to contribute you can reach out to us via our [Matrix Channel](https://matrix.to/#/!TiDqlLmEUaXqTemaLc:matrix.org?via=matrix.org). \ No newline at end of file +If you want to contribute you can reach out to us via our [Matrix Channel](https://matrix.to/#/!TiDqlLmEUaXqTemaLc:matrix.org?via=matrix.org). diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index 1cb2ba04e2..61df01aa12 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -20,11 +20,11 @@ includes adhering to specified requirement levels as indicated by [RFC2119 keywo SCS standards use RFC2119 keywords like: -* **MUST** (same as **REQUIRED**, **SHALL**), -* **MUST NOT** (same as **SHALL NOT**), -* **SHOULD** (same as **RECOMMENDED**), -* **SHOULD NOT** (same as **NOT RECOMMENDED**), -* **MAY** (same as **OPTIONAL**) +- **MUST** (same as **REQUIRED**, **SHALL**), +- **MUST NOT** (same as **SHALL NOT**), +- **SHOULD** (same as **RECOMMENDED**), +- **SHOULD NOT** (same as **NOT RECOMMENDED**), +- **MAY** (same as **OPTIONAL**) to define requirements for SCS-compatible IaaS and KaaS resources. These standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. @@ -32,23 +32,23 @@ To ensure readability and comprehensibility, only the main keywords **MUST**, ** ## 2. Interpreting RFC2119 Keywords in the Context of SCS -* **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or -non-compliance directly affects the standard meeting. For visualization, fulfilled -requirements could be marked in green, unfulfilled ones in red. -* **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do -not directly affect standard compliance, it is advised for future-proofing against -potential standard evolutions. Non-compliance could be visualized in yellow, possibly -with a valid reason for non-fulfillment. -* **MAY**: This is more flexible and could be used to provide additional -information or best practices. Their impact on compliance is not as direct as **MUST** or -**SHOULD** categories. These can be marked as blue. +- **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or + non-compliance directly affects the standard meeting. For visualization, fulfilled + requirements could be marked in green, unfulfilled ones in red. +- **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do + not directly affect standard compliance, it is advised for future-proofing against + potential standard evolutions. Non-compliance could be visualized in yellow, possibly + with a valid reason for non-fulfillment. +- **MAY**: This is more flexible and could be used to provide additional + information or best practices. Their impact on compliance is not as direct as **MUST** or + **SHOULD** categories. These can be marked as blue. ## 3. Channels for Output in Test Scripts -* Align the RFC2119 keywords with specific channels in test scripts: - * **MUST** and **MUST NOT** lead to the ERROR channel for failed tests - * **SHOULD** and **SHOULD NOT** lead to the WARNING channel, the test is still passed - * **MAY** to leads to the INFO channel, the test is still passed +- Align the RFC2119 keywords with specific channels in test scripts: + - **MUST** and **MUST NOT** lead to the ERROR channel for failed tests + - **SHOULD** and **SHOULD NOT** lead to the WARNING channel, the test is still passed + - **MAY** to leads to the INFO channel, the test is still passed ## 4. Compliance and Test Passing Criteria @@ -59,9 +59,9 @@ important for overall quality and compatibility with future standards. A test can have one of three results: -* fail: if it has at least one message on ERROR -* DNF (did not finish): if it has no message on ERROR, but at least one on CRITICAL -* pass: otherwise +- fail: if it has at least one message on ERROR +- DNF (did not finish): if it has no message on ERROR, but at least one on CRITICAL +- pass: otherwise ## 5. Example: Standards Document using RFC2119 Keywords @@ -77,23 +77,23 @@ This document provides a concise set of requirements for installing a secure web #### Requirements -* Web Server Software: Apache HTTP Server 2.4.x or Nginx 1.18.x **MUST** be installed. No -other web server software versions or types are permitted. +- Web Server Software: Apache HTTP Server 2.4.x or Nginx 1.18.x **MUST** be installed. No + other web server software versions or types are permitted. -* Operating System: The server **MUST** run on an OS that receives current security updates. -Unsupported versions **MUST NOT** be used. +- Operating System: The server **MUST** run on an OS that receives current security updates. + Unsupported versions **MUST NOT** be used. -* TLS Configuration: TLS 1.2 or higher **MUST** be enabled for all connections. SSL and -earlier TLS versions **MUST NOT** be used. +- TLS Configuration: TLS 1.2 or higher **MUST** be enabled for all connections. SSL and + earlier TLS versions **MUST NOT** be used. -* Security Patches: The web server and OS **MUST** be kept up-to-date with security patches. -Critical patches **SHOULD** be applied within 48 hours of release. +- Security Patches: The web server and OS **MUST** be kept up-to-date with security patches. + Critical patches **SHOULD** be applied within 48 hours of release. -* Firewall Setup: The firewall **SHOULD** be configured to allow only necessary ports, -such as 80 (HTTP) and 443 (HTTPS). All unnecessary ports **SHOULD NOT** be open. +- Firewall Setup: The firewall **SHOULD** be configured to allow only necessary ports, + such as 80 (HTTP) and 443 (HTTPS). All unnecessary ports **SHOULD NOT** be open. -* Backup Plan: Regular backups of the web server data **MAY** be performed, and it is -recommended that backup integrity checks are conducted to ensure recoverability. +- Backup Plan: Regular backups of the web server data **MAY** be performed, and it is + recommended that backup integrity checks are conducted to ensure recoverability. #### Compliance @@ -116,13 +116,13 @@ def process_requirements(var1, var2): try: # debug log for input variables logger.debug(f"Received input var1: {var1}, var2: {var2}") - + # example of an error case (equivalent to MUST and MUST NOT) if var1 < 0: logger.error("var1 must be positive") elif var1 > 100: logger.error("var1 must not exceed 100") - + # add debug log to confirm evaluation of var1 logger.debug(f"var1 evaluated: {var1 > 0 and var1 < 100}") @@ -131,7 +131,7 @@ def process_requirements(var1, var2): logger.warning("var2 should be at least 10 for optimal performance") elif var2 > 50: logger.warning("var2 should not exceed 50 for optimal performance") - + # add debug log to confirm evaluation of var2 logger.debug(f"var2 evaluated: {var2 > 10 and var2 < 50}") @@ -140,7 +140,7 @@ def process_requirements(var1, var2): logger.info("Combination is within recommended range") else: logger.info("Combination exceeds the recommended range but may still proceed") - + # debug log before completing the process logger.debug(f"Final evaluation before completion: var1 + var2 = {var1 + var2}") @@ -153,4 +153,5 @@ def process_requirements(var1, var2): logger.critical(f"Process failed: {e}", exc_info=True) ``` -Note: The DEBUG channel is used for additional information for developers to better understand a process and does not apply to RFC2119 keywords. \ No newline at end of file + +Note: The DEBUG channel is used for additional information for developers to better understand a process and does not apply to RFC2119 keywords. From 9b1800f5eaae13c9947ab2fcfd3e9dad4b65b4d5 Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Thu, 4 Apr 2024 11:04:28 +0200 Subject: [PATCH 09/11] fixup! Merge branch 'main' into General-guidelines-for-writing-test-scripts/#447 --- .../tests/rfc2119-keyword-test-guide.md | 52 ++++++++++++------- 1 file changed, 34 insertions(+), 18 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index 61df01aa12..89b63b2f95 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -7,14 +7,11 @@ track: Global ## Introduction -The process of test creation and standards definition in Sovereign Cloud Stack (SCS) -involves developing and validating tests that align with the specific requirements and -standards set by SCS. This entails a comprehensive understanding of the SCS framework, -which is built on open-source components and aims to provide interoperable and sovereign -cloud services. In creating tests for SCS, it's essential to consider the various aspects -of cloud technology, including infrastructure, services, and the integration of these -components. The tests should be designed to ensure that the SCS standards are met, which -includes adhering to specified requirement levels as indicated by [RFC2119 keywords](https://datatracker.ietf.org/doc/html/rfc2119). +The development and validation of tests within the Sovereign Cloud Stack (SCS) is a critical endeavor aimed at ensuring that cloud technologies meet the highest standards of interoperability, security and sovereignty. This process is not only about aligning with the technical specifications and standards set by SCS but also about safeguarding the principles of open-source collaboration and the autonomy of cloud services. By thoroughly crafting tests that adhere to the stringent requirement levels as indicated by [RFC2119 keywords](https://datatracker.ietf.org/doc/html/rfc2119), we ensure that the infrastructure, services and their integrations within the SCS ecosystem are robust, secure and sovereign. + +This detailed process is crucial for stakeholders seeking to develop or utilize sovereign cloud services that are reliable, compliant and capable of standing up to the demands of modern cloud computing. It serves as an indispensable guide for developers, operators and policymakers who are involved in the creation, deployment and governance of cloud services within SCS. By understanding and implementing the standards and tests defined by the SCS, professionals can contribute to a cloud infrastructure that is not only technologically advanced but also aligns with the core values of sovereignty and open-source ethics. + +Therefore, for individuals tasked with writing tests and defining standards within SCS, this document is particularly important. It serves as a base with guidelines on how to use and understand RFC2119 keywords in the context of SCS. Its insights are crucial for anyone from technical architects to regulatory bodies within the SCS ecosystem, underlining the foundational principles necessary for achieving a sovereign cloud environment. ## 1. Understanding SCS Standards and RFC2119 Keywords @@ -26,14 +23,14 @@ SCS standards use RFC2119 keywords like: - **SHOULD NOT** (same as **NOT RECOMMENDED**), - **MAY** (same as **OPTIONAL**) -to define requirements for SCS-compatible IaaS and KaaS resources. These standards are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes. +to define requirements for SCS-compatible IaaS and KaaS resources. These standards & tests are crucial for ensuring interoperability and sovereignty in cloud services, as they are built on open-source components such as Kubernetes and OpenStack. To ensure readability and comprehensibility, only the main keywords **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT** and **MAY** are referred to below. ## 2. Interpreting RFC2119 Keywords in the Context of SCS - **MUST and MUST NOT**: These are binary and straightforward to test. Compliance or - non-compliance directly affects the standard meeting. For visualization, fulfilled + non-compliance directly affects whether the standard is satisfied. For visualization, fulfilled requirements could be marked in green, unfulfilled ones in red. - **SHOULD and SHOULD NOT**: These represent recommendations. While fulfilling these do not directly affect standard compliance, it is advised for future-proofing against @@ -45,10 +42,15 @@ To ensure readability and comprehensibility, only the main keywords **MUST**, ** ## 3. Channels for Output in Test Scripts -- Align the RFC2119 keywords with specific channels in test scripts: - - **MUST** and **MUST NOT** lead to the ERROR channel for failed tests - - **SHOULD** and **SHOULD NOT** lead to the WARNING channel, the test is still passed - - **MAY** to leads to the INFO channel, the test is still passed +In test scripts, different channels are used to convey information of different importance to the user. These channels are based on the ubiquitous and de-facto standard logging levels of common logging libraries in scripting and programming languages, in particular Python: DEBUG, INFO, WARNING, ERROR and CRITICAL. + +Alignement of the RFC2119 keywords with specific channels in test scripts: + +- **MUST** and **MUST NOT** lead to the ERROR channel for failed tests +- **SHOULD** and **SHOULD NOT** lead to the WARNING channel, the test is still passed +- **MAY** leads to the INFO channel, the test is still passed + +However, the CRITICAL level is not directly associated with any of the RFC2119 keywords; it is instead used to signify that a test was unable to complete or was interrupted due to various issues, such as runtime failures. ## 4. Compliance and Test Passing Criteria @@ -57,13 +59,27 @@ ERROR nor the CRITICAL channel. The presence of warnings or informational items **SHOULD** and **MAY** categories do not directly impact the pass/fail status but is important for overall quality and compatibility with future standards. -A test can have one of three results: +That means a test can have one of three results: - fail: if it has at least one message on ERROR - DNF (did not finish): if it has no message on ERROR, but at least one on CRITICAL - pass: otherwise -## 5. Example: Standards Document using RFC2119 Keywords +In addition, a test **MUST** exit with a non-zero exit code (e.g., via `sys.exit(…)`) if there are any ERROR or CRITICAL messages, thus signaling a failure to meet a standard. + +It is crucial to determine and explicitly state within this document the expected conventions for logging output, specifically regarding the redirection of all channel outputs to standard error (**stderr**), which aligns with Python's default logging behavior. The format which **MUST** be used is **CHANNEL(MESSAGE)**, where **CHANNEL** represents the log level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL) and **MESSAGE** encapsulates the actual log message. + +For instance, when logging an ERROR regarding a variable, the format should be as follows: + +```python +logger.error("var1 must be positive") +``` + +This example adheres to our proposed format by clearly indicating the severity level (ERROR) followed by the specific message intended for the user or developer. + +## Examples + +### 5.1 Example: Standards Document using RFC2119 Keywords The following example of a fictitious document for a "Web Server Installation Standard" is intended to illustrate the use of RFC2119 keywords in connection with the creation of @@ -100,7 +116,7 @@ This document provides a concise set of requirements for installing a secure web Failure to comply with these requirements results in the web server being considered non-compliant with the organization's security policies. -## 6. Example: Test script using RFC2119 Keywords +### 5.2 Example: Test script using RFC2119 Keywords The following example of a fictitious code for a test of "process_requirements" is intended to illustrate the use of RFC2119 keywords in connection with the @@ -154,4 +170,4 @@ def process_requirements(var1, var2): ``` -Note: The DEBUG channel is used for additional information for developers to better understand a process and does not apply to RFC2119 keywords. +**Note**: The DEBUG channel is used for additional information for developers to better understand a process and does not apply to RFC2119 keywords. From d0d275af7e77795a6134438a294731dcd3e30d6f Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Fri, 12 Apr 2024 10:59:32 +0200 Subject: [PATCH 10/11] fixup! Merge branch 'main' into General-guidelines-for-writing-test-scripts/#447 --- .../operations/tests/rfc2119-keyword-test-guide.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index 89b63b2f95..9d5f750419 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -44,7 +44,7 @@ To ensure readability and comprehensibility, only the main keywords **MUST**, ** In test scripts, different channels are used to convey information of different importance to the user. These channels are based on the ubiquitous and de-facto standard logging levels of common logging libraries in scripting and programming languages, in particular Python: DEBUG, INFO, WARNING, ERROR and CRITICAL. -Alignement of the RFC2119 keywords with specific channels in test scripts: +Alignment of the RFC2119 keywords with specific channels in test scripts: - **MUST** and **MUST NOT** lead to the ERROR channel for failed tests - **SHOULD** and **SHOULD NOT** lead to the WARNING channel, the test is still passed @@ -67,15 +67,15 @@ That means a test can have one of three results: In addition, a test **MUST** exit with a non-zero exit code (e.g., via `sys.exit(…)`) if there are any ERROR or CRITICAL messages, thus signaling a failure to meet a standard. -It is crucial to determine and explicitly state within this document the expected conventions for logging output, specifically regarding the redirection of all channel outputs to standard error (**stderr**), which aligns with Python's default logging behavior. The format which **MUST** be used is **CHANNEL(MESSAGE)**, where **CHANNEL** represents the log level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL) and **MESSAGE** encapsulates the actual log message. +It is crucial to determine and explicitly state within this document the expected conventions for logging output. In test scripts it is a **MUST** to redirect all channel outputs to standard error (**stderr**), which aligns with Python's default logging behavior. The format which **MUST** be used is **CHANNEL: MESSAGE**, where **CHANNEL** represents the log level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL) and **MESSAGE** encapsulates the actual log message. -For instance, when logging an ERROR regarding a variable, the format should be as follows: +For instance, when logging an ERROR regarding a Kubernetes cluster, the format on the command line should be like this: -```python -logger.error("var1 must be positive") +```md +INFO: Checking cluster specified by context 'my-cluster' in /path/to/.kube/config. ``` -This example adheres to our proposed format by clearly indicating the severity level (ERROR) followed by the specific message intended for the user or developer. +This example adheres to our proposed format by clearly indicating the severity level (INFO) followed by the specific message intended for the user or developer. ## Examples From ea3f39c40c0fbb9101edd791194c5e2352d713df Mon Sep 17 00:00:00 2001 From: cah-patrickthiem Date: Fri, 12 Apr 2024 12:24:19 +0200 Subject: [PATCH 11/11] fixup! Merge branch 'main' into General-guidelines-for-writing-test-scripts/#447 --- .../operations/tests/rfc2119-keyword-test-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md index 9d5f750419..79a8f396c8 100644 --- a/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md +++ b/contributor-docs/operations/tests/rfc2119-keyword-test-guide.md @@ -67,9 +67,9 @@ That means a test can have one of three results: In addition, a test **MUST** exit with a non-zero exit code (e.g., via `sys.exit(…)`) if there are any ERROR or CRITICAL messages, thus signaling a failure to meet a standard. -It is crucial to determine and explicitly state within this document the expected conventions for logging output. In test scripts it is a **MUST** to redirect all channel outputs to standard error (**stderr**), which aligns with Python's default logging behavior. The format which **MUST** be used is **CHANNEL: MESSAGE**, where **CHANNEL** represents the log level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL) and **MESSAGE** encapsulates the actual log message. +In test scripts it is a **MUST** to redirect all channel outputs to standard error (**stderr**), which aligns with Python's default logging behavior. The format which **MUST** be used is **CHANNEL: MESSAGE**, where **CHANNEL** represents the log level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL) and **MESSAGE** encapsulates the actual log message. -For instance, when logging an ERROR regarding a Kubernetes cluster, the format on the command line should be like this: +For instance, when logging an INFO regarding a Kubernetes cluster, the format on the command line should be like this: ```md INFO: Checking cluster specified by context 'my-cluster' in /path/to/.kube/config.