Add general guidelines for writing test scripts regarding RFC 2119 Ke…

[cah-patrickthiem]

This PR adds a document for general guidelines in regard of writing test scripts, especially on how to use RFC2119 keywords.

Resolves: SovereignCloudStack/standards#447

[martinmo]

@cah-patrickthiem is the text ready for review from your point of view or are you still working on it? If you're finished feel free to request a review from me :-)

I already have a few remarks/suggestions:

• Your change appears to be very out of sync with the current state in main: contributor-docs/ should be used instead of dev-docs/.
• Did you discuss adding the numbering scheme to the subfolders? I am asking because this might break existing links.
• To remain consistent with the other files in this repo, RFC2119-keyword-test-guide.md should be completely lower case (slug format).

[mbuechse]

Not a real review, just a remark and a request: once these guidelines are merged, please add a link to https://github.com/SovereignCloudStack/standards/blob/main/.github/ISSUE_TEMPLATE/stabilize-standard.md?plain=1#L15

[cah-patrickthiem]

@cah-patrickthiem is the text ready for review from your point of view or are you still working on it? If you're finished feel free to request a review from me :-)

I already have a few remarks/suggestions:

• Your change appears to be very out of sync with the current state in main: contributor-docs/ should be used instead of dev-docs/.
• Did you discuss adding the numbering scheme to the subfolders? I am asking because this might break existing links.
• To remain consistent with the other files in this repo, RFC2119-keyword-test-guide.md should be completely lower case (slug format).

You are completely right here. The branch is out of sync. Also, the subfolders were just a quick catch of me, I understand that this needs to be discussed, but right now I also think this is not necessary.
I will change and fix the things you suggested since I wanted to tackle this anyway.

[cah-patrickthiem]

Pushed a !fixup to address several inconsistensies. @martinmo the text itself is ready for review, feel free to check it. Also, if you want to add more content regarding tests, feel free to do so.

[mbuechse]

After the remarks I just made, I have a suspicion that these docs may actually become a standard one day. But it's fine to start out with docs.

[martinmo]

I reviewed it and left some suggestions for some minor issues I have found. Overall this looks really promising and we should get this merged soon 👍

[martinmo]

One more thing, because I just read @mbuechse's comment #154 (comment) again.

We should definitely discuss whether to specify in the document that all channels output should go to stderr (for Python logging, this is the default) and what kind of format we expect (<CHANNEL>: <MESSAGE>).

I'm in favor of putting it into this document.

[cah-patrickthiem]

One more thing, because I just read @mbuechse's comment #154 (comment) again.

We should definitely discuss whether to specify in the document that all channels output should go to stderr (for Python logging, this is the default) and what kind of format we expect (<CHANNEL>: <MESSAGE>).

I'm in favor of putting it into this document.

Added this as well.

[martinmo]

It is crucial to determine and explicitly state within this document the expected conventions for logging output

This sounds like a prompt for the reader to determine the conventions and should be shortened to something like "the scripts are expected to ...".

Instead of CHANNEL(MESSAGE) the actual format we use is CHANNEL: MESSAGE. Example:

INFO: Checking cluster specified by context 'my-cluster' in /path/to/.kube/config.

[cah-patrickthiem]

done

[martinmo]

With regard to my comment:

We should definitely discuss whether to specify in the document that all channels output should go to stderr

I double checked with the existing scripts in the standards repo. Some scripts explicitly set the logging stream to sys.stdout (example). Also, the "umbrella script" scs-compliance-check.py supports both stdout and stderr.

Maybe we should leave the stderr part out and allow both. Otherwise, we need to adapt the affected scripts to conform to this new guideline :-) @mbuechse do you have a strong opinion here?

[mbuechse]

Maybe we should leave the stderr part out and allow both. Otherwise, we need to adapt the affected scripts to conform to this new guideline :-) @mbuechse do you have a strong opinion here?

I wouldn't say strong, but I think the bit with stdout is a mistake, and stderr should be a recommendation at the very least.

[cah-patrickthiem]

Maybe we should leave the stderr part out and allow both. Otherwise, we need to adapt the affected scripts to conform to this new guideline :-) @mbuechse do you have a strong opinion here?

I wouldn't say strong, but I think the bit with stdout is a mistake, and stderr should be a recommendation at the very least.

Fyi I talked to @martinmo and we came to the conclusion to make "stderr" a MUST for test scripts.

[martinmo]

LGTM!