[BUG] SKIPPED tests seem to still be added to overall score results

[agentpoyo]

Describe the bug
It seems that the SKIPPED tests are still being added in the total score results when running the CNF Test Suite.

When running a test, the total output displays a score total of 76 points but the final result states 86 points total. (see screenshot)

To Reproduce
Steps to reproduce the behavior:

1. Follow common install steps to cnf-conformance
2. Setup and use the sample-coredns-cnf cnf-conformance cnf_setup cnf-config=./sample-cnfs/sample_coredns_cnf/cnf-conformance.yml
3. Then run a workload by doing LOG_LEVEL=error ./cnf-conformance workload
4. See output

Expected behavior
The total points for SKIPPED tests should be removed from the totals in the final results.

These should also provide more information if SKIPPED. Perhaps 0 points of possible 10 points not counted.

Screenshots

Device (please complete the following information):

• OS Linux
• Distro Ubuntu
• Version 18.04
• Architecture x86

[wvwatson]

@sishbi

[agentpoyo]

This one kind of leans and depends on https://github.com/cncf/cnf-conformance/issues/707

[williscool]

This one kind of leans and depends on https://github.com/cncf/cnf-conformance/issues/707

@agentpoyo just noticed this. had been working on code for the solution. have you guys talked about this in a pairing session?

@wavell @denverwilliams ?

[wvwatson]

Planning
https://hackmd.io/6lp0LeL5S06FBr32PVYy1w @taylor

[HashNuke]

Verified issue with result score calculation.

• Workload test says 149 of 296
• Total points in the results yml file is 159 ((5pts * 30) + (10pts * 2) + (-1pts * 11) = 159)

The total score is 10 points less than what it should be. Screenshot below.

[HashNuke]

The screenshot below are some notes from the workload test that I was running to validate the issue.

• I made note of the tests that are failing or skipped.
• Cross-verified the output on the console with the results yaml file.

Created tickets for any issues that were found during the validation exercise.

• [BUG] cluster-tools in default namespace interferes with test results #1199
• [BUG] privileged and privileged_containers tests have very similar pass messages #1200
• [BUG] database_persistence and elastic_volumes test have very similar success and failure messages #1201

[HashNuke]

Verified the points assigned to all tests and the behaviour for skipped tests. Compared the output on the console with the points assigned in the results yaml file.

• Skipped tests that are prototypes or destructive are not listed in the results yaml file. I think they should be listed appropriately and marked as zero like the other skipped tests
• ip_addresses test assigns 0 points for passing test. Will verify with the team that this is ok.

[HashNuke]

Next: Will look into how the test score is calculated when outputting it on the console.

[HashNuke]

• Found two tests that are not defined in the points.yml file: insecure_capabilities and dangerous_capabilities
• This accounts for the 149->159 difference.

[HashNuke]

Found 7 skipped tests amounting to 40 points being included in the total maximum score.

non_root_user: 5
secrets_used: 5
prometheus_traffic: 10
open_metrics: 5
routed_logs: 5
tracing: 5
node_drain: 5

[agentpoyo]

Acceptance Criteria:

• when running workload, I should see skipped tests not adding the overall points for categories.
• I can see an example output here of skipped tests not adding to points total.

[agentpoyo]