Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENACE] Clarify usage and behavior of testing multiple CNFs #2095

Open
martin-mat opened this issue Jul 1, 2024 · 3 comments
Open

[MAINTENACE] Clarify usage and behavior of testing multiple CNFs #2095

martin-mat opened this issue Jul 1, 2024 · 3 comments

Comments

@martin-mat
Copy link
Collaborator

martin-mat commented Jul 1, 2024

Currently, it is possible to run cnf_setup multiple times for multiple CNFs/helm charts. When running tests then, tests are executed on each configured CNFs.
Clarify whether this is a supported feature. Currently, there is no documentation nor tests of this behavior.
If this should be supported behavior, address those aspects:

  1. Add this to documentation
  2. Add spec test(s) to verify expected functionality
  3. Fix behavior of generating result yaml files. Currnently, just one yaml file is being generated; results of particular test are overwritten, and it records result only of the latest CNF. Which if obviously not expected behavior. Right behavior needs to be figured out and implemented.

If this is not supported usage, don't allow to call cnf_setup multiple times.

@martin-mat
Copy link
Collaborator Author

#2090

@martin-mat
Copy link
Collaborator Author

My input for the discussion/decision:

I propose not to have support for this. Reasons:

  • The value of this feature is very low: The same results can be obtained by calling setup/test/cleanup in series with almost no duration difference.
  • That feature creates confusions by usage by mistake. Users sometimes call multiple cnf_setup with different CNFs and then they are confused by results.
  • That feature would require resources for documentation, testing and maintenance. That resources can be used better.
  • Let's keep things simple.

@Smitholi67
Copy link
Collaborator

+1 for not supporting - let us keep it simple.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants