Custom repos in benchpark

[scheibelp]

Define a repos.yaml like:

repos:
  experiments:
  - ../experiments
  - ../experiments/test
  # and systems, applications, and packages, see config/repos.yaml added in this PR for full example

in <benchpark_root>/config to customize where experiment definitions are pulled from.

Benchpark can pull config from one location, with the following priority (highest first):

• benchpark -C <dir>...
• If CWD where you call benchpark has benchpark-config directory
• If <benchpark_root>/config is a directory (this must exist if the first two don't)

More generally:

• Add a rudimentary YAML-based config system to benchpark
• Define a Repos section
  • Benchpark now uses this repos config to instantiate experiments and systems repos for benchpark itself, and packages/applications repos for Spack/Ramble
• Accommodates other possible sections
  • These sections support property-like access in python (but that assume the property exists, which they all should in the case of repos config)

TODOs

• several files are called repos.yaml or repo.yaml w/ different purposes, maybe I should call this new one something different
• schema validation
• Consolidate with benchpark bootstrapping config (currently gets consumed via a different mechanism, lives in a file in configure.yaml)

[michaelmckinsey1]

From our discussion

Regarding the benchpark configure comand - take benchpark.yaml rename it to bootstrap.yaml benchpark configure will write to the config directory.

[michaelmckinsey1]

(1) I think we need docs to instruct users to add the path to their experiments/systems/repo to config/repos.yaml. And that if they wish for their addition to be used over the existing version, it should go first in the config.

(2) Can you merge develop and run a gitlab pipeline?

[michaelmckinsey1]

Some docs edits

[michaelmckinsey1]

Good now

[codecov-commenter]

Codecov Report

❌ Patch coverage is 83.70787% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.45%. Comparing base (e2fef44) to head (5f2e26e).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
lib/main.py	60.86%	9 Missing ⚠️
lib/benchpark/config.py	88.70%	7 Missing ⚠️
lib/benchpark/base_paths.py	85.18%	4 Missing ⚠️
lib/benchpark/cmd/mirror.py	25.00%	3 Missing ⚠️
lib/benchpark/cmd/system.py	33.33%	2 Missing ⚠️
lib/benchpark/cmd/unit_test.py	33.33%	2 Missing ⚠️
lib/benchpark/accounting.py	80.00%	1 Missing ⚠️
lib/benchpark/cmd/info.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1190      +/-   ##
===========================================
+ Coverage    63.70%   64.45%   +0.75%     
===========================================
  Files           46       48       +2     
  Lines         3507     3618     +111     
  Branches       265      273       +8     
===========================================
+ Hits          2234     2332      +98     
- Misses        1264     1277      +13     
  Partials         9        9              

Files with missing lines	Coverage Δ
experiments/test/saxpy/experiment.py	50.00% <ø> (ø)
lib/benchpark/cmd/setup.py	88.48% <100.00%> (+1.08%)	⬆️
lib/benchpark/cmd/show_build.py	21.42% <100.00%> (ø)
lib/benchpark/directives.py	71.07% <ø> (-0.24%)	⬇️
lib/benchpark/paths.py	100.00% <100.00%> (+14.28%)	⬆️
lib/benchpark/repo.py	73.84% <100.00%> (+1.53%)	⬆️
lib/benchpark/runtime.py	92.10% <100.00%> (ø)
lib/benchpark/test/caliper.py	97.43% <100.00%> (ø)
lib/benchpark/test/commands.py	100.00% <100.00%> (ø)
lib/benchpark/test/paths.py	100.00% <100.00%> (ø)
... and 8 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.