Conformance Testing Setup for Shim #134
Conversation
kate-goldenring
force-pushed
the
conformance-tests
branch
from
5b3abd1
to
9744ac0
Compare
|
@rylev for visibility |
|
@Mossaka indeed the idea is to add this to CI, but I think we'll wait until we're confident that this can run more quickly and reliably than it currently can. The issue is that this requires pushing to an OCI registry, and we're currently using ttl.sh which is probably not a viable strategy. @kate-goldenring What do you think about merging this as is? I'm fine keeping this as a draft PR until we add this to CI (and I can just keep making PRs to your branch), but it also might be easier to just merge and make PRs directly to this repo instead of your fork. |
kate-goldenring
requested review from
jsturtevant,
radu-matei and
devigned
as code owners
kate-goldenring
changed the title
|
Need a rebase |
Signed-off-by: Kate Goldenring <[email protected]>
Signed-off-by: Ryan Levick <[email protected]>
Signed-off-by: Ryan Levick <[email protected]>
Signed-off-by: Ryan Levick <[email protected]>
Signed-off-by: Ryan Levick <[email protected]>
Signed-off-by: Ryan Levick <[email protected]>
Signed-off-by: Kate Goldenring <[email protected]>
kate-goldenring
force-pushed
the
conformance-tests
branch
from
2f973d3
to
9c09de2
Compare
|
For now, I am skipping the TCP tests as they are failing #147 |
Signed-off-by: Kate Goldenring <[email protected]>
kate-goldenring
force-pushed
the
conformance-tests
branch
from
9c09de2
to
cecce6c
Compare
conformance-tests/src/main.rs
Outdated
| .next() | ||
| .expect("expected second arg to be path to ctr binary"); | ||
|
|
||
| 'test: for test in conformance_tests::tests_iter(&tests_dir).unwrap() { |
There was a problem hiding this comment.
Is the reason you're not using conformance_tests::run_tests because the tests run in parallel? Should we provide an API that allows the tests to not run in parallel?
That API is preferred since it gives better test output based on libtest-mimic.
There was a problem hiding this comment.
I wasn't aware of the run_tests API being added. I can change to use that.
There was a problem hiding this comment.
Running them in parallel we run into issues concurrently pushing to the local registry
There was a problem hiding this comment.
Okay, got them running in parallel. There was an issue with the registry service not coming up -- restarting docker resolved it. Commit c3a0a58 leverages run_test and updates these tests to configure the port for the Spin app and generate a random hash for the run ID to enable running the tests in parallel.
kate-goldenring
force-pushed
the
conformance-tests
branch
from
c3a0a58
to
61b58a1
Compare
…allel - Set unique port for each run - Set unique ctr run ID for each run Signed-off-by: Kate Goldenring <[email protected]>
kate-goldenring
force-pushed
the
conformance-tests
branch
from
61b58a1
to
5e4ac59
Compare
Spin has created a set of conformance test to ensure all Spin runtimes conform to baseline requirements/expectations
Status: runs conformance test to successful termination 🎉
This is very draft: