Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Replace difftest model with Quint=>MBT #1369

Merged
merged 117 commits into from
Dec 18, 2023
Merged

test: Replace difftest model with Quint=>MBT #1369

merged 117 commits into from
Dec 18, 2023

Conversation

p-offtermatt
Copy link
Contributor

@p-offtermatt p-offtermatt commented Oct 19, 2023

Description

Closes: #1241

Changes:

  • Changes to the Quint model: bug fixes, things that make it easier to use to generate traces, ...
  • Removes the difference tests in tests/difference
  • Adds tests/mbt
  • Adds a driver, which uses ibctesting to take Quint traces and run them against provider and consumer apps
  • Adds infrastructure to generate traces from the Quint model
  • Adds the mbt tests to the CI
  • Adds a README explaining the MBT approach, how to use, etc
  • Made some changes to the Quint model. There are 2 types of changes: Making the traces more useful for MBT (doing a change amount instead of new voting power for VotingPowerChange, keeping the parameter values in a variable, ...) and making the model actually correct (splitting timestamps into running and last timestamps, ...)

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • Included the correct type prefix in the PR title
  • Targeted the correct branch (see PR Targeting)
  • Provided a link to the relevant issue or specification
  • Reviewed "Files changed" and left comments if necessary
  • Confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • Confirmed the correct type prefix in the PR title
  • Confirmed all author checklist items have been addressed
  • Confirmed that this PR does not change production code

@github-actions github-actions bot added the C:Testing Assigned automatically by the PR labeler label Oct 19, 2023
@p-offtermatt p-offtermatt linked an issue Dec 14, 2023 that may be closed by this pull request
Copy link
Contributor

@MSalopek MSalopek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any glaring issues with the driver.

LGTM.

tests/mbt/driver/core.go Outdated Show resolved Hide resolved
tests/mbt/driver/core.go Outdated Show resolved Hide resolved
tests/mbt/driver/README.md Outdated Show resolved Hide resolved
tests/mbt/run_invariants.sh Outdated Show resolved Hide resolved
tests/mbt/driver/core.go Outdated Show resolved Hide resolved
// delivering the packet should give an error
// if the consumer is timed out in the model
var expectError bool
if ConsumerStatus(currentModelState, consumerChain) == "timedout" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it'd be helpful to declare the consumer status as constant.

@sainoe sainoe self-requested a review December 18, 2023 10:22
Copy link
Contributor

@sainoe sainoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this big work @p-offtermatt. My review of the Quint code doesn't carry much weight as I'm not yet very familiar with the model yet. Apart from tiny nits, the driver code LGTM and is easier to read than the diff testing!

@p-offtermatt p-offtermatt added this pull request to the merge queue Dec 18, 2023
Merged via the queue into main with commit 6a95c49 Dec 18, 2023
17 of 18 checks passed
@p-offtermatt p-offtermatt deleted the ph/mbt branch December 18, 2023 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C:Build Assigned automatically by the PR labeler C:CI Assigned automatically by the PR labeler C:Testing Assigned automatically by the PR labeler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Run Quint tests in the CI Write a shim/driver that handles Quint-generated traces
4 participants