Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: Make example tests smarter #328

Open
matevz opened this issue Dec 3, 2024 · 0 comments
Open

cli: Make example tests smarter #328

matevz opened this issue Dec 3, 2024 · 0 comments
Assignees
Labels
ci Related to continuous integration tests Related to testing and tests

Comments

@matevz
Copy link
Member

matevz commented Dec 3, 2024

We have way too much .static examples meaning the output is not actually tested inside CI. I suggest we get rid of using .static for content that changes often and try to make a "smarter" diff.

For example, recent regression caused that ParaTime names got lost:

$ oasis network status
=== NETWORK STATUS ===
Network:      localhost_testnet
Node's ID:    5MsgQwijUlpH9+0Hbyors5jwmx7tTmKMA4c9leV3prI=
Core version: 24.3

==== Consensus ====
Status:               syncing
Version:              7.0.0
Chain context:        0b91b8e4e44b2003a7c5e23ddadb5e14ef5345c0ebcb3ddcae07fa2f244cab76
Latest height:        24115729 (2024-11-28 08:29:38 +0100 CET)
Latest block hash:    b7a9ad2fe687b9b2887512b03707347b0ec743020a175bc45c09979e3cc90d82
Latest epoch:         40176
Is validator:         false
Registration:         false

==== ParaTimes ====
  Status:               waiting for consensus sync
  Latest round:         7933058 (2024-11-28 08:29:38 +0100 CET)
  Last finalized round: 18446744073709551615
  Storage status:       initializing
  Number of peers:      23
  Status:               waiting for consensus sync
  Latest round:         8930789 (2024-11-28 08:29:38 +0100 CET)
  Last finalized round: 18446744073709551615
  Storage status:       initializing
  Number of peers:      22
  Status:               waiting for consensus sync
  Latest round:         9219372 (2024-11-28 08:29:38 +0100 CET)
  Last finalized round: 18446744073709551615
  Storage status:       initializing
  Number of peers:      23

We should ignore the latest height, block hash and epoch in this case, but still test everything else.

Perhaps a kind of a regex "mask" per example that should define how the output should look like. So when you run "make examples" and the regex is matched, then don't generate the new .out files.

@matevz matevz self-assigned this Dec 3, 2024
@matevz matevz added tests Related to testing and tests ci Related to continuous integration labels Dec 3, 2024
@matevz matevz assigned abukosek and unassigned matevz Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Related to continuous integration tests Related to testing and tests
Projects
None yet
Development

No branches or pull requests

2 participants