Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test run output #139

Open
5 of 10 tasks
cgay opened this issue Jan 8, 2021 · 4 comments
Open
5 of 10 tasks

Improve test run output #139

cgay opened this issue Jan 8, 2021 · 4 comments
Labels

Comments

@cgay
Copy link
Member

cgay commented Jan 8, 2021

I'd like to see several improvements to Testworks output:

  • An option to display no progress output except for failing tests. --progress failures I suppose. i.e., don't display pass, not implemented, expected to fail.
  • It should be rare that tests do output to the console. When they do it's generally for debugging. (threads-test-suite stands out as doing a lot of essentially meaningless output.) All such debug output should be via test-output:testworks and should by default go to test-temp-directory()/test-name.log and there can be an option to also send it to the console. --test-output=all, for example...not sure of name.
  • Display test run time as a number of microseconds, or heuristically ms, usec, ns. My time library already does this so perhaps wait and use that. Seconds as the unit is just too big these days.
  • Always display the final summary output. Specifically, with the options that are passed to make check (--report json --report-file foo.json) I still want the final count of passing/failing tests to be shown on the terminal.
  • No need to display passing/failing suites. I've never found this remotely useful.
  • In the summary output, display the counts for tests last, after checks, because that last line should have the info I really care about.
  • Don't display "0 benchmarks". (This might might already be the case?) Our plan is to run benchmarks as separate libraries.
  • Possibly: note which tests are new after each --load so they can be associated with their .so file and we can execute them in that same order and display the .so filename before any progress output for that set of tests. (The output from make check is currently pretty random and hard to know what tests belong to what library.)
  • Can probably elide "Running test " before each test in the progress output. Just print the test name there.
  • Can greatly improve output for test-equal failures:
    test-encode/decode-varint failed
        encode-varint-bytes(i) = bytes: [{stretchy vector 128, 128, 128, 128, 248, 255, 255, 255, 255, 127} and {stretchy vector 3} are not =.  sizes differ (10 and 1), element 0 is the first non-matching element]
          FAILED in 0.002164s and 58KiB
    
    should become something like this
    test-encode/decode-varint: values differ starting at element 0
        encode-varint-bytes(i): size=10 {stretchy vector 128, 128, 128, 128, 248, 255, 255, 255, 255, 127}
        bytes:                  size=1 {stretchy vector 3}
    

Those are my thoughts. We'll see how it develops.

@cgay cgay added the output label Jan 8, 2021
@housel
Copy link
Member

housel commented Jan 8, 2021

Don't display "0 benchmarks", and if there are no tests (because it's a benchmark suite) then don't display "0 tests". The two cases can be symmetric.

cgay added a commit to cgay/testworks that referenced this issue Jan 17, 2021
* Normally only display the test count OR the benchmark count.
  If they're both 0, both are displayed.
* Don't display the number of suites that passed/failed.
* Display test/benchmark counts after checks instead of before.

part of dylan-lang#139
cgay added a commit to cgay/testworks that referenced this issue Jan 17, 2021
* Normally only display the test count OR the benchmark count.
  If they're both 0, both are displayed.
* Don't display the number of suites that passed/failed.
* Display test/benchmark counts after checks instead of before.

part of dylan-lang#139
@cgay
Copy link
Member Author

cgay commented Nov 27, 2021

None of this makes sense:

$ ../_build/bin/time-test-suite --test test-load-tzif-version-1
Running test test-load-tzif-version-1: NOT IMPLEMENTED in 0.000822s and 68KiB

test-load-tzif-version-1 not implemented

Ran 0 checks: PASSED
Ran 1 test: PASSED (1 not implemented)
NOT IMPLEMENTED in 0.000822 seconds
  • Huge amounts of duplication. I don't need to be told 4 times that the test is not implemented.
  • "Ran 0 checks: PASSED" should not say PASSED. Either say NOT IMPLEMENTED or just leave it blank.
  • "Ran 1 test: PASSED (1 not implemented)" -- which is it? passed or not implemented?
  • "NOT IMPLEMENTED in 0.000822 seconds" -- Strange wording. Avoid the issue by saying "Result: NOT IMPLEMENTED, Total run time: 822µs"

@cgay
Copy link
Member Author

cgay commented Oct 29, 2022

I still find this output for failed checks to be pretty bad:

FAILED: $expected-failure = test-results.result-status [#"expected-failure" and #"unexpected-success" are not =.]

I think better would be

FAILED: #"expected-failure" ~= #"unexpected-success" for assert-equal($expected-failure, test-results.result-status)

Or even better, a multi-line display:

FAILED: assert-equal($expected-failure, test-results.result-status)
    Expected: #"expected-failure"
    Got: #"unexpected-success"

cgay added a commit to cgay/testworks that referenced this issue Sep 30, 2023
* Use multi-line output with "want" and "got" values, and detailed output if
  present, on separate lines.
* Don't sort detail results since they aren't guaranteed to be sortable.
* Separate do-check-equal into same plus do-check-not-equal; it's less
  complex.

dylan-lang#139
cgay added a commit to cgay/testworks that referenced this issue Oct 1, 2023
* Use multi-line output with "want" and "got" values, and detailed output if
  present, on separate lines.
* Don't sort detail results since they aren't guaranteed to be sortable.
* Separate do-check-equal into same plus do-check-not-equal; it's less
  complex.

dylan-lang#139
cgay added a commit to cgay/testworks that referenced this issue Oct 13, 2023
* Use multi-line output with "want" and "got" values, and detailed output if
  present, on separate lines.
* Don't sort detail results since they aren't guaranteed to be sortable.
* Separate do-check-equal into same plus do-check-not-equal; it's less
  complex.

dylan-lang#139
@cgay
Copy link
Member Author

cgay commented Jun 12, 2024

Current assert-equal output now looks like this:

FAILED: lox-test-suite
  FAILED: test-eval-precedence
    FAILED: 7 = %eval("2*(3+4)/3;")
      want: 7
      got:  4.6666667d0
  FAILED: test-eval-program
    FAILED: #[1, 2, 3, 4] = #[1, 2, 5, 4]
      want: #[1, 2, 3, 4]
      got:  #[1, 2, 5, 4]
      detail: element 2 is the first mismatch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants