-
Notifications
You must be signed in to change notification settings - Fork 5
/
to-do.txt
98 lines (71 loc) · 3.72 KB
/
to-do.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
* Testworks laundry list, from
http://bugzilla.opendylan.org/show_bug.cgi?id=7472
+ TDD support -- specifically, a way to mark tests as "expected to
fail" so that it's feasible to commit tests for code that is not
yet written.
+ Better selection/filtering -- for example, a way to mark a test as
"commit", "regress", "pound", and a way to run a subset of those
tests. (Perhaps have a general mechanism for adding attributes to
tests and then selecting tests based on those attributes.)
+ Support for parallel execution of suites and/or tests. For
example, by running subsets of tests in different threads or
processes. (There needs to be a way to mark a test as not being
parallelizable.)
+ Call the teardown method if there's a failure in the setup method.
(I don't know whether Testworks does this or not, actually.)
+ Ability to (optionally) generate a per-test log file.
+ Better reporting -- e.g., subunit/xunit output formats
+ Don't crap all over the screen during a test run; use a log.
stdout should just display progress. (There could be an option
to have the current behavior.)
+ Show -progress by default, and show it BEFORE starting each test.
Show timing for each test as well.
+ Ability to parse previous test runs and compare to find
regressions. (Implies a standard place to store results, probably
svn.)
+ Ability to re-run only the failures in a previous test run.
+ Benchmark support should be integrated. e.g., just capture the
time it takes for every test and add it to the report rather than
defining separate benchmarks. Tracking test timings could be an
important way to find performance regressions.
+ Add new assert-* macros that are similar to the existing check-*
macros but the description is optional. Although the description
can be a good thing, often it's unnecessary and redundant. For
example, if there's only one assertion in the test then it's
obvious which one failed. Assertions can also be auto-numbered
within a test and that number displayed in the error message.
When the assertion is in a loop, a message should be supplied by
the author, and we'll rely on their good taste to do that.
+ Add config file support. It should be possible to use a config
file to specify most common options and then override those
options on the command-line where needed.
+ These two additional restarts should always be available:
* Skip the remainder of this test
* Skip the remainder of this suite
+ I really want hierarchical test names like suite1.suite2.test1
It's very convenient for UI purposes.
+ The testworks-specs macros drive me crazy because it's hard to
tell/remember what the names of the tests they generate are.
+ ``assert-every(f, collection)`` (and assert-none) might be useful.
There are a lot of tests in apply-dylan-test-suite that do
check-true("", every?(...)). This could improve the failure
message significantly.
+ Ability to define tests and suites within another suite::
define suite top ()
suite subsuite-1;
suite subsuite-2 ()
test ...;
end suite subsuite-2;
test test-1;
test test-2 ()
assert-equal(2, 1 + 1);
end test test-2;
end suite top;
This would be backward compatible with current define suite.
+ check/assert-equal could display a diff for any collection class.
* Bruce Mitchener's wish list:
+ For one, I'd like an HTML report with some fancy JS stuff in it to
make it easy to figure things out
+ It should be documented how to diff reports
+ We should have some sort of archive of them for released builds on
various platforms.