Plugins #17
Merged
Plugins #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Didn't even notice these two concerns were tangled Also disables the timeout plugin for now, because that's a huge perf hog
It's a bug that it only tests timeouts per func and not per "whole test"
The results plugin should swallow these and figure out what to do
So, uhh, fixes #16
build-test-actions: * have a userFunction builder with subordinate test/hook ones * pass in arg that says whether they should immediately error * get the logging funcs out of here altogether results plugin: * userFunction immediately prints errors if the action has !belongsToTest * suite failures should immediately print, mark overall failing, still pop
This is totally the right place to do this.
Before any hook failures were printed above a test, even if (as in the case of a beforeEach) they logically belong as part of the test. Now any hook failures are printed at the test-level, right at the same time that OK or NOT OK is printed
But it seems like it should behave sorta similar instead of being fatal #NoTest
This bug was caught by Standard JavaScript™ c/o @feross
In general afterAll should be run even if there was a same-nesting-level error. afterEach should be run if there was a same-nesting-level error and it was a test-specific error
In this case, afterAll should run but not the test or any subordinate tests. In the case of outright-skipped test functions we print "SKIPPED"
This was certainly one way to make sure the right hooks are firing when we want. Particularly nasty is the use of `distanceFromTest` as a crude instrument to determine whether an afterEach is at a deeper depth than an earlier error (to avoid running them unnecessarily). Just ugly as hell.
* Test the logging for suite-wide failures (which really ought only be possible if you have a plugin triggering them) * Expose the file name for suites (and the glob itself for the top-level suite)
Not sure how this happened, but the nesting levels introduced in 73b4236 were counting beforeEach's completely backwards
We're effectively mucking with global-state so this will call it out and hopefully make it addressable for use by other plugins.
Thinking about a scheme for uniquely identifying tests & suites in order to call back the results of each
This is the first green commit in like 40 commits. Wow.
Basically I've built a teenytest to test teenytest Great.
This required me to inline the done plugin sadly. Oh well!
(Disabling the result-value test while we refactor) Right now the word "result" in this codebase means "test & suite results", but a lot of useful userFunction plugins will want to know what the return/result value of a given userFunction was. Moreover, some plugins might want to set their own result value for a particular userFunction as a means of intra-plugin communication if they register multiple lifecycle hooks This will require yet another "store" object, with its own statefulness, and 4 of these managed ad hoc is just too much to not make an error at some point, so next up we should refactor this into a common pattern.
There was way too much state floating around previously through 3 different "store" modules, each with their own initialization and root data structure. The potential for leaking state across multiple runs was far too great to leave this as-was, especially given the need for additional state to be stored. This means that all non-functional state is always reset when the test runner is kicked off. It also means that users will not be permitted to register plugins except when the runner deems it appropriate, which is a little bit inconvenient, but we've given users 3 ways of configuring plugins, so hopefully that's enough.
This is a little nutty, but every time a userFunction runs, its store
will persist a structure like this:
{
value: <whatever the user func returned or calledback>
error: <whatever the user func threw or calledback>
plugins: {
somePluginName: {
value: <whatever the somePluginName called back>
error: <whatever the somePluginName errored>
}
}
}
This was enabled by ca4f6e2 In hindsight, the error store only ever existed to tangle up the user function results with the test runner results, so now they're more distinct
This will allow plugins to know the values set by the user's defined functions as well as any plugins
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This has been my hobby project for the last week. Still pretty half baked.
The tl;dr is that this PR rewrites teenytest to implement most of its user-facing logic into plugins (docs). The internal plugins are all defined here and loaded by the runner here.
This is a pretty drastic detangling of all the runner-time logic teenytest contained before, and should allow us to do things like have user-defined plugins which transform test functions like #14 with promises or a first-class reporter plugin (in lieu or in addition to the TAP13 one) as #15 wants.
$ teenytest --config some/requireable/pathwhich would in turn export a function registering what it needs with teenytest, and invoked by teenytest's runnerresultswill change how subsequent wrappers get called