🚀 Feature: Add option to find the call site of a test (file name and location)

[nicojs]

Is your feature request related to a problem or a nice-to-have?? Please describe.
I would love a way to discover where tests live inside the files. Could we attach this metadata to a Test, or Runnable?

runner.on(EVENT_TEST_BEGIN, function(test) {
  const { fileName, column, line } = test.getCallSite(); 
  //      ^ string  ^ numbers ^
});

Describe the solution you'd like
The only generic way I can think of how to discover the location of the caller of our it or test functions is by using new Error().stack. This might come with a performance penalty, so we might want to add it behind a new configuration option. Similar to Jest's --testlocationinresults.

Describe alternatives you've considered

This workaround works:

const original = Mocha.Suite.prototype.addTest;
Suite.prototype.addTest = function addTest(test) {
  const suite = original.call(this, test);
  test.callSite = getCallSite(new Error()); // magic of parsing the stack
  return suite;
};

There are numerous libraries available that can parse the stack for us. Jest and tabjs are using stack-utils.

Additional context

You might have guessed it, but we want to use this new callSite to improve Stryker's reporting and performance 😅

[nicojs]

Let me do a proposal.

I want to create and maintain a good package that provides you with the most accurate location of the call site. I've already started it here: https://github.com/nicojs/call-id. It will support all actively used browsers (as long as sauce labs or BrowserStack allows me to test them). It can be used as an ESM or CJS module, so it should be easily bundled in browser packages.

I want to implement this feature in both here in mocha and jasmine (jasmine/jasmine#1884). What do you think?

[JoshuaKGoldberg]

@nicojs is this still relevant given stryker-mutator/stryker-js#3504?

[nicojs]

Hi Josh 🙋‍♂️. It's very cool to see you here! 😄

@nicojs is this still relevant given stryker-mutator/stryker-js#3504?

Yes, it's still very relevant. Since we only know in which file the test was declared, not its location.

1. This results in a plain list of tests right above the source code. Not very elegant IMO.
   
   See https://dashboard.stryker-mutator.io/reports/github.com/stryker-mutator/stryker-js/master?module=core#test/concurrent/resource-decorator.spec.js
2. We use the test location to implement incremental mode. This is a feature where Stryker tries to detect which tests have changed since a previous run. Since we currently don't know the exact location of a mocha test, we assume every test changed when a test file changed, which is bad for performance.

About the implementation. There are a couple of open questions:

1. Since the only way to get the location of the call site is to throw an error and catch it, it would add a linear performance penalty. Would this be acceptable?
   • We could make it configurable, like jest's --testlocationinresults.
2. Users that run Mocha tests often transpile to JS before running the tests (for example, using esbuild or tsc). Is resolving source files part of this feature?
   • I think not. Resolving source maps is a lot of work. There are also good solutions out there, like --enable-source-maps or source-map-support.
3. Would you be willing to add call-id as a dependency of mocha?
4. You've marked this issue with area: nodejs. This feature could also work in the browser. That was my goal originally with call-id.
   • Which brings me to: what browsers does Mocha need to support?

[JoshuaKGoldberg]

Ok! Sorry for taking so long to answer @nicojs. The rest of the team has been quite busy and I didn't want to run roughshod over everyone when triaging nuanced issues like this. But it's been almost a year and you deserve answers. 😄

Since the only way to get the location of the call site is to throw an error and catch it, it would add a linear performance penalty. Would this be acceptable?

Investigation required. Unless trying it out shows the difference to be approximately 0 (i.e. that runtimes optimize it incredibly well), I'd think it definitely should be behind an option. My hunch is that we'd want the option to be opt-in at least to start.

Users that run Mocha tests often transpile to JS before running the tests (for example, using esbuild or tsc). Is resolving source files part of this feature?

Agreed that I think not. I don't think we want to build any source map resolution, file transpiling, etc. into Mocha. The API surface right now is mostly around importers and the like - and should stay that way.

Would you be willing to add call-id as a dependency of mocha?

If that ends up being how the feature would be implemented in Mocha, no problem from me. But I think we'd want to determine what the best implementation for Mocha is. Once there's a design we can figure out if a pre-existing solution like that fits in.

FWIW I think call-id is a great reference of how to do this thing, and I wouldn't be at all surprised if we end up using it. I just don't want to promise "yes let's use that!" and then, after some rounds of PR review, realize there's some other way that would let us bypass some of the work in it.

You've marked this issue with area: nodejs. This feature could also work in the browser.

Good point! Switched. Thanks.

Which brings me to: what browsers does Mocha need to support?

Good question. I'd filed #5226 on this. The current proposal I have is #5226 (comment):

... sync the timeline of browser support with Node.js support. As in, Mocha should support the versions of mainstream browser versions that were most current when the supported Node.js versions were released. Including data on the Node.js 14 (released April 21st, 2020) support equivalents from #5225 (comment):

• Chrome/Chromium 81 released April 7, 2020: https://developer.chrome.com/blog/new-in-chrome-81
• Firefox 75 released April 7, 2020: https://www.mozilla.org/en-US/firefox/75.0/releasenotes
• Safari 13.1 released March 24, 2020: https://developer.apple.com/documentation/safari-release-notes/safari-13_1-release_notes

But that issue is still in discussion.

---

Anyway, cc @mochajs/maintenance-crew - I think this is a reasonable and good feature request. But I'm a little biased because I'm a fan of the Stryker project.