[meta issue] persistent outputDir for better perf

[stefanpenner]

As part of the broccoli-plugin refactor, we have unlocked the ability for a further optimization.

We can now keep outputDir persistent between rebuilds. (not to be confused with broccoli-persistent-filter – which keeps the cache persistent between restarts). This enables plugins who opt-in the ability to do a more intelligent diff based update of their own outputDir, and should cut down on FS activity nicely.

Some early experiments (just batching mkdirp) have lead to some very favorable numbers (up to 50% rebuild speed up in broccoli-filter). The good thing is, with this approach the vast majority of output IO can likely be skipped.

The filters that will likely benefit the most are:

• https://github.com/broccolijs/broccoli-merge-trees
• https://github.com/broccolijs/broccoli-funnel
• https://github.com/broccolijs/broccoli-filter
• https://github.com/broccolijs/broccoli-persistent-filter

[stefanpenner]

This week, I plan to start with the following:

• logging additions
  • input processing time
  • computation (like Filter.prototype.processString) time
  • output Processing time
• FS instrumentation
  • counts
  • times
  • enabled only in some debug mode

These steps will both help identify current issues, but allow us to prevent regressions going forward. With this, I hope we will have a more confident understanding of what the current cost are.

After-which I hope to begin working on funnel, but will welcome help if someone has spare cycles.

[joliss]

• Big 👍 on exploiting persistent output dirs in all those plugins.
• Instrumenting fs seems like a pretty big hack, but also practically very useful. Maybe we can isolate this into a package? Also, for commonly used packages like broccoli-filter we could simply provide debug logging on file system operations (like at this line) without needing to resort to hooking into fs.
• Logging more stuff in general is great. I'd also be interested in hearing your views at some point (doesn't need to be now) on what kind of first-class API we might turn the node-debug-based logging into - like what kind of tagging we need, or if we want structured timing reporting.

[stefanpenner]

Instrumenting fs seems like a pretty big hack, but also practically very usefu

Yes, It is extremely hacky. I figure if we make it opt-in, it will be less scary.

Maybe we can isolate this into a package?

I would like to make it a separate package, to work best though. Some small hooks into builder would be needed. Something similar to how times are currently calculated -> https://github.com/broccolijs/broccoli/blob/master/lib/builder.js#L129-L132

In theory something like:

builder.on('startRunning', function(node) {

});

builder.on('endRunning', function(node) {

});

would allow us to totally externalize various monitoring solutions. This could even include the existing timings

Also, for commonly used packages like broccoli-filter we could simply provide debug logging on file system operations (like at this line) without needing to resort to hooking into fs.

The issues are as follows:

• manually instrumenting all plugins is tedious and brittle
• many problematic plugins are actually one-off user created ones.
• some operations like (mkdirp), have somewhat opaque internals. It may not be obvious how sub-operations they trigger.

If we can explore instrumentation hooks, we can externalize this complexity.

[joliss]

Yeah, builder hooks sound good to me.

[tsing80]

@stefanpenner 👍 👍

[stefanpenner]

This all landed ages ago :)