cut FS activity down dramatically #33
Conversation
|
When operating on large dir trees, this cuts rebuild time down dramatically. In some of my tests (against ember.js) it cuts them down by 40% or 50%. We can likely restore the lack of needing mkdir in the cache, but that now barely costs us anything. The big win is priming the output dirs together. Previously when so if we had Commonly, we are actually creating.
etc. before that would be atleast 8 internal calls to mkdir + lstat per entry. ^^ would be at least 24 mkdir. Now we attempt to create the dirs together, which allows us to skip over repeated work. The above example reduces to 11 mkdir. This bulk mkdir really scales nicely as project trees grow large, as large chunks of the dir structure is commonly shared. |
* better logging * use mkdir-batch to group mkdirs together and do the less overall work (don’t attempt to re-create the same dirs repeatedly) * rather then writing to output and then doing a copyDereferenceSync to the cache, we just write to cache and link to output
|
Very excited about this! |
| return self.outputPath + '/' + p; | ||
| })); | ||
|
|
||
| if (this._cacheDirsPrimed === undefined) { |
There was a problem hiding this comment.
I'm a bit confused why we're checking this at all, but we're setting this to false (not undefined) below, so this would only seem to run once.
There was a problem hiding this comment.
so this would only seem to run once.
Ya the goal is for it to only run exactly once.
The variable name was written in haste, and pattern was written in haste.
|
You said on slack that some of the code is still work in progress, but 👍 on what this is doing in general. |
|
Ya. I'll likely circle back soon and clean this up. It was more of an exploration of what is possible, and the results where good. Also indicates a better patch variant would be dramatically faster (but also more work). Chances are I'll massage this one further. Release it then later come back to the patch approach we discussed |
fixes #32