Limit the number of documents processed at a single time.

[ebolyen]

Proposal

Currently it appears that all documents are processed at the same time in parallel.
When using an executable plugin, this results in a process for each document. In the event that this executable plugin does something computationally expensive (like execute a code example), this quickly thrashes your machine.

I'm not sure if it makes sense to have a process pool for executable plugins, so that only X number run at a time (leaving promises on the event loop to wait around for a worker). Or, if the entire mdAST transform (getFileContent) should be batched, so that only X documents are processed at the same time.

We would probably want something like a single transform in our executable-plugin to run per CPU available.

[agoose77]

This is a good enhancement suggestion. We need to think carefully about the best solution, because we also want to support remote execution on different compute environments! In those cases, the concept of CPU availability may be less well defined.

Nevertheless, our local execution could definitely do this by default.