Use compiler.inputFileSystem to make thread-loader work with memfs

[dmnsgn]

I couldn't make thread-loader open an entry file from memory system as it yielding the following error: ENOENT: no such file or directory, open. I assume it has to do with the fact worker.js uses fs https://github.com/webpack-contrib/thread-loader/blob/master/src/worker.js#L3.

Would using _compiler.inputFileSystem or allowing an option to specify a in-memory file system in the loader allow that?

Repro below (create the package.json/index.js and run npm test):

package.json

{
  "name": "webpack-thread-loader-memfs",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "node index.js"
  },
  "dependencies": {
    "memfs": "^3.2.0",
    "thread-loader": "^2.1.3",
    "unionfs": "^4.4.0",
    "webpack": "^4.43.0",
    "webpack-cli": "^3.3.12"
  }
}

index.js

const fs = require("fs");
const path = require("path");
const webpack = require("webpack");
const { Union } = require("unionfs");
const { createFsFromVolume, Volume } = require("memfs");

const entry = path.join(__dirname, "entry.js");

const memoryFs = createFsFromVolume(new Volume());
memoryFs.join = path.join.bind(path);
const ufs = new Union();
ufs.use(fs).use(memoryFs);

memoryFs.mkdirpSync(__dirname);
memoryFs.writeFileSync(entry, `console.log('entry')`, "utf8");

const config = {
  entry,
  output: {
    filename: "bundle.js",
    path: __dirname,
  },
  mode: "production",
  module: {
    rules: [
      {
        test: /\.(js|jsx)$/,
        exclude: /node_modules/,
        use: [{ loader: require.resolve("thread-loader") }],
      },
    ],
  },
};

const compiler = webpack(config);
compiler.inputFileSystem = ufs;

compiler.run((error, stats) => {
  console.log(
    stats.toString({
      chunks: false,
      entrypoints: false,
      hash: false,
      version: false,
      modules: false,
      colors: true,
    })
  );
});

Output

ERROR in ./entry.js
Module build failed (from ./node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 0)
ENOENT: no such file or directory, open '/Users/path-to-project/webpack-thread-loader-memfs/entry.js'
    at PoolWorker.fromErrorObj (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:262:12)
    at /Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:204:29
    at mapSeries (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/neo-async/async.js:3625:14)
    at PoolWorker.onWorkerMessage (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:170:35)
    at /Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:152:14
    at Socket.onChunk (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/readBuffer.js:36:9)
    at Socket.emit (events.js:315:20)
    at Socket.Readable.read (_stream_readable.js:513:10)
    at Socket.read (net.js:623:39)
    at flow (_stream_readable.js:986:34)
    at emitReadable_ (_stream_readable.js:577:3)
    at processTicksAndRejections (internal/process/task_queues.js:83:21)

[jsg2021]

I am very interested in this as well... it would probably have to stub fs and proxy them to the parent process.

[jsg2021]

@evilebottnawi @Mesoptier thoughts on this?

[jsg2021]

I'm wondering if the fs API surface could be proxied to the parent process or at least the methods explicitly defined in the input file system wrapper.

[alexander-akait]

@jsg2021 we can create wrapper for fs calls, like we do it for getResolve

[jslq]

not resovled yet?

[alexander-akait]

Need added a test case with webpack-virtual-modules