Crashes with "EMFILE: too many open files" on large projects

[alltom]

My project has 35,085 files in 3,652 directories. When I load the web interface, it fails with:

Error: EMFILE: too many open files, open '/usr/local/lib/node_modules/codecrumbs/node_modules/watchpack/lib/DirectoryWatcher.js'
    at Object.openSync (fs.js:448:3)
    at Object.readFileSync (fs.js:348:35)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:809:20)
    at Module.load (internal/modules/cjs/loader.js:666:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:606:12)
    at Function.Module._load (internal/modules/cjs/loader.js:598:3)
    at Module.require (internal/modules/cjs/loader.js:705:19)
    at require (internal/modules/cjs/helpers.js:14:16)
    at WatcherManager.getDirectoryWatcher (/usr/local/lib/node_modules/codecrumbs/node_modules/watchpack/lib/watcherManager.js:14:25)
    at WatcherManager.watchDirectory (/usr/local/lib/node_modules/codecrumbs/node_modules/watchpack/lib/watcherManager.js:32:14)

macOS 10.14.3
Node 11.11.0
codecrumbs 0732382

[Bogdan-Lyashenko]

Hey, well, 35,085 files probably too much for Nodejs too keep in memory, I tested on projects with ~5k files. That means I need to add chunks while doing initial scan of project source, it shouldn’t be too hard.

[Bogdan-Lyashenko]

@alltom hi, how is it going. I've been thinking to add global patch for fs:

// Make sure to read the caveat below.
var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)

by https://github.com/isaacs/node-graceful-fs. And probably activate it by CLI param, disable by default since it might causes performance issues.

And more I think about that, more doubts I have 🤔

Can you please confirm that 35k files is a reasonable number of files? I.e. it's not node_modules (I filter node_modules, but can be a bug).

Or, may you consider to change increase limits on OS level?
http://blog.mact.me/2014/10/22/yosemite-upgrade-changes-open-file-limit

Let's discuss. Thanks.

[Bogdan-Lyashenko]

*yeah, after my first reply I noticed that the bug is inside watchpack (module to watch file changes, webpack use it as well) so it's not that trivial

[alltom]

It's actually a Go project, for the most part. (I cloned the Java support files to add Go support. :)

The biggest chunk of files, though, is a sub-directory containing a third party library I included in my project that has code in a variety of languages. I could easily exclude that if there were a command-line flag for it. The only directory in there that looks like codecrumbs could ignore as a rule is a Python site-packages directory.

[Bogdan-Lyashenko]

I see, thanks. Well, then I will add CLI params to “exclude” directories, this will be the first step. Also, I probably need to add filtering for default folders with dependencies per language, currently there is only node_modules

[Bogdan-Lyashenko]

@alltom hi there, sorry this took so long, was in the middle of another feature ;)

I pushed quick fix to exclude directories by CLI param, should solve your problem for now.
Update codecrumbs to the v1.2.1 and add -x src/dir-to-exciude,src/one/two/dir-to-exclude(list of dirs separated by comma).

Hope that helps. Ping me if anything.
Thanks.

[alltom]

Awesome, thank you! I'll give it a shot.

…

[BrunoDG]

I'm having the same problem, but in Windows 10 with a C++ project.

It has 9.107 files with 721 folders and keeps giving me a crashing message similar to @alltom 's:

Error: EMFILE: too many open files, open 'C:\Users\xxxxxxx\AppData\Local\Yarn\Data\global\node_modules\to-regex\node_modules\extend-shallow\index.js'
at Object.openSync (fs.js:439:3)
at Object.readFileSync (fs.js:344:35)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:20)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object. (C:\Users\xxxxxxx\AppData\Local\Yarn\Data\global\node_modules\to-regex\index.js:5:14)
at Module._compile (internal/modules/cjs/loader.js:689:30)

[Bogdan-Lyashenko]

Do you need 9k files? Try to exclude folders you don’t need by -x param (see Readme, CLI info)

[BrunoDG]

Unfortunately, yes. Because there are several libraries linked to the project which are being used on code.

[Bogdan-Lyashenko]

Alright, then I need to fix that)

[BrunoDG]

Bogdan, nevermind, I looked after the code and it seems that it's working now. I'm sorry for the issue tracked, and thanks for your attention! :)

[Bogdan-Lyashenko]

Sorry, didn’t really have time to look at this, was working on other features :) just so you know, there is -x CLI parameter (see readme for details) you can use to exclude folders you don’t need (so there are less files)