fix: melos commands fails with “OS Error: Too many open files” when working with large projects

[mahmuttaskiran]

Is there an existing issue for this?

• I have searched the existing issues.

Version

6.1.0

Description

Running melos commands fails with a FileSystemException due to “Too many open files” when the project contains a large number of local packages. In our Flutter project, we have over 290 packages, and the error occurs when the number of open files exceeds the system’s ulimit -n value.

The root cause seems to be Melos trying to read all the pubspec.yaml files in parallel, which results in too many simultaneous file operations.

Err details:

FileSystemException: Cannot open file, path = '/Users/mahmut.taskiran/StudioProjects/x-mobile-app-flutter/packages/features/rider_tip/pubspec.yaml' (OS Error: Too many open files, errno = 24)
#0      _checkForErrorResponse (dart:io/common.dart:55:9)
#1      _File.open.<anonymous closure> (dart:io/file_impl.dart:381:7)
<asynchronous suspension>
#2      _File.readAsBytes.<anonymous closure> (dart:io/file_impl.dart:562:24)
<asynchronous suspension>
#3      _File.readAsString (dart:io/file_impl.dart:621:18)
<asynchronous suspension>
#4      PubSpec.loadFile (package:pubspec/src/pubspec.dart:128:33)
<asynchronous suspension>
#5      PackageMap.resolvePackages.<anonymous closure> (package:melos/src/package.dart:537:25)
<asynchronous suspension>
#6      Future.wait.<anonymous closure> (dart:async/future.dart:534:21)
<asynchronous suspension>
#7      PackageMap.resolvePackages (package:melos/src/package.dart:534:5)
<asynchronous suspension>
#8      MelosWorkspace.fromConfig (package:melos/src/workspace.dart:48:25)
<asynchronous suspension>
#9      _Melos.createWorkspace (package:melos/src/commands/runner.dart:114:13)
<asynchronous suspension>
#10     _BootstrapMixin.bootstrap (package:melos/src/commands/bootstrap.dart:12:9)
<asynchronous suspension>
#11     CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#12     MelosCommandRunner.runCommand (package:melos/src/command_runner.dart:77:5)
<asynchronous suspension>
#13     melosEntryPoint (package:melos/src/command_runner.dart:130:5)
<asynchronous suspension>
#14     main (file:///Users/mahmut.taskiran/StudioProjects/melos/bin/melos_dev.dart:13:3)
<asynchronous suspension>

Steps to reproduce

1. Create a flutter project has over 290 packages (or set ulimit to any number lower than the total number of packages you have in your project)
2. Run any melos command

Expected behavior

Melos shouldn't do file operations in parallel at once for reading pubspec.yaml files for packages. There should be a Pool limited to a number.

Screenshots

No response

Additional context and comments

No response

[spydon]

@mahmuttaskiran the proposed solution sounds sensible, are you interested in working on a fix for this?

[mahmuttaskiran]

For sure!