PNPM V9 peerDependencies and git merge conflicts

[andschub-ms]

This is more of a design problem than a bug, so opening a discussion here.
We have been having frequent git merge conflicts in our pnpm-lock.yaml due to peerDependencies adding extra versions in parentheses, like
2.5.4([email protected])([email protected])([email protected])([email protected])
which sometimes would stretch into dozens depending on how many peerDeps we had. If anything on the same line is modified, from two different authors, git will treat that as a merge conflict and it will need manual resolution, which interrupts the developer workflow on a team with dozens of devs.

In PNPM v8, we saw fluctuating versions, so sometimes they would be added/dropped from the parenthesis list seemingly at random. This motivated us to try upgrading to v9. The early versions of v9 had a memory leak and would crash during install. I tested v9.2.0 today after seeing some patches around memory leaks. Now, it appears the fluctuating problem is gone, and thank you also for fixing the absolute paths problem in the lockfile.

However, in PNPM v9, I see the lockfile format is changed and now recursively lists every peerDependency in the entire dependency graph of each component as a version. This is troubling, because now it is amplifying the git merge conflict problem by orders of magnitude. If any two authors change the same line, it will cause a merge conflict. We have some lines which are stretching into tens-of-thousands of characters long, making the surface area large for conflicts.

We have a unique situation where we are using more peerDependencies than normal, since the original architect (not me) decided to use peerDependencies as a mechanism to avoid bundling duplicate packages in our application. One possibility is to migrate our entire codebase (thousands of packages) to "dependencies" instead, using proper workspace-linking setups, but this could take over a year, and we are trying to onboard PNPM in a shorter time frame.

I first tried using a .pnpmfile.cjs to just delete the peerDependencies from every one of our package.json manifests, and this cleaned up our lockfile tremendously. However, it causes us to use NODE_PATH to find the missing modules which were not sandboxed/linked in the .pnpm store, which worked fine for webpack, but seems like TypeScript (ts-loader) has no way of supporting that option, and I've reached a dead-end on this solution.

From a design perspective, it might be better if PNPM, for example, listed these peerDependency versions one-per-line in the lockfile. Then git should be able to handle merge conflicts effortlessly (unless two authors are deliberately changing the same package to different versions, which is expected). For example, in the lockfile, instead of a giant concatenated string, store the peerDeps versions as a nested YAML arrays under the "version" node, then do the concatenation as needed in the PNPM library. Or another possible solution, if that is unsatisfactory, is to use some sort of line breaking to just break the string up and strip out the whitespace later such as mentioned in https://stackoverflow.com/questions/3790454/how-do-i-break-a-string-in-yaml-over-multiple-lines

Another solution would involve a design change in PNPM's treatment of peerDeps. Is it possible to use identifiers to instances instead of fully-qualified peerDeps? e.g. define react-hot-loader#1 = peer dep react v15, react-hot-loader#2 = peer dep react v17, and then refer to these instances elsewhere in the lockfile. By using identifiers we can avoid listing the entire peer dependency graph redundantly in the lockfile, assuming we can find a way to name the instances in a way that is deterministic and does not fluctuate too much either between commits. The feasibility of this approach relies on the stable choice of identifiers.

I'm putting this out there in case there are some solutions that I'm not aware of that may help us to get unblocked. I wish git was smarter and worked on merging within a single line, unless there's some obvious config option I don't know about.

[zkochan]

I don't think we can implement identifiers for this. And even if we could, likely the same merge conflicts will happen with the identifiers instead. Also, making them deterministic would be really hard. We are going to replace very long peer dependency suffixes with hashes (related PR: #8177), which you could consider "deterministic identifiers". But this won't solve the git conflicts issue.

Right now I am not sure there is a way to come up with a format that will solve all issues and reduce the number of conflicts. We could remove some context from the lockfiles but that would make installation with up-to-date lockfiles slower.

You can try Git branch lockfiles to reduce the number of conflicts you have to deal with.

[andschub-ms]

Thank you for taking the time to look into this. We are evaluating the git branch lockfiles and other options and will get back to you as soon as we found a working solution.

[andschub-ms]

We decided the best approach was just to get rid of using peerDependencies. We used the .pnpmfile.cjs to transform our peerDependencies to dependencies. The merge conflicts are much less now.