Using pnpm as a Node.js version manager

[zkochan]

Currently pnpm need Node.js to be installed in order to be used. However, if we will bundle pnpm to an executable using pkg, we will be able to use pnpm without Node.js. If pnpm will run without Node.js preinstalled, then we can use pnpm for installing Node.js (Node.js is currently available as a npm package). Furthermore, we could use pnpm as a version manager for Node.js (like an alternative to nvs, nvm, volta).

---

Related pull requests:

• feat: new plugin for managing Node.js versions #3459
• fix: download Node.js from nodejs.org #3475
• fix: pnpm exec should use the right Node.js version #3477
• feat: a command for installing Node.js #3620
• feat(env): install Node.js using a version range #3629
• feat: pin Node.js to global package #3780

Related issues:

• pnpm should use the active Node.js version to check package compatibility #3785
• Locking CLI tools to specific Node.js versions #3595

[ScottAwesome]

I'd be into this.

Sure would simplify some things at my job.

The key with pkg is having a good entry point, and another caveat worth mentioning is basically for anything that isn't a guaranteed static path, you must use process.cwd() in order to resolve correctly.

Other than that, it ain't so bad.

[zkochan]

I was experimenting with it yesterday. So far seems good but I have found one issue. pkg overrides the child_process library and any scripts that contain "node" are overwritten. As a result, any postinstall scripts that run Node.js get broken.

I don't really understand why they patch child_process. Maybe we could fork it and remove the patch part.

As a dirty workaround, seems like adding an empty space before "node" avoids the override and installation works fine as a result.

[amitgilad3]

sounds good ,
i only have one concern , when running pnpm commands for packages that require a higher or lower version of node than the one pnpm is bundled with (but i could be mistaken).

for example if pnpm would be bundled with node@14 and we run pnpm add and that package would require node16 . The context of which pnpm would be executing the add is node@14 add we would get an error for the installation that the node version is not good.

[zkochan]

Indeed, but I think we can figure out some workaround for this.

[zkochan]

Just to follow up. I have made a fork of vercel/pkg to bundle pnpm. The Node.js that runs pnpm is only used by pnpm. The scripts of the installed packages use the globally installed Node.js or the one specified using the use-node-version setting.

[zkochan]

Just tried it. It seems to work.

I bundled pnpm with a fork of pkg. Here's a WIP PR: #3444

I uploaded the linux artifact to get.pnpm.io: https://github.com/pnpm/get/tree/6be4bd15212a87cb527dd080510b721f3f556c2a/v6.3.0

Then I started a clean OS with no Node.js installed: docker run -it centos sh

Then I used these steps to install Node.js using pnpm:

curl -O https://get.pnpm.io/v6.3.0/pnpm.tar.gz
tar -xf pnpm.tar.gz
cd pnpm
chmod +x pnpm
./pnpm add -g node-linux-x64

Node.js was successfully installed.

---

EDIT: the same with and install.sh that download the prebuilt tarball from the npm registry

curl https://get.pnpm.io/beta-install.sh | bash
export PATH=~/.pnpm:$PATH
pnpm add -g node-linux-x64

---

EDIT 2:

Using dnf package manager:

curl --silent --location https://get.pnpm.io/rpm/pnpm.repo | tee /etc/yum.repos.d/pnpm.repo
dnf install pnpm -y
pnpm env use -g lts

[zkochan]

Looks like Deno has the possibility to compile a Deno app to an independent executable: https://deno.land/manual/tools/compiler#cross-compilation

It would probably work bettern than the workarounds with Node.js and pkg. But rewriting everything to be Deno-compatible is probably a huge task.

[ScottAwesome]

if I correctly understand the biggest pain point would be switching all your dependency references to URLs, I think, beyond learning the quirks of deno itself of course. if I recall correctly the deno std library has good interop with NodeJS built-in modules in places where they don't have an obvious replacement.

[zkochan]

I don't think we can realistically do it. We are some really low level optimizations tied to Node.js.

[melvinotieno]

Just to address how your concern is valid. I recently was experimenting with pnpm and I have a project that runs on node version 10. Usually, I use nvm to switch between node versions except this time, I used pnpm use --global 10.16.0. Trying to switch back to the lts version was hell. Everytime I try doing so with pnpm, I get the following

ERROR: This version of pnpm requires at least Node.js v12.17
The current version of Node.js is v10.16.0

Also, I think it's best to support setting node version on a per project basis to avoid the hell I just had to go through as it affected not only the current pnpm but also npm and yarn. I could not even switch node version using nvm.

[woss]

Have you seen how the N works? out of all of the NVMs out there, this one is the cleanest and fastest. Binding yourself to the npm IMHO is not a smart choice, due to the high centralization of the platform.

[woss]

i would like to help to make this reality since we really will be working on this in near future. if you wish I can send you the draft of the issues we are facing related to the centralized and semver in the near future. Let me know where to send the link

[zkochan]

Could you create a discussion or issue? If it will be implemented in pnpm, it will become public anyway.

[woss]

Sure, the project that i'm building this for is public anyway, so no secrets here. But it will be some time ( weeks ) until I have anything ready and consolidated into one document.

[zkochan]

If you just want to chat, you can post to the "contributing" channel in the pnpm Discord

[hinell]

n is a joke compared to nvm

[vjpr]

nvm has a feature to re-install global packages when installing a new version of Node. Will pnpm support this?

Also, any symlinked packages (pnpm link -g) should be re-linked.

I haven't had much success getting this feature to work, and often I will not upgrade Node's as regularly as I should because of this small friction.

$ nvm --help

  nvm install [-s] <version>                Download and install a <version>, [-s] from source. Uses .nvmrc if available
    --reinstall-packages-from=<version>     When installing, reinstall packages installed in <node|iojs|node version number>

[zkochan]

Do you frequently use global packages? I personally never use global packages because any needed tooling is in dev dependencies and is callable via pnpx.

If we need support of global packages, I think the right approach would be to bundle every global package with its node.js. So when you install pnpm add -g eslint, eslint is installed and attached to the currently active Node.js. Then even if you switch Node.js and run eslint, eslint will still use the Node.js version that it was installed with.

[vjpr]

Do you frequently use global packages?

Sometimes. There are a few use cases.

There are a few cli utilities that I globally link to do some stuff un-related to any particular project.

Here is an example of a util installed via npm: https://www.npmjs.com/package/zx

I think pnpmx here would add too much overhead.

There are a few tools that insist on using globally installed stuff, and then they check your current project dir to see find a local installed version. I think expo-cli does this.

Then even if you switch Node.js and run eslint, eslint will still use the Node.js version that it was installed with.

This would be cool. It would mean I could upgrade my node version, and my utils would still work. However, for tools that spawn new node processes, things could become unexpected...

[zkochan]

However, for tools that spawn new node processes, things could become unexpected...

I think we can workaround that issue. We can create a command shim that explicitly runs the package with the given version of Node.js and does not modify the PATH, so any child processes will use the right Node.js version.

[zkochan]

I have created an issue for this: #3595

[Djaler]

btw, do you have any plans about pnpm version management? I mean, something that will switch pnpm version for different projects?

[Djaler]

I made a simple utility to solve this problem. https://github.com/Djaler/pnpm-wrapper
It's currently in 0.y.z stage, so API may be changed in future

[zkochan]

@Leif-W being fast is a feature of pnpm. Adding +180ms for this feature is not an option. If we will run one pnpm process and then it will switch to another process, the overhead will be about 180ms. Maybe we can come up with some workarounds, I am not sure.

• We may write a small shell wrapper that would pick the right pnpm version
• Or maybe we could always link the active pnpm process to the global scope. So the first run would be slower but the next run on the same project would have no overhead.

@Djaler thanks for working on it.

[Djaler]

@zkochan what do you think about my solution?

[zkochan]

I have doubts that it would be a good idea to use a separate tool for wrapping pnpm because we're going to advertise pnpm as s Node.js package manager and it will be shipped as an executable.

I have already updated the installation script for POSIX systems: https://pnpm.io/installation

IMHO, we should just ask users to always use the latest version of pnpm. I don't agree with @Leif-W that the lack of pnpm version management will slow down innovation. On the contrary, I think it will allow projects to pin an old version of pnpm forever.

In any case, if we do think that a pnpm version manager is necessary, it should be probably built in to pnpm. But I think we should solve a lot of other issue before looking into it.

[zkochan]

Node.js just merge corepack: nodejs/node#39608

so I guess there will be a standard version to manage pnpm versions.

[zkochan]

Adding a new command for installing Node.js globally:

pnpm env use --global <version>

PR: #3620

[sunnz]

So what's it supposed to do without --global flag? Can you for example install v16 globally and switch to different nodejs versions such as v14 and v12 on a per project basis, even if just manually using the env use command without the --global flag each time?

[zkochan]

I suppose it should set this setting: https://pnpm.io/npmrc#use-node-version

[zkochan]

There is something I don't know how to handle. Currently there are some commands that pass through to the npm CLI (pnpm info, pnpm config, pnpm dist-tag, etc). Right now it is not a huge problem as pnpm is mostly installed on systems that already have Node.js and npm preinstalled. However, we can now install pnpm on a system that has neither Node.js nor npm installed. And in this scenario, some commands will fail.

• Should we bundle npm CLI with pnpm?
• Should we link the npm binaries as well when installing a Node.js environment using pnpm env use?
• Should we recommend installing npm separately using pnpm install -g npm
• Should npm and Node.js be installed automatically when running pnpm setup?

cc @pnpm/collaborators

Related issue: #3674

[KSXGitHub]

I don't mean bundle all of npm only to use some parts of it. npm itself is also made of multiple modular parts:

Alternatively, we can copy part of npm's code into pnpm. The npm registry has to maintain compatibility with all npm versions so we don't have to worry about our code suddenly becomes incompatible.

[zkochan]

I would rather not use their libraries directly. I had really bad experience with their internal libraries in the past. They make huge breaking changes frequently and it is hard to keep up. I had to fork a lot of their stuff to make it stable.

[ScottAwesome]

There is something I don't know how to handle. Currently there are some commands that pass through to the npm CLI (pnpm info, pnpm config, pnpm dist-tag, etc). Right now it is not a huge problem as pnpm is mostly installed on systems that already have Node.js and npm preinstalled. However, we can now install pnpm on a system that has neither Node.js nor npm installed. And in this scenario, some commands will fail.

• Should we bundle npm CLI with pnpm?
• Should we link the npm binaries as well when installing a Node.js environment using pnpm env use?
• Should we recommend installing npm separately using pnpm install -g npm
• Should npm and Node.js be installed automatically when running pnpm setup?

cc @pnpm/collaborators

Related issue: #3674

As an observer (and simple contributor!) I'd like to see npm be updated to the latest version when installing node.

I say this as someone who uses both yarn and pnpm in the workplace on our teams (slowly moving to a monorepo managed by pnpm in case you're wondering why both): When npm is not the main package manager being used by the team, people tend to forget to update it, and that can sometimes cause issues in some bootstrapping processes for us (mainly around getting global packages updated and using npx for things because neither yarn or pnpm are available yet)

[zkochan]

For now I am making pnpm env link npm/npx as well: #3696

We can change it to a different solution in pnpm v7

[zkochan]

Now the problem is that when node and npm are installed by pnpm env, npm i -g foo installs foo to a bin that is not in the PATH.

I guess there are two ways to solve it.

1. Change the global prefix of npm to the pnpm home directory
2. Instead of symlinking the node executable into the pnpm home directory, symlink the node bin directory and add it to PATH as well

I guess the 1st solution is the easier one. The pro of the 1st solution is that the globally installed packages will be available even after switching to another node. The con is that if the globally installed package has some native code, it might stop working when the Node.js is changed.

[jcayzac]

@zkochan Using use-node-version, the CLI wrappers installed under node_modules/.bin don't seem to use the specified version, just whatever is in the PATH. Is that a bug or is it working as intended?

[zkochan]

Probably a bug

[jcayzac]

I see now that the wrappers also test for the presence of a node executable in the node_modules/.bin folder, and use it if present. The problem is that using use-node-version doesn't seem to install a node executable there at all (pnpm 7.2.1, macOS 11.6.2).

[hinell]

Well...you are going to reinvent the wheel.

[javier-garcia-meteologica]

It's worth taking a look to the rest of the js ecosystem. At our workplace we use pnpm + turborepo. Starting with version 1.1, turborepo requires packageManager to be defined in package.json. The details of this field can be found at corepack, which is an official but experimental nodejs tool.

This is confusing, on the one hand pnpm wants manage nodejs versions and the other hand nodejs wants manage pnpm versions.

As a js developer I want to specify which are the dependencies required by my package after installation and for this I use dependencies. This is insufficient because my package also depends on a nodejs version-range (and possibly on a specific package manager too) but I cannot change that during installation. Luckily the engines entry in package.json allows me clearly state and enforce these restrictions. So regarding to what happens to my package after installation, everything is ok.

Managing requirements during development is more complex. Even if my package already has declared some dependencies and has specified a wide range of compatible nodejs versions and package managers in engines, it may use additional dependencies for development and may require a specific nodejs version and package manager to be used during development. The problem of development only dependencies is already solved locally thanks to devDependencies. But requiring a specific global nodejs version and package manager can create conflicts. As a js developer I use the same computer to work with different repos, some of which require different and possibly incompatible versions of nodejs and package manager. On top of that, the package manager itself depends on the nodejs version.

Ideally each development repository should have its own environment with its own version of nodejs and package manager. The question is, how should it be done? Notice that the development environment has to be bootstrapped from the shell.

Let's explore the alternatives.

• A third tool prepares the environment (nvm, asdf, ...)

  • Requires learning yet a different tool
  • Requires invocation before every work session (to setup $PATH)
  • Does not automatically infer which nodejs version to load for each repo, the user has to specify it manually every time (as far as I know)
  • Does not automatically infer which package manager to load for each repo and the choice is determined by the nodejs version.

• Nodejs prepares the environment

  • Doesn't require learning yet a different tool
  • Doesn't require a special invocation before every work session, it automatically catches calls to pnpm, yarn and npm forwarding them to the correct version of each package manager.
  • Infers which package manager to load for each repo
  • Does not automatically infer which nodejs version to load for each repo, the user is stuck with the global version

• The package manager, pnpm, prepares the environment

  • Doesn't require learning yet a different tool
  • It would be great if it didn't require a special invocation before every work session. If we assume that all calls are made using pnpm exec changeset add or pnpm run dev then this is possible. When pnpm is called, it can check if its version number is compatible with the package requirements, otherwise it should download the appropriate version and forward all the arguments.
  • It would load the correct package manager for each repo if the above is true.
  • It could also load the appropriate nodejs version for each package, as long as it's always called through pnpm first.
  • IDEs and other tools would have to accommodate these changes and remember to route their nodejs calls through pnpm (or yarn, or npm if they choose to implement similar solutions)

[boda-sh]

We prefer Volta for managing node version in different repositories, because if you can add the following property to package.json

{
  "volta": {
    "node": "16.15.0" 
  }
}

Volta will automatically infer the node version from now on

[jcayzac]

@bodazhao I currently also use Volta but I really dislike this solution for a number of reasons. Most importantly, Volta seems to ignore that packages whose native parts were built against node X.Y aren't supposed to work with node Z.W (so all global packages would normally need to be rebuilt every time you switch node versions).

I would much rather have always the same system node version active in my shell if I need one (but maybe I don't), and the version of node required by my current project's scripts pulled as a dev dependency by the package manager —ideally a statically built executable I can just download.

[soc221b]

Update:

We can use pnpm as Node version manager, because yarn will use the global node. But you need to install pnpm via brew install pnpm instead of using standalone scripts.

---

Original:

Could we use both nvm and pnpm together?

We use nvm to switch Node versions across our projects. (we still need to use yarn in some projects, so we can't just using pnpm as a Node version manager.)

I use Node.js is preinstalled option to install pnpm, and it is installed in: /Users/user-name/.nvm/versions/node/v14.19.1/bin/pnpm.

When I switch to Node@16 via nvm use 16 and run pnpm install in one project which need Node@16 to be installed (engines.node), I got this error: Expected version "16.14.0". Got "14.19.1".

(I also tried testing it with yarn and it works as expected.)

[zkochan]

It is a tarball that has the node executable and other stuff.

[jamischarles]

But in theory it could use a local version right? No major blockers? Trying to think through some of the use cases here but maybe this isn't a real need.

[soc221b]

Users may want to use some node version manager tools too 😅 :

1. nvm
2. n

[kirinnee]

Hi, would like to weigh in here. I use nix to manage my node versions, I would really like it for pnpm can detect and use the local version from PATH, or some configuration to enable loading custom nodejs version instead of the default ones.

[Aghassi]

+1 to using what's already on the user PATH and then falling back on whatever pnpm fetches if none exists. We use bazel to manage our node versions and having pnpm essentially squirrel away it's own version is not ideal. It also causes confusion upon install when my system node version is ahead of whatever pnpm decided was the right version to use. I'd expect the priority to be

1. PNPM_NODE_PATH is something I can set somewhere that overrides all else and pulls from somewhere on disk I specify
2. Check the user's PATH for an existing node install. If it exists, use that. This allows for nvm, n, volta, corepack etc to manage it over your tool
3. fallback on this logic to install the version specified

[so1ve]

Deno now enhances its npm support (via npm: identifier), which allows us to import packages directly from npm. If you use a import_map to control the pnpm version and update it automatically, then we can use deno compile instead of pkg