Best practice for working around no hard links between filesystems in dev containers

[rhyek]

I'm using the remote containers extension for visual studio code with a monorepo project.

I'm trying to speed up rebuilds of the dev container using a combination of the following techniques:

• Buildkit build cache mounts: Cache the pnpm store folder during builds
• PNPM fetch command Fetch packages while the pnpm store cache mount is active
• While mounting the workspace folder (project source files) into the container as a volume, keep the root node_modules directory as a separate volume within the container itself to prevent host<->container IO for that folder.

Relevant dockerfile section:

RUN npm install -g pnpm
RUN pnpm config set store-dir /root/.pnpm-store
COPY pnpm-lock.yaml ./
RUN --mount=type=cache,target=/root/.pnpm-store \
    pnpm fetch

Docker compose file:

version: "3"

services:
  project:
    build:
      context: ../
      dockerfile: .devcontainer/Dockerfile
    init: true
    volumes:
      - ..:/workspace:cached
      - /workspace/node_modules/

Then after the container is up, running pnpm install --frozen-lockfile outputs the following:

Running the postCreateCommand from devcontainer.json...

[18635 ms] Start: Run in container: /bin/sh -c .devcontainer/post-create.sh
apps/admin-backend                       |  WARN  Unsupported engine: wanted: {"node":"12.x"} (current: {"node":"v16.6.1","pnpm":"6.11.5"})
Scope: all 12 workspace projects
Lockfile is up-to-date, resolution step is skipped
Packages: +3362
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 WARN  EXDEV: cross-device link not permitted, link '/root/.pnpm-store/v3/files/b0/8e9da7c516d186ae38b15f4ba796b524fbe8a7fb50775355d645b8c19f70038965374d9f1094f0aa02de94375481c0f9153aed5c48b3fe40ffe84d636790e6' -> '/workspace/node_modules/.pnpm/[email protected]/node_modules/_tmp_191_25608a6e88ce30bd17d5cb4bc70d7ea8/index.js'
Falling back to copying packages from store
Packages are copied from the content-addressable store to the virtual store.
  Content-addressable store is at: /root/.pnpm-store/v3
  Virtual store is at:             node_modules/.pnpm
 WARN  EXDEV: cross-device link not permitted, link '/root/.pnpm-store/v3/files/81/ffee7b54b11b42aaeeaf1b6fb13ac64e8900e94ffd249fe075c183c3adf8e8451529d82f6a13216c73e5d1a1f57d703d9fcf1e61674020d93b15066d37dc75' -> '/workspace/node_modules/.pnpm/[email protected]/node_modules/_tmp_191_bef82bfbe26d1d1283fe4c2b48882f04/package.json'
Falling back to copying packages from store
 WARN  EXDEV: cross-device link not permitted, link '/root/.pnpm-store/v3/files/a9/13511b9900c171c161cdf30f4948efb758f6323b796317e32c8c555ea0f339f53b782baa8ce649d20bffbc5db875d0c85faa89cb6c12a94ab6e6c542448e64' -> '/workspace/node_modules/.pnpm/[email protected]/node_modules/_tmp_191_0472b883c04ee859090d9ee6eb1942c8/README.md'
Falling back to copying packages from store
 WARN  EXDEV: cross-device link not permitted, link '/root/.pnpm-store/v3/files/84/28597cbf30c0d8d4526b8b989553f874ebe7e9a79f0dfa6e008eb4871bdc9d9daf860fb033614decff96a2d5d5e594054688f0abf9df16487c55279b311d83' -> '/workspace/node_modules/.pnpm/[email protected]/node_modules/_tmp_191_e7fd6bddab0d1e89825858df864729be/package.json'
Falling back to copying packages from store
Falling back to copying packages from store
 WARN  12 other warnings
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Falling back to copying packages from store
Progress: resolved 3362, reused 0, downloaded 1071, added 1067
Downloading registry.npmjs.org/typescript/4.3.5: 3.38 MB/10.6 MB
Downloading registry.npmjs.org/antd/4.15.6: 3.05 MB/7.88 MB
etc etc

findmnt output:

➜  /workspace git:(develop) findmnt /root/.pnpm-store
➜  /workspace git:(develop) findmnt /workspace
TARGET     SOURCE                                                 FSTYPE        OPTIONS
/workspace grpcfuse[/Users/carlos/Dev/leadgogo/leadgogo-monorepo] fuse.grpcfuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576
➜  /workspace git:(develop) findmnt /workspace/node_modules
TARGET                  SOURCE                                                                                            FSTYPE OPTIONS
/workspace/node_modules /dev/vda1[/docker/volumes/d356d3f0ac305e8c6f3042f44023964171b646746b58c42055d013e1d2ed800a/_data] ext4   rw,relatime

If I understand correctly, since the pnpm store and node_modules exist on different device mounts, the pre-cached packages can't be used and therefore they are all being re-downloaded during pnpm install --frozenlockfile.

[amlcodes]

What is the protocol here?

[sanmai-NL]

You could cp -a the cache with hardlinks in a RUN instruction with cache mount, then in a following RUN instruction without cache mount move it to the original location.

[tysonclugg]

The current behaviour of pnpm install within the Microsoft suggested devcontainer layout for Debian ("image": "mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm") with Node.js ("features": {"ghcr.io/devcontainers/features/node:1": {}}) is:

1. Create node_modules within the workspace folder (/workspaces/foo/node_modules).
2. Determine that /workspaces/foo/node_modules isn't on the same mount point as ~/.cache/pnpm so instead create /workspaces/foo/.pnpm-store so hardlinks work as expected.

Would it be better for step 1 that pnpm creates /workspaces/foo/node_modules as a symbolic link to a directory somewhere under ~/.cache/pnpm instead?