yarn focus and --immutable

[cakoose]

We have a Yarn 4.x workspaces monorepo, with TypeScript and the node_modules linker.

To reduce the size of the container image, I'm considering replacing yarn install with yarn focus and passing in just the names of the workspaces that I need in my container.

We currently do yarn install --immutable to ensure (I think?) that the installed packages match what's in my "yarn.lock". It appears that yarn focus does not have such an option.

I found an old GitHub issue (#6277) where someone is saying that --immutable doesn't make sense for yarn focus:

A focused install, almost by definition, would require the lockfile to be updated. A focused install literally mutates the in-memory data structures like parsed package.jsons, then runs a normal install. That would produce a different lockfile, we just don't write that back to disk.

I think I just want it to be have the same as yarn install, except exclude any packages that aren't needed by the workspaces that I have focus'ed. Is that possible?

[clemyan]

Specifically for your use case, you should just use yarn install --immutable to validate the lockfile, then yarn workspaces focus ... to create the final image.

You wouldn't put your linter in the final image, would you? No, you run the linter while you still have your dev-dependencies, then build the final image without dev-dependencies. By the time you build the final image, you should already know the project passes the linter, so you don't need the lint again in the final image.

In the same sense, you shouldn't need a immutable focused install. By the time you focus, you should already know the project passes the --immutable check, so you don't need to check again in the final image.

---

Now let's talk more generally about immutable focused installs. After the part you quoted, I also said

So you don't want an immutable focused install, you want a focused install where some mutations are okay and other are not. The question then becomes what mutations are okay and how to determine if a given change is okay or not.

Immutable installs are fundamentally about detecting mutations (i.e. changes) between two states: the state immediately after the last unfocused install ("before") when the lockfile was generated, and the state that is currently on-disk ("after"). Any mutation that would affect resolution is rejected. Such mutation may be external (e.g. registry went down) or caused by changes on-disk (e.g. a modified package.json).

(Note: While we don't have the full "before" state of the project, the lockfile on-disk is a good enough proxy because it records all resolution that happend during the last unfocused install)

The problem with immutable focused installs is that mutations can also be caused by the focusing itself. A immutable focused install needs to ignore those mutations while still being able to reject external and on-disk mutation. However it is not always possible to distinguish between the two.

And "exclude any packages that aren't needed" probably isn't what you want. I can take a workspace and remove all its dependencies. Now all packages are unneeded, so a immutable focused install should pass?

For some less extreme cases, say we start with a project that passes yarn install --immutable. Which of these should pass?

1. I manually change the package.json to remove a dependency, then focus --production --immutable
2. I manually change the package.json to remove a devDependency, then focus --production --immutable
3. I manually change the package.json to move a package from dependency to devDependency, then focus --production --immutable
4. I manually change the package.json to move a package from dependency to devDependency, then focus --immutable, then focus --production --immutable
5. I manually change the pacakage.json of an unfocused workspace to remove a dependency, then focus --immutable

Some of these aren't distinguishable from one another, or distinguisable from "run focus --immutable without changing anything". I even suspect not all people agree what is "correct" in some more extreme edge cases.

Most people who ask for immutable focused install think that's just "immutable install, but ignore these changes". But no, it is "ignore this change if it is caused by this reason, but don't ignore the exact same change if it is caused by that reason"

At the very least, we need to record more information about the project state during unfocused installs so that a future focused install can detect why mutations happen. But it isn't clear what needs to be recorded. Depending on the exact semantics of immutable focused installs (which, again I don't think there is a consensus on), the complexity can range from "lockfile format changes" to "re-architecting the entire core" to "literally impossible without a time machine"

In any case, if you have a specific project that need a specific form of immutable focused installs, you can implement it as a plugin. But at this point, I'm don't think I can be convinced that a first-party immutable focused install is a good idea until I see a PR that people can point to and say "that's what I'm talking about when I say immutable focused installs"

[cakoose]

I think when people (like me) are asking about --immutable, we're not experts in Yarn concepts. It's coming from the perspective of:

• To make sure there aren't inconsistencies between the package.json and the yarn.lock in CI, we're told to yarn install --immutable.
• To only install the dependencies for a subset of workspaces, we're told to use yarn focus instead of yarn install.

So I end up wondering: if I use yarn focus instead of yarn install --immutable, do I lose the consistency check?

What I gather from your message:

• Yes, I lose the consistency check.
• To recover it, I should do a separate yarn install --immutable
• If it fails, then the whole build should fail.

If the yarn install --immutable succeeds, can I assume that a yarn focus on a fresh copy of the same repo will install the exact same package versions as yarn install --immutable except without the packages that aren't needed by the focused workspaces?

And "exclude any packages that aren't needed" probably isn't what you want. I can take a workspace and remove all its dependencies. Now all packages are unneeded, so a immutable focused install should pass?

Are you referring to me saying "except exclude any packages that aren't needed by the workspaces that I have focus'ed"?

I don't follow how removing all dependencies makes sense there. Is there another way I should have phrased it?

[clemyan]

So I end up wondering: if I use yarn focus instead of yarn install --immutable, do I lose the consistency check?

• Yes, I lose the consistency check.
• To recover it, I should do a separate yarn install --immutable
• If it fails, then the whole build should fail.

Correct. Checking consistency requires resolving all dependencies, and a focused install explicitly does not resolve all dependencies, making the two incompatible.

If the yarn install --immutable succeeds, can I assume that a yarn focus on a fresh copy of the same repo will install the exact same package versions as yarn install --immutable except without the packages that aren't needed by the focused workspaces?

Yes, as long as the all relevant package.json and yarn.lock were not modified after the immutable install

Are you referring to me saying "except exclude any packages that aren't needed by the workspaces that I have focus'ed"?
I don't follow how removing all dependencies makes sense there. Is there another way I should have phrased it?

Whenever someone requests focus --immutable as a feature. The idea that it should behave "the same as yarn install [--immutable], except exclude any packages that aren't needed by the workspaces that I have focus'ed" is commonly suggested. I'm pointing out that, if taken literally, it is probably not what you actually want.

Consider this scenario:

1. I have a project with a single workspace with a few dependencies.
2. I run yarn install to generate a lockfile. Now I have a project that passes the yarn --immutable check.
3. I manually edit the package.json to remove all the dependencies.
4. I run the hypothetical yarn workspaces focus --immutable.

Question: Should step 4 succeed or fail?

Well, since we have removed all dependencies, no packages are needed by our project. So, to "exclude any packages that aren't needed by the focus'ed workspaces" is to ignore all packages in the check and thus step 4 should pass.

I don't know about you in particular, but I would guess most of those who request focus --immutable would say they want this case to fail.

So it's a case of "being unable to precisely define with words what one understand only by intuition". It is quite a common occurrence in the study of philosophy. For example, a "typical" exchange when introducing someone to the Trolley Problem goes like this:

A: A runaway trolley is speeding down a track and will hit and kill 5 people. You are standing by a lever that would change its course to a different track, but then the trolley will hit and kill 1 person. What is the morally correct choice? Would you switch the track?
B: Of course I would. 1 person dying is obviously better than 5 people dying.
A: So, if killing a healthy man to harvest his organs would save 5 critically injured people, that would be the morally correct thing to do, right?
B: Well, no...
A: In terms of "number of people dying" those two scenarios are the same. So the number of people dying is not what you actually base your morals on in these scenarios. You intuitively understand what "sounds right", but when you try to put it into words you say things you don't actually mean.

It would then lead to a long conversation resulting in B better understanding the moral framework they are operating under, or at least thinking more deeply about that. Most importantly, morals are subjective -- there isn't a single answer that all people agree is the correct moral framework.

But I digress. My point is that I think "immutable focused install" is a thing like that -- people who requests it as a feature only understand what they want by intuition, and when asked to precisely define that with words they won't be able to. Moreover, there is no single, objectively correct answer to "what is this 'immutable focused install' you want?".

I understand that, from a Yarn user perspective, immutable install and focused install looks orthogonal and immutable focused install sound trivial to implement. But unless and until we have that single answer, requests for a first-party immutable focused install feature are unactionable.

[cakoose]

Thanks, that's exactly what I was wondering!

So for my GitHub Actions CI build, I have a couple options:

1. Two jobs:
   • One job that does yarn install --immutable to check for consistency.
   • One job that builds the container image with yarn focus ...
   • Only allow using the built container image if both jobs succeed.
2. One job:
   • Build the container by first doing:
     - yarn install --immutable
     - rm -r node_modules (do I also have to delete the "node_modules" folders in each sub-package?)
     - yarn focus ...

One downside is that the yarn install --immutable will download many more packages than I actually need for this particular build.

Is it possible (just in theory) to accomplish the same end result without downloading all those extra packages? Like, if I were an expert in Yarn internals, would I be able to add a Yarn subcommand or flag that could do that? Or do we actually need to download the packages (or at least each package's "package.json") to do that?

[clemyan]

I'm not that well-versed in GH Actions but both options sounds reasonable. I think it depends on whether there are multiple jobs that depend on having a verifiably consistent project.

Without having access to the package.jsons of all packages in the projects, it won't be possible to verify the whole lockfile. If you want to verify only part of the lockfile then the question is once again "which part(s)?"

If you have a precise definition of what "immutable focused install" means to you/your team then you can probably implement it as a new command. Let's say, to me, "immutable focused install" means "do a focused install and check if we've only removed data from the lockfile, not added any new data". That can be implemented relatively easily. But the "removing all dependency" scenario also passes this check. If I think that's fine then I have my immutable focused install command.

Again, from my perspective, the problem with immutable focused install is more "what to implement" than "how to implement". A first-party implementation that disagrees with users' expectation is worse than no implementation.