support pnp resolver #1876

gun-yu · 2025-10-11T18:56:59Z

This PR adds support for PnP resolution.

#1875 in the previous discussion, I have internalized pnp-go into the repository.

Please note that the PR has become quite large due to the inclusion of the PnP source and test code. I apologize for the length and appreciate your understanding.

what is pnp

Yarn Plug’n’Play (PnP) is a dependency resolution system that removes the need for a traditional node_modules folder.
describe at #460

how to support

add pnp resolution config in host

type host struct {
	orchestrator *Orchestrator
	host         compiler.CompilerHost
	// Caches that last only for build cycle and then cleared out
	extendedConfigCache tsc.ExtendedConfigCache
	sourceFiles         parseCache[ast.SourceFileParseOptions, *ast.SourceFile]
	configTimes         collections.SyncMap[tspath.Path, time.Duration]

	// caches that stay as long as they are needed
	resolvedReferences parseCache[tspath.Path, *tsoptions.ParsedCommandLine]
	mTimes             *collections.SyncMap[tspath.Path, time.Time]
	resolvedReferences  parseCache[tspath.Path, *tsoptions.ParsedCommandLine]
	mTimes              *collections.SyncMap[tspath.Path, time.Time]
	pnpResolutionConfig *pnp.ResolutionConfig
}

add pnp branch in resolver.go

// resolveTypeReferenceDirective
// skip typeRoots because PnP knows exactly where each @types is located.
	if r.resolver.pnpResolutionConfig != nil {
		if resolvedFromNearestNodeModulesDirectory := r.loadModuleFromNearestNodeModulesDirectory(true /*typesScopeOnly*/); !resolvedFromNearestNodeModulesDirectory.shouldContinueSearching() {
			return r.createResolvedTypeReferenceDirective(resolvedFromNearestNodeModulesDirectory, true /*primary*/)
		}
	} else {
		if len(typeRoots) > 0 {
			if r.tracer != nil {
				r.tracer.write(diagnostics.Resolving_with_primary_search_path_0.Format(strings.Join(typeRoots, ", ")))
			}

// loadModuleFromNearestNodeModulesDirectory
if r.resolver.pnpResolutionConfig != nil {
			if result := r.loadModuleFromPNP(priorityExtensions, typesScopeOnly); !result.shouldContinueSearching() {
				return result
			}
		} else {
			if result := r.loadModuleFromNearestNodeModulesDirectoryWorker(priorityExtensions, mode, typesScopeOnly); !result.shouldContinueSearching() {
				return result
			}
		}

gun-yu · 2025-10-11T18:59:20Z

internal/project/ata/ata.go

 		logger.Log(fmt.Sprintf("ATA:: Installed typings %v", packageNames))
 		var installedTypingFiles []string
-		resolver := module.NewResolver(ti.host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "")
+		resolver := module.NewResolver(ti.host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "", nil)


todo: add pnp support later

Unless you plan to do it in this PR, we this should have a comment in the code.

internal/module/resolver.go

gun-yu · 2025-10-11T19:17:57Z

internal/module/pnp/lib_test.go

+}
+
+// need fixtures to be yarn install and make global cache
+func TestGlobalCache(t *testing.T) {


These test cases were taken from pnp-rs, but since they are relatively complex to run and maintain.
I think it would be fine to remove them if they’re not considered necessary.
Please let me know your thoughts!

gun-yu · 2025-10-12T14:32:09Z

@microsoft-github-policy-service agree

gun-yu · 2025-10-13T01:17:58Z

internal/compiler/host.go

+	pnpResolutionConfig := TryGetPnpResolutionConfig(currentDirectory)
+
+	if pnpResolutionConfig != nil {
+		fs = pnpvfs.From(fs)


On second thought, this way I wouldn’t be able to take advantage of the cached VFS, so I’m thinking of moving its location instead.

jakebailey · 2025-10-13T04:43:45Z

You said some of this comes from pnp-rs. Is it a pure port? What is the license of that code? Do you own it? (You signed the CLA, but that may not be enough if it's not yours.)

gun-yu · 2025-10-13T07:00:55Z

@jakebailey
The original source code, pnp-rs, is licensed under the BSD 2-Clause License. As the ported code is derived from that source, it should also fall under the BSD 2-Clause License — I missed that part, thank you for pointing it out.

As far as I understand, this means I need to include the original author’s license and copyright notice in the comments.

Would this cause any issues? If everything is fine, I’ll go ahead and add the appropriate BSD 2-Clause License notice.

jakebailey · 2025-10-13T07:21:08Z

We could pull it, but would need to explicitly declare that dependence in some extra metadata to generate NOTICE.txt (something we have avoided entirely by ensuring all code was written by us from spec or tests, but it's not really different than other deps).

But you definitely need to declare that there are files derived from code under a different license.

…escript-go into feat/add-pnp-resolver

gun-yu · 2025-10-13T18:56:05Z

@jakebailey
Sorry for the late reply. I just want to confirm my understanding—would it be sufficient to directly add a notice for the BSD 2-Clause License of pnp-rs in the NOTICE.txt file?

jakebailey · 2025-10-13T19:46:18Z

No, the files containing the code need to have headers with said license. For example, these tests:

typescript-go/internal/jsnum/ryu_test.go

Line 3 in a05e479

…escript-go into feat/add-pnp-resolver

gun-yu · 2025-10-14T02:59:14Z

Thank you. I’ve added the license notice as suggested. @jakebailey
b368c1c

jakebailey

A "few" comments; my overall worry is that the code style is not so much similar to our code, but maybe that's okay.

(My reviewing it is not really saying yes or no to the PR, we haven't discussed it)

internal/module/pnp/utils/normalize_path.go

internal/module/pnp/utils/portable_path.go

internal/module/pnp/builtins.go

internal/vfs/pnpvfs/pnpvfs.go

go.mod

jakebailey · 2025-10-15T18:43:07Z

internal/module/pnp/lib.go

+	return "", false
+}
+
+var rePNP = regexp.MustCompile(`(?s)(const[\ \r\n]+RAW_RUNTIME_STATE[\ \r\n]*=[\ \r\n]*|hydrateRuntimeState\(JSON\.parse\()'`)


This scares me a bit; because of the regex, but also because this reads JS files? Does the spec really require such a thing?

Good point! esbuild uses the JS AST for parsing. Since I was primarily focused on porting pnp-rs, I initially brought the code over as-is, but as you mentioned in another comment, there are definitely some risky areas. I believe the TypeScript Go codebase also includes a JS parser, so I’ll try using that for parsing instead.

According to the official spec, in a Node.js environment, pnp.cjs patches Node’s module resolution, so it’s executed first to override the default behavior. In Go, however, this part needs to be implemented manually.

Ultimately, parsing pnp.cjs is necessary to read the metadata, so this part seems unavoidable.
If we parse it using an AST like esbuild does, it might be a bit more reliable — would that be okay?

That makes more sense to me. Though in general I really thought that PnP had been specified to just emit a plain JSON file...

a9dbc68
I’ve changed the implementation from using regex to parsing with an AST.
The original pnp-rs implementation (the official one by the Yarn team) also uses regex, so I believe the regex-based approach is stable.
However, if switching to the AST-based approach is considered better, I’m happy to proceed with it — the code works fine as is.

In my project, where pnp.cjs is around 200,000 lines, the regex implementation performed slightly better.

https://github.com/yarnpkg/pnp-rs was happy with a regex, then I don't mind behind happy with a regex. I don't know if this is even in the hot path. My main concern was just whether or not it was "stable".

I understand. pnp-rs uses regex while esbuild uses AST, and both are stable implementations.
As you mentioned, this isn’t a hot path and there shouldn’t be any significant difference in performance, so I think it’s fine.
I’ll keep the AST-based implementation as it is in the commit.

internal/module/pnp/lib.go

jakebailey · 2025-10-15T18:45:02Z

internal/module/pnp/manifest.go

+}
+
+func (r *RegexDef) compile() (*regexp.Regexp, error) {
+	return regexp.Compile(r.Source)


This will fail because regex is re2 syntax, not ECMAScript. You'd have to use regexp2 in ECMAScript mode.

That being said, is this really ever used? Does the PnP spec really require regex parsing? (How do esbuild and so on do this without pulling on totally different regex implementations?)

Thanks for the feedback!
The original ignorePatternData looks like this:

"ignorePatternData": "(^(?:\\\\.yarn\\\\/sdks(?:\\\\/(?!\\\\.{1,2}(?:\\\\/|$))(?:(?:(?!(?:^|\\\\/)\\\\.{1,2}(?:\\\\/|$)).)*?)|$))$)",

Esbuild also compiles and uses the regex in the same way. Since this is part of the Yarn PnP spec, it seems unavoidable to use a regular expression here.

For now, I’ve optimized this commit to avoid the inefficiency of compiling the regex on every use. f0ee55b

If esbuild gets away with it, then that's fine I suppose.

Out of curiosity, is this string always the same?

According to the specification, I understand that this value is dynamic.
yarn berry

gun-yu · 2025-10-16T00:41:10Z

@jakebailey Thank you for the thoughtful review! I know feedback like this takes a lot of time, and I truly appreciate it. Reading your comments made me realize how unfamiliar I still am with the TypeScript Go codebase—I’m sorry I didn’t reference the existing code more. Consistent code style is especially important in open source, so I’m happy to adopt all of your suggestions. I’ll start by addressing your comments.

gun-yu · 2025-10-18T18:25:45Z

@jakebailey
I believe I’ve incorporated all the review feedback.
Please let me know if there’s anything I might have overlooked.

jakebailey · 2025-10-20T21:42:00Z

internal/compiler/pnp.go

+	"os"
+	"path/filepath"


The compiler should not depend on these; any accesses it makes must go through a provided FS.

(#1911 will enforce this in the future)

jakebailey · 2025-10-20T23:00:52Z

internal/module/pnp/manifest.go

+	IsAlias   bool
+}
+
+func (p *PackageDependency) UnmarshalJSON(data []byte) error {


Not strictly required, but all of these UnmarshalJSON functions would perform better and probably be simpler to write using the v2 versions of this, as in UnmarshalJSONFrom(dec *jsontext.Decoder) error).

That's not exactly what I meant per se; you want to read from the input in a more streaming manner. I would look at other instances of the method, since by calling Value you are just doing what the old code did and double decoding.

e697dce
I did some work on it, but the code has gotten quite complicated, so I'm not sure if it's heading in the right direction. There might be some minor memory or performance gains, but I haven’t benchmarked it yet.

jakebailey · 2025-10-20T23:02:37Z

internal/module/resolver.go


 func ResolveConfig(moduleName string, containingFile string, host ResolutionHost) *ResolvedModule {
-	resolver := NewResolver(host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "")
+	resolver := NewResolver(host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "", nil)


This is an interesting one; nil here means that one can't load tsconfigs through PnP, which is probably fine, but I am curious how the old PnP patch did this (if at all).

I actually wanted to get feedback on this part, but I forgot to mention it in the description — thanks for pointing it out!

I think adding PnP resolution would be the right approach, but for now, I set it to nil because I wasn’t sure how best to pass the pnpResolutionConfig from the host to NewResolver given the current structure.

(I’ll also mention this here since the concern is similar to the zipvfs layer you pointed out.)

I considered two possible approaches:

Inject both pnpResolutionConfig and zipvfs externally.

Handle them inside NewCompilerHost (which is how it’s currently implemented in this PR).

The first approach would make the PnP-related code spread across quite a few parts of the system, which I’d prefer to avoid — I’d like to keep PnP logic as isolated as possible.

The second approach avoids scattering PnP logic across tsgo, but it introduces some implicit behavior in NewCompilerHost and prevents taking advantage of cachedvfs.

From a maintainer’s perspective, do you have any thoughts on which direction would be preferable?

jakebailey · 2025-10-20T23:04:03Z

internal/module/resolver_pnp.go

+	// trim trailing slash makes a bug in packageJsonInfoCache.Set
+	// like @emotion/react/ -> @emotion/react after packageJsonInfoCache.Set
+	// check why it's happening in packageJsonInfoCache and need to fix it


Yes, because the cache I think assumes normalized paths with trailing trimmed. Definitely need to ensure paths are minimal like that. I think.

jakebailey · 2025-10-20T23:04:28Z

internal/project/ata/ata.go

 		logger.Log(fmt.Sprintf("ATA:: Installed typings %v", packageNames))
 		var installedTypingFiles []string
-		resolver := module.NewResolver(ti.host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "")
+		resolver := module.NewResolver(ti.host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "", nil)


Unless you plan to do it in this PR, we this should have a comment in the code.

jakebailey · 2025-10-20T23:07:56Z

internal/compiler/host.go

+	if pnpResolutionConfig != nil {
+		fs = zipvfs.From(fs)
+	}


I do wonder if this is the right layer for this. Surely the language server also needs to know this information? For example, we need to read random files to do source mapping in the LS layer, which would have to dig into zips.

But, we don't want cached info to be there indefinitely either.

I also don't know where this should go in relation to the caching layer...

go.mod

gun-yu · 2025-10-21T00:28:58Z

Unless you plan to do it in this PR, we this should have a comment in the code.

Once the direction for the NewCompilerHost design is decided, I think I can add everything.

However, I consider the ata file related to LSP behavior, and I haven’t really tested LSP properly yet, nor do I fully understand it, so I’m a bit concerned about that part.

I’m fine either working on it immediately or continuing in the next PR. I can proceed according to your guidance.

gun-yu added 3 commits October 11, 2025 21:11

add pnp resolution support

c315580

remove fmt

99dc36c

pnp internalize

25b13ac

gun-yu commented Oct 11, 2025

View reviewed changes

remove useless code

f24a170

gun-yu changed the title ~~Feat/add pnp resolver~~ support pnp resolver Oct 11, 2025

remove useless code

b4aaac8

gun-yu commented Oct 11, 2025

View reviewed changes

internal/module/resolver.go Show resolved Hide resolved

gun-yu commented Oct 11, 2025

View reviewed changes

gun-yu and others added 3 commits October 12, 2025 04:30

use go-json-experiment/json

2373552

lint fix

f09cda5

Merge branch 'main' into feat/add-pnp-resolver

d00e36b

gun-yu added 2 commits October 12, 2025 23:43

change test data path

0a0f050

fix hereby format

94a15b7

gun-yu commented Oct 13, 2025

View reviewed changes

gun-yu added 3 commits October 13, 2025 16:40

fix hereby check:format

35d6188

fix window test

35ee2ef

Merge branch 'feat/add-pnp-resolver' of https://github.com/gun-yu/typ…

9463e11

…escript-go into feat/add-pnp-resolver

gun-yu and others added 3 commits October 14, 2025 09:54

Merge branch 'main' into feat/add-pnp-resolver

dd08ae5

add License

b368c1c

Merge branch 'feat/add-pnp-resolver' of https://github.com/gun-yu/typ…

688455e

…escript-go into feat/add-pnp-resolver

jakebailey reviewed Oct 15, 2025

View reviewed changes

gun-yu and others added 9 commits October 16, 2025 10:55

remove filepath and utils

305fd49

make regex more efficient

f0ee55b

fix lint

ef6d713

Merge branch 'main' into feat/add-pnp-resolver

8d165ca

apply codereview

ae272f1

generate error message

67752dc

add error message

1f06041

parse pnp.cjs file, regex to ast

a9dbc68

make go version 1.25.0

df40d62

remove useless test

1485fe3

gun-yu requested a review from jakebailey October 18, 2025 18:36

jakebailey reviewed Oct 20, 2025

View reviewed changes

gun-yu added 2 commits October 21, 2025 10:59

apply UnmarshalJSONFrom

bfa05c3

apply UnmarshalJSONFrom in manifest

e697dce

support pnp resolver #1876

Are you sure you want to change the base?

support pnp resolver #1876

Conversation

gun-yu commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

what is pnp

how to support

add pnp resolution config in host

add pnp branch in resolver.go

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gun-yu commented Oct 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakebailey commented Oct 13, 2025

Uh oh!

gun-yu commented Oct 13, 2025

Uh oh!

jakebailey commented Oct 13, 2025

Uh oh!

gun-yu commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakebailey commented Oct 13, 2025

Uh oh!

gun-yu commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakebailey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gun-yu Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gun-yu Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gun-yu commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gun-yu commented Oct 11, 2025 •

edited

Loading

gun-yu commented Oct 13, 2025 •

edited

Loading

gun-yu commented Oct 14, 2025 •

edited

Loading

gun-yu Oct 16, 2025 •

edited

Loading

gun-yu Oct 21, 2025 •

edited

Loading

gun-yu commented Oct 16, 2025 •

edited

Loading