Updates vfs to be more future safe when running on node + 2021 dts files for playground #2802

orta · 2023-04-25T07:03:48Z

On node a readdir is possible, so we can ignore the known libs completely, making it future proof
For the browser (aka the playground), the vfs need to know about a bunch of new lib files

…include new .d.ts files on the playground

packages/typescript-vfs/src/index.ts

jakebailey · 2023-04-25T23:44:06Z

Truthfully, I don't know the interaction between this file listing and what the playground will use, though; I assume they are related? But, monaco has its own list. So hopefully this won't break?

orta · 2023-04-26T14:29:39Z

The CIs are red because I declared an update to the package json version, I'll yarn install to hit the lockfile.

Good point, the playground uses its own ts.Program instance for thing like js/dts emit / ATA etc which is created using the tsvfs dts files, which the program is then used for a bunch of introspection stuff.

jakebailey · 2023-04-27T17:18:18Z

Turns out that libMap was not exhaustive; new API pushed to microsoft/TypeScript#54011 but I guess we'll wait until it's finalized.

I'll work on a snippet that can produce the same-ish list but in old versions. I'll need it for monaco anyway.

jakebailey · 2023-04-27T18:31:19Z

It turns out that there's no way that works across versions to get a full listing of the lib files in the package. The only way I think we can get the same result is to use the new API if available, or as a fallback, encode a huge list of all lib files that have ever existed and verify that they do actually exist before using them.

That full list is:

[
    "lib.d.ts",
    "lib.decorators.d.ts",
    "lib.decorators.legacy.d.ts",
    "lib.dom.d.ts",
    "lib.dom.iterable.d.ts",
    "lib.es2015.collection.d.ts",
    "lib.es2015.core.d.ts",
    "lib.es2015.d.ts",
    "lib.es2015.generator.d.ts",
    "lib.es2015.iterable.d.ts",
    "lib.es2015.promise.d.ts",
    "lib.es2015.proxy.d.ts",
    "lib.es2015.reflect.d.ts",
    "lib.es2015.symbol.d.ts",
    "lib.es2015.symbol.wellknown.d.ts",
    "lib.es2016.array.include.d.ts",
    "lib.es2016.d.ts",
    "lib.es2016.full.d.ts",
    "lib.es2017.d.ts",
    "lib.es2017.full.d.ts",
    "lib.es2017.intl.d.ts",
    "lib.es2017.object.d.ts",
    "lib.es2017.sharedmemory.d.ts",
    "lib.es2017.string.d.ts",
    "lib.es2017.typedarrays.d.ts",
    "lib.es2018.asyncgenerator.d.ts",
    "lib.es2018.asynciterable.d.ts",
    "lib.es2018.d.ts",
    "lib.es2018.full.d.ts",
    "lib.es2018.intl.d.ts",
    "lib.es2018.promise.d.ts",
    "lib.es2018.regexp.d.ts",
    "lib.es2019.array.d.ts",
    "lib.es2019.d.ts",
    "lib.es2019.full.d.ts",
    "lib.es2019.intl.d.ts",
    "lib.es2019.object.d.ts",
    "lib.es2019.string.d.ts",
    "lib.es2019.symbol.d.ts",
    "lib.es2020.bigint.d.ts",
    "lib.es2020.d.ts",
    "lib.es2020.date.d.ts",
    "lib.es2020.full.d.ts",
    "lib.es2020.intl.d.ts",
    "lib.es2020.number.d.ts",
    "lib.es2020.promise.d.ts",
    "lib.es2020.sharedmemory.d.ts",
    "lib.es2020.string.d.ts",
    "lib.es2020.symbol.wellknown.d.ts",
    "lib.es2021.d.ts",
    "lib.es2021.full.d.ts",
    "lib.es2021.intl.d.ts",
    "lib.es2021.promise.d.ts",
    "lib.es2021.string.d.ts",
    "lib.es2021.weakref.d.ts",
    "lib.es2022.array.d.ts",
    "lib.es2022.d.ts",
    "lib.es2022.error.d.ts",
    "lib.es2022.full.d.ts",
    "lib.es2022.intl.d.ts",
    "lib.es2022.object.d.ts",
    "lib.es2022.regexp.d.ts",
    "lib.es2022.sharedmemory.d.ts",
    "lib.es2022.string.d.ts",
    "lib.es2023.array.d.ts",
    "lib.es2023.d.ts",
    "lib.es2023.full.d.ts",
    "lib.es5.d.ts",
    "lib.es6.d.ts",
    "lib.esnext.array.d.ts",
    "lib.esnext.asynciterable.d.ts",
    "lib.esnext.bigint.d.ts",
    "lib.esnext.d.ts",
    "lib.esnext.full.d.ts",
    "lib.esnext.intl.d.ts",
    "lib.esnext.promise.d.ts",
    "lib.esnext.string.d.ts",
    "lib.esnext.symbol.d.ts",
    "lib.esnext.weakref.d.ts",
    "lib.scripthost.d.ts",
    "lib.webworker.d.ts",
    "lib.webworker.importscripts.d.ts",
    "lib.webworker.iterable.d.ts"
]

orta · 2023-04-28T17:09:05Z

I've added code like:

 const files =
    "getAllLibFileNames" in ts
      ? // @ts-ignore - see https://github.com/microsoft/TypeScript/pull/54011
        (ts.getAllLibFileNames() as string[])
      : [
          "lib.d.ts",
          "lib.decorators.d.ts",
          "lib.decorators.legacy.d.ts",
          "lib.dom.d.ts",
          "lib.dom.iterable.d.ts",

As that fn seems to be the one which is just the file name equivalents in your PR

jakebailey · 2023-04-28T17:39:21Z

packages/typescript-vfs/src/index.ts

+    "getAllLibFileNames" in ts
+      ? // @ts-ignore - see https://github.com/microsoft/TypeScript/pull/54011
+        (ts.getAllLibFileNames() as string[])
+      : [


Does this list get filtered in any way to the actual set based on what's available? I'm not sure I totally know how this list is used.

Yeah, there's a rough heuristic which cuts down the list based on target + lib - just below:

const targetToCut = ts.ScriptTarget[target] const matches = files.filter(f => f.startsWith(`lib.${targetToCut.toLowerCase()}`)) const targetCutIndex = files.indexOf(matches.pop()!) const getMax = (array: number[]) => array && array.length ? array.reduce((max, current) => (current > max ? current : max)) : undefined // Find the index for everything in const indexesForCutting = lib.map(lib => { const matches = files.filter(f => f.startsWith(`lib.${lib.toLowerCase()}`)) if (matches.length === 0) return 0 const cutIndex = files.indexOf(matches.pop()!) return cutIndex }) const libCutIndex = getMax(indexesForCutting) || 0 const finalCutIndex = Math.max(targetCutIndex, libCutIndex) return files.slice(0, finalCutIndex + 1) }

So the order is important WRT downloading which have some tests covering the cases, but the rough gist is that the CDN version will download all of the files (and cache them locally if possible) and put them into the VFS which TypeScript will then use synchronously later

I just added some checks for handle CDNs returning NOOPs for dts files which dont exist for that ts version

jakebailey · 2023-07-24T22:40:04Z

I haven't had time to revisit the TS API for this quite yet; in the meantime to fix everyone, maybe we'd be better off keeping that full list without checking that new API, and we can come back to adding a conditional use later?

I'm hoping to get all of the PRs that need to be published to npm out somewhat soon.

orta · 2023-07-25T14:46:36Z

Yep, this PR keeps the full list and if an API called getAllLibFileNames ever ends up in TypeScript, it will prefer that instead, so kinda future protected for it 👍🏻

jakebailey · 2023-07-25T15:35:17Z

Right, I mean that I'm not sure that function is even the one that's going to go into TS itself, so I was thinking we'd just delete that reference for now.

That and I want to double check that the files list is up to date for 5.3 which I'll do shortly.

jakebailey · 2023-07-25T18:28:48Z

Here is an updated array of all lib.d.ts files to use in the PR:

[
    "lib.d.ts",
    "lib.decorators.d.ts",
    "lib.decorators.legacy.d.ts",
    "lib.dom.d.ts",
    "lib.dom.iterable.d.ts",
    "lib.es2015.collection.d.ts",
    "lib.es2015.core.d.ts",
    "lib.es2015.d.ts",
    "lib.es2015.generator.d.ts",
    "lib.es2015.iterable.d.ts",
    "lib.es2015.promise.d.ts",
    "lib.es2015.proxy.d.ts",
    "lib.es2015.reflect.d.ts",
    "lib.es2015.symbol.d.ts",
    "lib.es2015.symbol.wellknown.d.ts",
    "lib.es2016.array.include.d.ts",
    "lib.es2016.d.ts",
    "lib.es2016.full.d.ts",
    "lib.es2017.d.ts",
    "lib.es2017.date.d.ts",
    "lib.es2017.full.d.ts",
    "lib.es2017.intl.d.ts",
    "lib.es2017.object.d.ts",
    "lib.es2017.sharedmemory.d.ts",
    "lib.es2017.string.d.ts",
    "lib.es2017.typedarrays.d.ts",
    "lib.es2018.asyncgenerator.d.ts",
    "lib.es2018.asynciterable.d.ts",
    "lib.es2018.d.ts",
    "lib.es2018.full.d.ts",
    "lib.es2018.intl.d.ts",
    "lib.es2018.promise.d.ts",
    "lib.es2018.regexp.d.ts",
    "lib.es2019.array.d.ts",
    "lib.es2019.d.ts",
    "lib.es2019.full.d.ts",
    "lib.es2019.intl.d.ts",
    "lib.es2019.object.d.ts",
    "lib.es2019.string.d.ts",
    "lib.es2019.symbol.d.ts",
    "lib.es2020.bigint.d.ts",
    "lib.es2020.d.ts",
    "lib.es2020.date.d.ts",
    "lib.es2020.full.d.ts",
    "lib.es2020.intl.d.ts",
    "lib.es2020.number.d.ts",
    "lib.es2020.promise.d.ts",
    "lib.es2020.sharedmemory.d.ts",
    "lib.es2020.string.d.ts",
    "lib.es2020.symbol.wellknown.d.ts",
    "lib.es2021.d.ts",
    "lib.es2021.full.d.ts",
    "lib.es2021.intl.d.ts",
    "lib.es2021.promise.d.ts",
    "lib.es2021.string.d.ts",
    "lib.es2021.weakref.d.ts",
    "lib.es2022.array.d.ts",
    "lib.es2022.d.ts",
    "lib.es2022.error.d.ts",
    "lib.es2022.full.d.ts",
    "lib.es2022.intl.d.ts",
    "lib.es2022.object.d.ts",
    "lib.es2022.regexp.d.ts",
    "lib.es2022.sharedmemory.d.ts",
    "lib.es2022.string.d.ts",
    "lib.es2023.array.d.ts",
    "lib.es2023.collection.d.ts",
    "lib.es2023.d.ts",
    "lib.es2023.full.d.ts",
    "lib.es5.d.ts",
    "lib.es6.d.ts",
    "lib.esnext.array.d.ts",
    "lib.esnext.asynciterable.d.ts",
    "lib.esnext.bigint.d.ts",
    "lib.esnext.d.ts",
    "lib.esnext.decorators.d.ts",
    "lib.esnext.disposable.d.ts",
    "lib.esnext.full.d.ts",
    "lib.esnext.intl.d.ts",
    "lib.esnext.promise.d.ts",
    "lib.esnext.string.d.ts",
    "lib.esnext.symbol.d.ts",
    "lib.esnext.weakref.d.ts",
    "lib.scripthost.d.ts",
    "lib.webworker.d.ts",
    "lib.webworker.importscripts.d.ts",
    "lib.webworker.iterable.d.ts"
]

I think for now, I'd just like to get this list in as-is, no reference to that new API, and we can add it later. That way, we can release this and fix things up for everyone.

jakebailey · 2023-07-25T19:23:42Z

packages/typescript-vfs/src/index.ts

+        .then(contents => {
+          contents.forEach((text, index) => fsMap.set("/" + files[index], text))
+        })
+        // Return a NOOP for .d.ts files which aren't in the current build of TypeScript


Hm, looking back, is this right? Won't this fail if any of them fail?

jakebailey · 2023-07-25T20:55:16Z

With bump versions with #2904 to minimize the waiting within the current infrastructure.

jakebailey · 2023-07-25T21:16:36Z

Apparently ordering matters to pass tests. Hopefully it's the test that's wrong because otherwise, it's going to be challenging to fix because it's not like an FS-based API from TS for this will be returning things in any particular order...

github-actions · 2023-07-25T21:35:53Z

Azure Static Web Apps: Your stage site is ready! Visit it here: https://delightful-glacier-09aa73710-2802.centralus.azurestaticapps.net

github-actions · 2023-07-25T21:44:28Z

Azure Static Web Apps: Your stage site is ready! Visit it here: https://delightful-glacier-09aa73710-2802.centralus.azurestaticapps.net

github-actions · 2023-07-25T21:50:16Z

Azure Static Web Apps: Your stage site is ready! Visit it here: https://delightful-glacier-09aa73710-2802.centralus.azurestaticapps.net

orta · 2023-07-26T06:19:01Z

Yep, order is important to the implementation (which is why there's tests!): #2802 (comment)

Looks right to me though

orta added 2 commits April 25, 2023 08:00

Updates vfs to be more future safe when running on node, and to also …

a2e8661

…include new .d.ts files on the playground

Take tsLibDirectory into account

1d0bc44

Matsuuu mentioned this pull request Apr 25, 2023

@typescript/vfs: Module resolution on createProgram breaks #2801

Closed

jakebailey reviewed Apr 25, 2023

View reviewed changes

packages/typescript-vfs/src/index.ts Show resolved Hide resolved

jakebailey mentioned this pull request Apr 25, 2023

Export API to provide lib.d.ts listing/mapping microsoft/TypeScript#54011

Draft

typescript-bot closed this Apr 25, 2023

andrewbranch reopened this Apr 25, 2023

microsoft deleted a comment from typescript-bot Apr 26, 2023

Update lockfile

b5187a7

orta force-pushed the vfs_update branch from 2950277 to b5187a7 Compare April 26, 2023 18:53

Include all dts files ever shipped

d1dd3ca

jakebailey reviewed Apr 28, 2023

View reviewed changes

Handle the CDN potentially causing fetch throws

438d263

jakebailey mentioned this pull request May 19, 2023

Add lib files of es2022 to knownLibFilesForCompilerOptions #2458

Closed

Replace with new list

0488958

jakebailey reviewed Jul 25, 2023

View reviewed changes

jakebailey added 2 commits July 25, 2023 13:41

Merge branch 'v2' into pr-2802

fefe86e

Undo package.json changes, will push in a different PR

c2feaab

jakebailey mentioned this pull request Jul 25, 2023

Bump versions for recent changes #2904

Merged

4 tasks

jakebailey approved these changes Jul 25, 2023

View reviewed changes

jakebailey enabled auto-merge (squash) July 25, 2023 21:03

jakebailey added 3 commits July 25, 2023 14:09

Maybe order matters

abd38b7

Maybe order matters again

82a915f

Merge branch 'v2' into pr-2802

ec60dc5

jakebailey added the deploy-preview Enables automatic deployments to preview environments on a PR label Jul 25, 2023

jakebailey merged commit 4b1f83c into microsoft:v2 Jul 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates vfs to be more future safe when running on node + 2021 dts files for playground #2802

Updates vfs to be more future safe when running on node + 2021 dts files for playground #2802

orta commented Apr 25, 2023

jakebailey commented Apr 25, 2023

orta commented Apr 26, 2023

jakebailey commented Apr 27, 2023

jakebailey commented Apr 27, 2023

orta commented Apr 28, 2023

jakebailey Apr 28, 2023

orta May 2, 2023

jakebailey commented Jul 24, 2023

orta commented Jul 25, 2023

jakebailey commented Jul 25, 2023

jakebailey commented Jul 25, 2023

jakebailey Jul 25, 2023

jakebailey commented Jul 25, 2023

jakebailey commented Jul 25, 2023

github-actions bot commented Jul 25, 2023

github-actions bot commented Jul 25, 2023

github-actions bot commented Jul 25, 2023

orta commented Jul 26, 2023

Updates vfs to be more future safe when running on node + 2021 dts files for playground #2802

Updates vfs to be more future safe when running on node + 2021 dts files for playground #2802

Conversation

orta commented Apr 25, 2023

jakebailey commented Apr 25, 2023

orta commented Apr 26, 2023

jakebailey commented Apr 27, 2023

jakebailey commented Apr 27, 2023

orta commented Apr 28, 2023

jakebailey Apr 28, 2023

Choose a reason for hiding this comment

orta May 2, 2023

Choose a reason for hiding this comment

jakebailey commented Jul 24, 2023

orta commented Jul 25, 2023

jakebailey commented Jul 25, 2023

jakebailey commented Jul 25, 2023

jakebailey Jul 25, 2023

Choose a reason for hiding this comment

jakebailey commented Jul 25, 2023

jakebailey commented Jul 25, 2023

github-actions bot commented Jul 25, 2023

github-actions bot commented Jul 25, 2023

github-actions bot commented Jul 25, 2023

orta commented Jul 26, 2023