Add AsyncPCK support (`ResourceLoader.load_threaded_*` version) by adamscott · Pull Request #116673 · godotengine/godot

adamscott · 2026-02-23T16:28:44Z

Supersedes #114690

Changes from the previous PR

This PR is essentially #114690 rebased to master, but with a few changes following the TLC comments.

It removes the AsyncPckInstaller node which was an developer utility to speed up integration of Async in their project.
It removes the OS::async_pck_* methods from the public API.
It integrates async support with the ResourceLoader.load_threaded_* API.

The idea behind putting the AsyncPCK logic behind ResourceLoader.load_threaded_* was twofold:

to prevent adding new APIs.
to simplify the async loading for users.

Important caveats when using `ResourceLoader.load_threaded_*` instead of a dedicated AsyncPCK API

In my opinion, I think that the fact that load() cannot stop the engine on the Web¹ until the missing files are downloaded and installed really complicates everything.

The docs for ResourceLoader.load_threaded_* have now multiple caveats attached to them. (see screenshots below)
On single-threaded Web exports, as the download and the load phases are merged, when big assets are downloaded, the engine will heavily stutter to load them. There's currently no way to download files independently (during playing the level) from loading (during a loading or black screen).
Essentially forces the user to load every asset with the ResourceLoader.load_threaded_* API, effectively banning the simpler load() method.
There's currently no way to track how many bytes have been downloaded, and the bytes total of each files. The information has been constrained to the ResourceLoader.load_threaded_get_status() progress value. (IMHO, the fact that Godot doesn't display that data (or offer devs to choose to) on the first load is a bug, not a feature.)

Docs for `ResourceLoader.load_threaded_*`

Yellow boxes are for emphasis.

`ResourceLoader` method	Screenshot
`ResourceLoader.load_threaded_get()`
`ResourceLoader.load_threaded_get_status()`
`ResourceLoader.load_threaded_request()`

It's not possible without building the WASM file with Emscripten's Asyncify API (which really bumps up the size of the WASM output, which kinda defeats the purpose of the PR to load games faster on Web). ↩

AdriaandeJongh · 2026-02-23T17:15:35Z

High level note: the additional (and extremely lengthy) descriptions on these functions are IMO a good indication that these should be separate functions, even if they do a lot of similar things under the hood. Especially the state within a state with the download and load phases is particularly gross API design.

AR-DEV-1

LGTM. There are some issues here & there but they'll be fixed.

AThousandShips · 2026-02-24T15:24:14Z

I think there's a risk that the simplified API becomes more cumbersome to use and has more pitfalls that might be hard to notice, the necessity to use load_threaded and being unable to use @GDScript.load feels like a big constraint just to avoid new API

Based on what I can tell is the workflow in the original PR the way to do general loading would be largely unchanged for your basic actions and usage, with some additional API calls to handle invoking loading of async resources

While I haven't tested either implementation in practical use I feel that having just some dedicated fetch and install code running specifically on Web with minimally intrusive code and then just loading as usual for all practical load code with no additional adjustments feels less complex and intrusive than adding new API for this, especially if there's a requirement to adjust to use threaded loading operations universally for those cases. The threaded load operation would take a bit of overhead to use to replace any normal load calls etc. especially if one doesn't want to use that for other platforms

And I agree with the above that the need to clarify this is a good argument for dedicated API

dom-crazygames · 2026-02-24T16:10:25Z

This is looking good - is it possible to add an API for getting the amount of downloaded bytes? The big issue that this initiative solves is players churning during load times - lower load times are obviously the best case (less time for the players to get bored and leave!) but a crucial addition is visible progress. If the player can see what's going on/how long is left, particularly with a solid number that they can see ticking upwards, they are less likely to churn.

adamscott · 2026-02-26T13:05:53Z

This is looking good - is it possible to add an API for getting the amount of downloaded bytes? The big issue that this initiative solves is players churning during load times - lower load times are obviously the best case (less time for the players to get bored and leave!) but a crucial addition is visible progress. If the player can see what's going on/how long is left, particularly with a solid number that they can see ticking upwards, they are less likely to churn.

The current PR do tell the download progress using ResourceLoader.load_threaded_get_status(). There's no way to get how it relates to bytes downloaded though.

adamscott · 2026-02-26T15:27:55Z

Added @mihe and @syntaxerror247 as reviewers as I am touching the way platform code is shared.

mihe · 2026-02-26T17:03:14Z

+	const HashSet<String> &get_dependencies() const {
+		return dependencies;
+	}


This obviously isn't my area of expertise, but since this comment was left unresolved in #114690, this seems worth bringing up again.

Last I heard there was quite the fundamental problem with having GDScriptParser::get_dependencies return GDScript dependencies like this, since ResourceLoader requires (?) that resource dependencies form an acyclical graph, whereas GDScript can have cyclical dependencies.

Is there something about the changes to ResourceLoader in this PR, or the way dependencies is populated, that addresses that concern? If not, then this change would presumably end up reopening issues like #91726 again, unless they've been mitigated in some other way since then.

Given what you said about this whole PR falling apart without this change it seems like this would be worth clearing up.

I'd suggest to just add an additional get_cyclic_dependencies method for the time being. We can add a default impl that just returns get_dependencies and GDScript can only implement the later. Systems from which we explicitly know that they were written with cyclic dependencies in mind can utilize the information without the risk of breaking existing stuff.

That would add get_cyclic_dependencies() for all Resources, and IMHO, is redundant. Especially since Resources cannot have cyclic dependencies with exceptions.

Especially since Resources cannot have cyclic dependencies with exceptions.

I mean that's really up to the resource format loader at the end of the day, isn't it?

Alternatively we can also double check that every usage of get_dependencies does support cycles (maybe you did?).

As far as I'm aware there is no concrete issue with cycles just a general uncertainty on whether they are sufficiently supported (but if someone knows of some concrete reports please correct me). The previous PR was reverted due to a regression which does not stem from dependency cycles as far as I understood it.

HolonProduction · 2026-02-26T23:11:05Z

Let's leave concerns about support for cyclic references aside for a second.

I'm not convinced that tracking deps in the analyzer is a valid approach. Our other resource format loaders take special care not to actually load dependencies in get_dependencies. And previous regressions clearly show that current users are relying on this behaviour (If someone wants to play detective: 519fce9).

Best we can do with that restraint is doing stuff in the parser. (I don't think anyone is keen on adding more behaviour flags to the analyzer?)

Best we can do with that restraint is doing stuff in the parser. (I don't think anyone is keen on adding more behaviour flags to the analyzer?)

But the parser doesn't have any concept of files other than itself, if I remember well.

Fair, but where you place it codewise isn't really the issue. The analyzer would probably be fine. More like that I don't think we should load any external files for obtaining dependencies (from my understanding that caused the regression which lead to reversion the last time).

I don't think we would need to load them either. A soon as you would need to load an external file, you have your dependency and just skip ahead. Basically like island parsing but for the analyzer if that makes sense.

dsnopek

Thanks!

I haven't been able to test, because the PR won't compile right now, but I personally really like the high-level direction!

I think using the threaded resource API is the right choice, because it is meaningful on all platforms. I suspect most Godot apps with a web build aren't web-exclusive, but run on multiple platforms, and I worry if this was only exposed as a web-specific API, that no one would actually use it. Using the threaded resource loader for loading levels can provide a smoother experience on desktop, and now that will be true for the web too, just for different reasons.

I've only had a chance to skim the changes, and there's a lot here, so I'll probably need to take a 2nd pass later. :-) I'll do that when I can also test it

KoBeWi

I don't have setup for Web build, so I can't test this in action, but I did a code review.

Looks like this PR includes some refactoring and minor enhancements. Such changes shouldn't be part of big feature PRs, unless they are minor codestyle changes (like the gdscript.h change here), or necessary for the feature (although in that case, if they can exist separately, a separate PR is better). At the very least they should be in separate commits. Are the EdtiorExportPlatform changes necessary though? (i.e. extracting some methods/enums to new files)

The API changes (which are none xd) are fine; in the future we might want to expose the new OS methods, depending on feedback. It would allow some greater flexibility in installing files.

KoBeWi · 2026-03-12T18:46:17Z

These don't seem to be used anywhere?
And even then, adding SceneStringNames to existing identifiers should also involve replacing existing usages, so ideally it should be a separate PR.

I'll remove the definitions!

KoBeWi · 2026-03-13T17:14:43Z

+	using DebugFlags = EditorExportPlatformData::DebugFlags;
+	using ExportMessageType = EditorExportPlatformData::ExportMessageType;
+	using ExportMessage = EditorExportPlatformData::ExportMessage;


Like here .-.

KoBeWi · 2026-03-13T17:23:37Z

+	if (r_error != nullptr) { \
+		*r_error = (m_err); \
+	} \
+	((void)0)


What is the ((void)0) thing for?

It ensures that a ; is needed after the macro use.

The compiler will want the semi-column, but will essentially remove the ((void)0) code as this is a no-op.

KoBeWi · 2026-03-13T17:26:52Z

+	if (data->native_file.exists && !data->remap_file.exists) {
+		resource_file = &data->native_file;
+	} else if (data->remapped_file.exists) {
+		resource_file = &data->remapped_file;


Suggested change

if (data->native_file.exists && !data->remap_file.exists) {

resource_file = &data->native_file;

} else if (data->remapped_file.exists) {

resource_file = &data->remapped_file;

if (data->remapped_file) {

resource_file = &data->remapped_file;

} else if (data->native_file.exists) {

resource_file = &data->native_file;

data->remapped_file is always true, it's not a pointer.

KoBeWi · 2026-03-13T17:29:35Z

+		} break;
+
+		case ASYNC_INITIAL_INSTALL_MODE_ONLY_REQUIRED_RESOURCES: {
+			// TODO: Add AsyncPCK contents to the cache.


I assume this TODO is blocking for the PR?

I'll check if it's still needed.

Also: - Add bindings to `ResourceLoader::{load,load_threaded_*} methods`. - Add Web platform support to export a project using AsyncPCK. - Add Web platform backend code to support AsyncPCK `OS` calls.

adamscott added this to the 4.7 milestone Feb 23, 2026

adamscott requested review from clayjohn and dsnopek February 23, 2026 16:28

adamscott requested review from a team as code owners February 23, 2026 16:28

adamscott added the enhancement label Feb 23, 2026

adamscott requested review from a team as code owners February 23, 2026 16:28

adamscott added the platform:web label Feb 23, 2026

adamscott requested review from a team as code owners February 23, 2026 16:28

adamscott added topic:core topic:buildsystem topic:platforms topic:export documentation labels Feb 23, 2026

AThousandShips reviewed Feb 23, 2026

View reviewed changes

AThousandShips removed the documentation label Feb 23, 2026

AR-DEV-1 reviewed Feb 23, 2026

View reviewed changes

Comment thread platform/web/js/libs/library_godot_os.js

Comment thread platform/web/js/libs/library_godot_os.js

AR-DEV-1 reviewed Feb 24, 2026

View reviewed changes

Comment thread platform/android/export/export_plugin.cpp Outdated

Comment thread platform/android/export/export_plugin.cpp Outdated

adamscott force-pushed the async-resource-loader-mk8 branch from e20e318 to f426f49 Compare February 26, 2026 15:21

adamscott requested review from mihe and syntaxerror247 February 26, 2026 15:27

mihe reviewed Feb 26, 2026

View reviewed changes

HolonProduction reviewed Feb 26, 2026

View reviewed changes

HolonProduction mentioned this pull request Feb 26, 2026

GDScript: Fix get_dependencies by deferring resolution #113791

Closed

adamscott mentioned this pull request Mar 2, 2026

Add AsyncPCK support (+ Web implementation) #114690

Open

dsnopek reviewed Mar 2, 2026

View reviewed changes

KoBeWi reviewed Mar 13, 2026

View reviewed changes

adamscott force-pushed the async-resource-loader-mk8 branch 2 times, most recently from 2a795bf to bf811fd Compare April 1, 2026 15:12

Add AsyncPCK support

cce6bd2

Also: - Add bindings to `ResourceLoader::{load,load_threaded_*} methods`. - Add Web platform support to export a project using AsyncPCK. - Add Web platform backend code to support AsyncPCK `OS` calls.

adamscott force-pushed the async-resource-loader-mk8 branch from bf811fd to cce6bd2 Compare April 9, 2026 13:02

adamscott added 3 commits April 9, 2026 09:36

Apply dsnopek suggestions

ba67dc0

Apply some of KoBeWi's suggestions

ee98422

Remove incompatible Closure compiler

e7af7f1

Uh oh!

Conversation

adamscott commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes from the previous PR

Important caveats when using ResourceLoader.load_threaded_* instead of a dedicated AsyncPCK API

Docs for ResourceLoader.load_threaded_*

Footnotes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdriaandeJongh commented Feb 23, 2026

Uh oh!

AR-DEV-1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AThousandShips commented Feb 24, 2026

Uh oh!

dom-crazygames commented Feb 24, 2026

Uh oh!

adamscott commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamscott commented Feb 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dsnopek left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KoBeWi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamscott commented Feb 23, 2026 •

edited

Loading

Important caveats when using `ResourceLoader.load_threaded_*` instead of a dedicated AsyncPCK API

Docs for `ResourceLoader.load_threaded_*`

adamscott commented Feb 26, 2026 •

edited

Loading

dsnopek left a comment •

edited

Loading