One-to-many asset processing

[andriyDev]

Objective

• Allow processors to produce multiple files instead of just one.

Solution

• Instead of passing a &mut Writer to Process, instead we pass a WriterContext. Users then just call write_single or write_multiple to decide whether they want a single file or multiple files. Once users are done writing, they call SingleWriter::finish or MultipleWriter::finish.
• I also added a new asset action - Decomposed. This makes it more obvious to callers that they are querying for the wrong asset path.

This means if you have an asset path like path/to/my_asset.json, and the Process produces files "part1.thing", "part2.thing", and "dir/mesh.gltf", these assets will be accessible at path/to/my_asset.json/part1.thing, path/to/my_asset.json/part2.thing, and path/to/my_asset.json/dir/mesh.gltf. In essence, the asset in the unprocessed directory becomes a folder in the processed directory.

Testing

• The asset_processing example works and has been updated to include a "multiple" processor.
• Updated unit tests to account for "multiple" files.
• Created new tests too.

[JMS55]

Very exciting! Left some feedback

• I really don't like the single vs multi naming. I think we need better naming.
• As someone not very familiar with the asset system, I don't quite get how the multi files get mapped to asset server concepts. Sub assets? Or?
• Do we support asset savers besides filesystems? How does this work with those?

[JMS55]

write_multiple() in the loop, for writing a single item, feels kind of weird to me. Can we call it something else?

write_part, write_subasset, write_new_asset, ?

[andriyDev]

Fair point! I tried renaming write_single and write_multiple to write_full and write_partial respectively. The naming here really sucks. We shouldn't use subasset because that's already taken and it has a completely different meaning here.

I want to make sure that the "full" vs "partial" are like antonyms (like single vs multiple). Just to give the intuition that these can't be mixed.

[LikeLakers2]

I'm a little late but - what about write and write_many? Similar to how some rust standard library types have X and X_many - like get and get_many on collection types.

[andriyDev]

I think that would have the same problem as write_multiple: calling write_many in a loop feels wrong.

[andriyDev]

To respond to your questions:

• Good point! Discussed in a comment.
• Multi-files are just treated as individual assets. There is no requirement that subassets map to some multi-file. If the process wants to throw out the bytes of the original asset and write 100 copies of the Bee Movie script, that's totally up to it!
• The Process trait has no inherent relationship to AssetSaver, both before and after this PR. The AssetSaver is used by the particular Process implementation LoadTransformAndSave, which makes it slightly more convenient to write Process implementations. We could figure out a similar convenience for one-to-many, but I think we should leave that for another PR, since that seems a little tricky to get right, and I don't think we need it for a glTF processor.

[andriyDev]

Fair point! I tried renaming write_single and write_multiple to write_full and write_partial respectively. The naming here really sucks. We shouldn't use subasset because that's already taken and it has a completely different meaning here.

I want to make sure that the "full" vs "partial" are like antonyms (like single vs multiple). Just to give the intuition that these can't be mixed.

[JMS55]

In essence, the asset in the unprocessed directory becomes a folder in the processed directory.

How does this work for non-filesystem-based asset saver things?

I tried renaming write_single and write_multiple to write_full and write_partial respectively

After doing some more thinking, I think it would be better if WriteContext implemented Write directly to keep the API the same as mainline for single-asset mode, and then make some kind of multi_asset_writer(self) function on WriteContext that you can use to get a multi-asset writer.

What do you think?

[andriyDev]

How does this work for non-filesystem-based asset saver things?

Asset sources currently have to act like a filesystem. They need to be able to have assets as well as directories. It's also important to note this only matters for the processed reader - not for the unprocessed reader. In practice, I expect the processed reader/writer will always be a local file system.

I think it would be better if WriteContext implemented Write directly

The problem with that is we still need to finalize the writer anyway (since we can no longer just return the meta, since there may be more than one), so the API will necessarily change. I'd prefer to make the writer explicit in that case rather than only fetching the writer on write, which is what would have to happen to support the API you're proposing.

[JMS55]

The problem with that is we still need to finalize the writer anyway (since we can no longer just return the meta, since there may be more than one), so the API will necessarily change. I'd prefer to make the writer explicit in that case rather than only fetching the writer on write, which is what would have to happen to support the API you're proposing.

Can't we just make this happen on writer drop?

[andriyDev]

No because we need the process to give us loader settings. That's really what finalization is for.

[JMS55]

I'm approving this for now, because I think having the infrastructure and functionality is more important than blocking on the user-facing API, but I really don't like the write_single / write_partial API.

Imo I would much rather either:

• User consumes the initial writer object, gets back a strongly-typed AssetWriter or MultiAssetWriter or something
• User only ever calls write_asset(), Bevy transparently detects when a second call is made and moves the first asset to a folder.

[andriyDev]

User consumes the initial writer object, gets back a strongly-typed AssetWriter or MultiAssetWriter or something

We almost have this. write_single consumes the context and gives you back a writer (which you eventually need to call finish on). For the partial case, it would just be an extra call to consume the context to give you back a MultiAssetWriter which you then need to give a path anyway to start a new write. So I avoid that and just say that the write context gives you the writer directly, at the cost of making it possible to call write_partial followed by write_single.

User only ever calls write_asset(), Bevy transparently detects when a second call is made and moves the first asset to a folder.

This would mean we can't parallelize writes, since you need the first write to be complete before you can move it to the folder. So now all of your writes need to happen one at a time, which is sad. I also am not a fan of how implicit it is, especially if your process "accidentally" writes only a single asset. Like imagine a glTF that we want to split into multiple pieces: If there's only a single mesh in the glTF, it may only write one asset, and now whether you need to access it as my.gltf or my.gltf/Mesh0.gltf is dependent on the contents, which can be confusing!

[JMS55]

Very fair, glad to see that you've thought this out much more than I have 😅

In that case my only remaining gripe is that write_partial feels like a poor name for it since the assets you write are unrelated to each other besides the fact that they come from the same processed asset, iiuc.

[NthTensor]

Looks good to me. I've been wanting this functionality for ages.