Serialize and deserialize texture userData in glTF sampler.

[repalash]

Description

Hello,

The userData in three.js Texture is not serialized into glTF. This PR adds support for that.

It also changes GLTFLoader so that userData from the Sampler is copied to texture.userData, instead of Image (as introduced in #28226)

Some background -
in glTF, textures are broken up into Texture, Sampler, TextureInfo and Image, all of which support storing extras. See - donmccurdy/glTF-Transform#645 (comment)

For three.js, its ideal that any file that is imported and exported across three.js, gltf-transform etc, retain as many properties as possible without changing or transferring them across primitives.
Based on how three.js objects are structured, and the above explanation by donmccurdy, it makes sense to map three.js Source to glTF Image, and three.js Texture to glTF Sampler or Texture. Since the threejs API does not prevent changing/sharing the Source instance attached to a Texture(which is the case in glTF I think), I think it makes sense to map it to Sampler instead of Texture.

Ofc, with this, the extras on source and texture def will never be read from glTF in three and lost when exported with GLTFExporter. This can be handled by reading them into something like texture.userData.imageExtras and texture.userData.textureExtras and used when exporting.

This contribution is funded by Threepipe

[donmccurdy]

... extras on source and texture def will never be read from glTF in three and lost when exported with GLTFExporter.

Hm, I am worried about this change — I expect extras/userData on the GLTF.Image or GLTF.Texture are much more common than extras on the GLTF.Sampler. What use case do you have in mind for extras on glTF Samplers?

My leaning is that because three.js does not have a data model that matches glTF 1:1, trying to guarantee that a round trip...

.glb → GLTFLoader → three.js → GLTFExporter → .glb

... keeps extras on objects that don't have clear counterparts in three.js might not be feasible to support. GLTFLoader could perhaps merge .extras from GLTF.Image, GLTF.Texture, and GLTF.Sampler objects all into THREE.Texture#userData, so none are lost on import. But on export THREE.Texture#userData would probably need to be written only once, probably on the GLTF.Texture extras.

[repalash]

Thanks for looking into it.

I expect extras/userData on the GLTF.Image or GLTF.Texture are much more common than extras on the GLTF.Sampler

Yes having extras on GLTF.Texture is the most ideal. I didn't do that here since it is not supported in gltf-transform (and alteast in my mind, mapping sampler to threejs texture makes more sense)

At the moment only extras from GLTF.Image is read into THREE.Texture#userData in the GLTFLoader, and not from GLTF.Texture or GLTF.Sampler.

Lets take an example -
In a scene we have 1 THREE.Texture with userData. Now in GLTFExporter, we can write it to textureDef(GLTF.Texture), baseColorMapDef(GLTF.TextureInfo), samplerDef(GLTF.Sampler), or imageDef(GLTF.Image).

To keep it consistent with the current GLTFLoader, it should go into imageDef but thats not correct, as 1 Texture could be mapped to the same HTMLImageElement, eg -

const img = new Image();
        img.crossOrigin = 'anonymous';
        img.src = '...'
        img.onload = () => {
            console.log('Image loaded');
            // Create 5 textures using the same image
            const textures = [];
            for (let i = 0; i < 5; i++) {
                const texture = new THREE.Texture(img);
                texture.needsUpdate = true;
	        texture.userData.myTextureID = i+1
                textures.push(texture);
                console.log(`Texture ${i + 1} created`);
            }
            for (let i = 0; i < 5; i++) {
                // create material, mesh
                material.emissive = textures[i];
                material.map = textures[i];
                scene1.add(mesh);
            }
}

The plan is to save it as gltf, do some processing, and read back in three.js while keeping the myTextureID.
Exporting this saves 5 Texture, 1 Image, 5 Samplers, 10 TextureInfo.
So saving in Image and TextureInfo is not possible.
If we save in Textures or Samplers, its erased by glTF-transform. Hence these 2 PRs. (will leave some more context in gltf-transform PR)

Saving in Texture vs Sampler -
From gltf-transform - Textures in glTF Transform are a combination of glTF's texture and image properties, and should be unique within a document, such that no other texture contains the same getImage() data

So GLTF.Texture maps 1:1 to a GLTF.Image.

Which is not true in three.js, as multiple HTMLImage or THREE.Source can be assigned to the same THREE.Texture so it makes more sense to map it to samplers.

... keeps extras on objects that don't have clear counterparts in three.js might not be feasible to support.

I would like to understand the issues if we try to map

GLTF.Sampler -> THREE.Texture
GLTF.Texture + GLTF.Image -> THREE.Source

Apologies if i repeated somethings, got confused in the middle because of all the terminology.