Roadmap for supporting wide-gamut color workflows

[donmccurdy]

Related:

• Roadmap for a color-managed workflow in three.js #23614

Roadmap and discussion for supporting for wide-gamut color spaces in three.js. Many displays now support Display P3 (+50% larger gamut volume than sRGB), and web browsers are beginning to extend that support to WebGL contexts. Other color wide-gamut color spaces exist (Rec. 2020, ACEScg) and may be relevant to our internal rendering pipeline, but probably will not be widely available as display color spaces in WebGL for a very long time.

Broadly the goals here are (1) to identify useful and correct workflows for users, and (2) to make any necessary changes in the three.js rendering pipeline. I suspect that (1) may be the harder part of the problem.

Decisions

• Working color space: In addition to "Linear sRGB", which working color space(s) should be supported?
  • Options include Linear P3, Linear Rec. 2020, and ACEScg. We may want to consider providing a mechanism to register new color spaces with three.js, rather than including conversions to/from each combination of color spaces out of the box. Each will also have different luminance calculations, different tone mapping, ...
• Tone mapping: What tone mapping should be used for wide-gamut displays?
• Unlit vs Lit: Support for unlit workflows (e.g. a photo editing app) will be easier than lit workflows, which require changes to tone mapping and perhaps PBR equations. Should we consider shipping unlit support first?

Workflows

I'm skeptical than wide-gamut features can be enabled automatically; it will require significant care from users to get the benefits of these displays, and should be opt-in. I'm not sure which 3D modeling tools support the P3 gamut at all. Files in standard formats like glTF are almost exclusively using sRGB today. Users will need to load textures manually and assign texture.colorSpace correctly. We cannot infer texture color space safely without user input of some kind.

Tasks

Task numbers begin at 2.1, following up from #23614.

1. (2.1) Support changing ColorManagement.workingColorSpace
   • Allow one or both of "acescg" or "p3-linear", in addition to current "srgb-linear"
2. (2.2) Support THREE.Color getters/setters in new working color space(s)
3. (2.3) Support renderer.outputColorSpace = THREE.DisplayP3ColorSpace
4. (2.4) Model loaders must delegate management to THREE.ColorManagement, not convert to Linear-sRGB explicitly
   • GLTFLoader: Specify color space of inputs #26534
5. (2.5) Texture loaders should identify and tag wide-gamut textures where possible (OpenEXR, KTX2, ...?)
6. (2.6) Use gl.unpackColorSpace to unpack "display-p3" PNG, JPEG, and WebP images into the working color space
7. (2.7) Provide tone-mapping implementations from wide-gamut working color space(s) to "srgb" and "display-p3" output color spaces

[donmccurdy]

@WestLangley – continuing from our technical discussion, I've added more TODOs in the task list above. Please let me know if you feel anything major is missing or doesn't belong. :)

[donmccurdy]

PR for 2.1, 2.2, 2.3, and 2.6 —

• WebGLRenderer: Support "Linear Display P3" working color space and "Display P3" unlit rendering #26644

---

We should work toward offering a tone map adapted for wide-gamut and HDR displays, and with a more neutral default appearance than ACES Filmic. Blender 4.0 will be moving to AgX as the default, and I believe that would be a good choice for us as well. Related details:

• Implementation details: https://projects.blender.org/blender/blender/pulls/111099
• Screenshots: https://projects.blender.org/blender/blender/pulls/106355

Blender 4.0 alpha builds are available with AgX support now. I've prototyped an implementation in three.js. I am not sure about how to add support for other working and display color spaces yet. The implementation can also be optimized by removing the 1D LUT.

• https://glitch.com/edit/#!/three-sb2383?path=README.md%3A1%3A0

/cc @bhouston

[bhouston]

@donmccurdy great work! I also strongly agree that we should aim to provide AgX as an alternative tone mapping and maybe if there is enough acceptance, we can switch it to the default.

[donmccurdy]

Updates:

• When drawing to a render target, we currently ignore the color space of the render target (WebGL will apply the sRGB OETF for us if necessary) and override outputColorSpace with LinearSRGBColorSpace. I believe this should semantically be the working color space, instead, but I'm not sure it affects the shader either way right now.
• Creating consistent color checker images in a variety of color spaces (sRGB, Linear sRGB, Display P3, Linear P3, ACEScg) has been challenging. OpenColorIO is not embedding the ICC Profiles we need for the web browser in PNG or JPEG, and so I'm waiting for a new alpha release of KTX-Software to get .ktx2 versions.
• In early tests with lit rendering in Linear Display P3, the most striking difference appears related to AO? If I light two scenes with a punctual light, they're very similar. If I light two scenes with an ambient light, they're similar only after disabling AO maps. The sRGB scene has much stronger AO. I've checked that the AO map (non-color data) is color managed correctly. Unsure why this is.
• It might be useful to have a utility function on KTX2Loader and EXRLoader (and anything else that supports RGBA8/16/32 uncompressed data) to assign or convert color spaces. Like .assignColorSpace( SRGBColorSpace ) and .convertColorSpace( LinearDisplayP3ColorSpace ). Assign would override our beliefs about the existing color space of the data. Convert would convert from the existing color space to the given one. I wouldn't necessarily advise users to convert color spaces at runtime in production, but it will make development much easier.

[donmccurdy]

@WestLangley from your comment in #26666 (comment) –

I don't think we should "honor" the color space of a render target when rendering to it. The color space is relevant only when reading from the render target.

If we're reading from the render target in sRGB we do need to get that data in correctly, right? We can safely "ignore" the color space now because writing Linear-sRGB to the sRGB target will lead WebGL to (automatically?) apply the sRGB OETF for us, and we don't want our fragment shader doing that.

I think we may want to go one step further here. If we are writing to a uint8 render target having Display P3 color space, from working color space Linear sRGB, our fragment shader should do the gamut mapping and let WebGL do the OETF. Writing non-linear data to a f16 or f32 render target would not be supported, because WebGL doesn't support it.

[WestLangley]

I think what you said is correct, and I don't think it is contrary to my statement. 🤔

[kitaedesigns]

Did this AgX feature request get merged into the latest version of threejs? I couldn't tell.

PR for 2.1, 2.2, 2.3, and 2.6 —

• WebGLRenderer: Support "Linear Display P3" working color space and "Display P3" unlit rendering #26644

We should work toward offering a tone map adapted for wide-gamut and HDR displays, and with a more neutral default appearance than ACES Filmic. Blender 4.0 will be moving to AgX as the default, and I believe that would be a good choice for us as well. Related details:

• Implementation details: https://projects.blender.org/blender/blender/pulls/111099
• Screenshots: https://projects.blender.org/blender/blender/pulls/106355

Blender 4.0 alpha builds are available with AgX support now. I've prototyped an implementation in three.js. I am not sure about how to add support for other working and display color spaces yet. The implementation can also be optimized by removing the 1D LUT.

• https://glitch.com/edit/#!/three-sb2383?path=README.md%3A1%3A0

/cc @bhouston

[donmccurdy]

It hasn't been — I don't think anyone is working on it currently. Additional discussion here:

• Pluggable Tone Mapping Operators (their own class that injects GLSL) #26661 (comment)

[kitaedesigns]

Gotcha thanks Don, I'm not sure your Glitch link is working anymore. With Blender 4.0 any assets users create in Blender and show in three.js look quite different. Your LUT approach although perhaps not totally accurate seems quite flexible and I think many 3D designers like myself are very used to LUTs due to their popularity in game engines like Unreal and Unity, would love to test with that LUT file, I assume it's a .CUBE or .png?

I am currently using pmndrs PostProcessing library and @vanruesc mentioned it would be more efficient to have natively in three.js first which I understand, but maybe I can just use your LUT in his library and see how close I can match the colors between Blender 4 and Three.js.

[donmccurdy]

The LUT is part of the prototyping process here – it isn't really something we want in the final implementation, and doesn't represent the complete tone mapper. A better realtime implementation (like Filament's) would replace the LUT with a fitted function.

I think AgX is something we can and should support independently of pursuing support for wide-gamut color, so let's continue that discussion in a new thread:

• Support AgX tone mapping #27362