Plan to support HDR images (.hdr, .exr)

[wasimabbas-arm]

ASTC encoder supports both .exr and .hdr formats. It will be nice that ktx can load these and then forward to encoders like ASTC encoder. Possibly true for basisu as well.

[MarkCallow]

ktx doesn't pass files to the ASTC encoder though so the fact the encoder has a file reader for .hdr should not factor into a decision to support it or not.

The author of the Radiance (.hdr) format (Greg Ward) points out the following shortcomings in this document:

"Firstly, the dynamic range is much more than anyone else could ever utilize as a color representation. ... Another problem is that any RGB representation restricted to a positive range, such as RGBE and Pixar’s Log format, cannot cover the visible gamut using any set of “real” primaries. ... Finally, the distribution of error is not perceptually uniform with this encoding. In particular, step sizes may become visible in saturated blue and magenta regions where the green mantissa drops below 20."

Currently ktx supports EXR via TinyEXR for creating floating point textures but reportedly "EXR supports too many compression formats so it's not interchangeable in practice. ... TinyEXR can only read a fraction of the .EXR files used in the wild, because many .EXR images utilize compression methods it doesn't support."

It seems both formats have limitations.

One question for @wasimabbas-arm, how many bits of precision does ASTC HDR support? If greater than 8 then .hdr will not allow achieving the best results for the format.

@lexaknyazev would love to hear your opinion as the primary author of the ktx tool suite specification on whether the suite should support .hdr input.

[solidpixel]

How many bits of precision does ASTC HDR support?

How many bits of effective precision a lossy format supports is a bit of a hard question to answer.

Notionally ASTC decompresses to an fp16 intermediate (or RGB9e5 if using decode_mode), BUT the actual precision in the compressed data will be lower than that due to the encoding. The actual dynamic range, and precision density within that used range, depends on the data you are compressing and will vary on a per block basis.

If greater than 8 then .hdr will not allow achieving the best results for the format.

Isn't that a judgement call the user is qualified to make? It's their content pipeline, and they are already making other such judgement calls that impact image quality such as compression level.

Firstly, the dynamic range is much more than anyone else could ever utilize as a color representation.

Many uses of float textures in graphics are not storing color.

Another problem is that any RGB representation restricted to a positive range, such as RGBE and Pixar’s Log format, cannot cover the visible gamut using any set of “real” primaries.

ASTC only supports encoding positive values in either case. (BC6H can do signed).

The only possible interpretation of float data in the graphics API is linear sampling. There is no built-in OETF for sampling a non-linear gamut or a specific set of primaries; the user would have to provide that in shader code if they cared. (The only non-linear OETF in the API is sRGB, which is explicitly LDR).

Finally, the distribution of error is not perceptually uniform with this encoding. In particular, step sizes may become visible in saturated blue and magenta regions where the green mantissa drops below 20."

As above, this assumes in the input data is color. For PBR it often isn't.

My personal opinion on this one is that the choice of image formats is highly likely to be determined by the asset creation pipeline before encoding for GPU consumption happens, and changing that is often not trivial. KTX should just give users tools that can handle as much of the input content space as possible, and let them make the judgement call on what is an acceptable image quality for their use case.

[wasimabbas-arm]

If we use stb_image is it possible to compile it so that only the .hdr codec is included?

This might be possible, you can compile out some decoders but

I'd hate to waste memory by having all the other codecs as we have our own jpeg and png.

We are already paying for this. stb_image is shipped in the ASTC encoder. We can re-use it from there.

I don't think rolling our own is efficient. stb_image is heavily tested. What deficiencies it might have we can overcome for example we can still read the .hdr file header and parse it separately as well.

Also I don't think its just .hdr that should be supported but I am also looking to support .exr files (another HDR format)

[MarkCallow]

We are already paying for this. stb_image is shipped in the ASTC encoder.

Is this and TinyEXR compiled into the ASTC encoder library we're building? Even if they are, since we don't call the part of the encoder that deals with reading input files would they be linked to the application or shared libktx from the static ASTC library?

Looking at the link @lexaknyazev provided, it looks like the header can be very long, having programs interspersed with exposure and colorcorr and the other header fields. There can be multiple exposure and colorcorr fields and they are cumulative. So I don't think parsing for this information separately will be that easy. If stb_image is not parsing and applying these values it is likely better to look for a different reader.

I am also looking to support .exr files (another HDR format)

.exr is already supported in ktx create via our imageio library and the TinyEXR that is in the ASTC encoder source.

[lexaknyazev]

having programs interspersed with exposure and colorcorr and the other header fields

What do you mean? The file format itself contains just 7 header fields (some of which could appear more than one time).

[MarkCallow]

What do you mean?

From the reference you linked:

"The information header begins with the usual "#?RADIANCE" identifier,
followed by one or more lines containing the programs used to produce the
picture. These commands may be interspersed with variables describing
relevant information such as the view, exposure, color correction, and so on."

Perhaps I am misinterpreting "one or more lines containing the programs used to produce the picture" and it just means the names of the programs used.

[lexaknyazev]

Yes, those are akin to KTXwriterScParams.

[arpu]

what i use now are gainmaps and Ultra HDR https://github.com/MONOGRID/gainmap-js there are some plans to include this in jpg and imagemagic ImageMagick/ImageMagick#6377 with https://github.com/google/libultrahdr

[MarkCallow]

We should look at UltraHDR (JPEG-R). I'm guessing this is the format read by the libultrahdr link provided by @arpu. I don't know if this is being proposed to ISO or if it is just an Adobe, Android thing.

[arpu]

looks like basisu has now UASTC HDR support https://github.com/BinomialLLC/basis_universal/releases/tag/v1_50_0_2

[MarkCallow]

We have a WIP PR to add UASTC HDR to the KTX specification. We are drawing up an RFP to find a contractor to add support to KTX-Software. We will also add support for encoding to ASTC HDR formats once the HDR input format support is in place.