How to help for developing new decompressor?

[mlouielu]

Hi there,

I want to help on #367 and want to get some enlightenment to point me out how to test on it. I had build and download the test images from raw.pixls.us, what will be the recommend process of testing?

For example, how can I determine the decompressor is successfully implement? Are there any viewer or way to test the result?

Thanks!

[LebedevRI]

I'm not sure what the specific question is.
You know that it is successfully implemented when,
after compiling darktable with updated rawspeed submodule,
the images load correctly in darktable :)

See also https://github.com/darktable-org/rawspeed/blob/develop/docs/IntegrationTesting.rst.
And:

rawspeed/build-Clang17-releaseWithAsserts$ ./src/utilities/rstest/rstest 
usage: ./src/utilities/rstest/rstest
  [-h] print this help
  [-c] for each file: decode, compute hash and store it.
       If hash exists, it does not recompute it, unless option -f is set!
  [-f] if -c is set, then it will final the existing hashes.
       If -c is not set, and the hash does not exist, then just decode,
       but do not write the hash!
  [-d] store decoded image as PPM
  <FILE[S]> the file[s] to work on.

  With no options given, each raw with an accompanying hash will be decoded
  and compared (unless option -f is set!) to the existing hash. A summary of
  all errors/failed hash comparisons will be reported at the end.

  Suggested workflow for easy regression testing:
    1. remove all .hash files and build 'trusted' version of this program
    2. run with option '-c' -> creates .hash for all supported files
    3. build new version to test for regressions
    4. run with no option   -> checks files with existing .hash
  If the second run shows no errors, you have no regressions,
  otherwise, the diff between hashes is appended to rstest.log

[mlouielu]

Thanks for the swift reply!

I'm not sure what the specific question is. You know that it is successfully implemented when, after compiling darktable with updated rawspeed submodule, the images load correctly in darktable :)

I would like to know more about the workflow, do you directly working in the darktable rawspeed-submodule and rebuild the darktable after modifying the rawspeed?

Since it will open up a relative heavy GUI, are there any way to prevent this, or there is a lightweight viewer that also depend on rawspeed?

Also, any suggestion on attaching the GDB for debugging/tracing purpose? Is it still need to attach to rebuilt darktable?

Thanks!

[LebedevRI]

I'm not sure what the specific question is. You know that it is successfully implemented when, after compiling darktable with updated rawspeed submodule, the images load correctly in darktable :)

I would like to know more about the workflow, do you directly working in the darktable rawspeed-submodule and rebuild the darktable after modifying the rawspeed?

Generally almost never, but everyones's mileage may differ.

Again, it depends on what you want to achieve.
You could just build the rawspeed as a standalone project, and,
as noted in the blurb i posted, use $ ./src/utilities/rstest/rstest -d to produce a PPM/PFM of the decoded image.

[mlouielu]

Again, it depends on what you want to achieve. You could just build the rawspeed as a standalone project, and, as noted in the blurb i posted, use $ ./src/utilities/rstest/rstest -d to produce a PPM/PFM of the decoded image.

Thanks for pointing me out! I tried rstest and stored the PPM, put the PPM back into darktable and adjust the exposure to get the image back! (btw, are there any viewer that is not darktable support adjusting exposure on the fly?)

So if the new decompressor works will, the above workflow should work, right?

[LebedevRI]

Thanks for pointing me out! I tried rstest and stored the PPM, put the PPM back into darktable and adjust the exposure to get the image back!

Usually it's not quite An Image, since usually a demosaicing is needed,
which doesn't happen for PPM's, but it's enough to get a rough view, i guess.

(btw, are there any viewer that is not darktable support adjusting exposure on the fly?)

I can't parse the question, sorry.

So if the new decompressor works will, the above workflow should work, right?

I guess so?

[LebedevRI]

@mlouielu thank you for taking a look at that!

[kmilos]

I suppose you can open the PGM in GIMP and hit Colors -> Auto -> Stretch Contrast... (or any other tool having similar functionality, like Adjustments -> Auto-Level in paint.NET on Windows etc.)

It'll be still CFA mosaiced and grayscale, but should at least give you an idea if there is a "sensible" image behind it or just garbage.

[kmilos]

Or, you could even temporarily just hack the rstest code to write out the PGM as MSB aligned and there's no need for external programs for exposure compensation then.

[kmilos]

You can also avoid any code changes and use pamdepth 65535 to shift the PGM up to 16b, then use pnmhisteq/pnmnorm to enhance, and then pamtopng/pnmtopng (or any other output format to view in any sw): https://netpbm.sourceforge.net/doc/directory.html

I'm sure you can do the equivalent w/ ImageMagick/GraphicsMagick as well on one line...