Dynamic range problem.

[VAGrishin]

Hi!

For the scenes of a large brightness dynamic range, the camera sets the maximum brightness of the image to 255, so the brightness of the dark areas is equal to zero. Is it possible to keep the brightness of dark areas in the image?
Bright image areas are out of interest.
The option "-drc high" does not improve the situation.

It is also possible that some kind of noise reduction algorithm is used in the camera.
In any case, irreversible image corruption occurs.
Is there any way to avoid this?

Camera Module v2 is used.

Thanks in advance!

[6by9]

Dynamic range compression (drc) is a post-processing step. It won't help in this situation.

You want exposure_compensation set to a positive value (units of 1/6 of a stop) to adjust the AE target point.

[VAGrishin]

We can try to do this, but I'm not sure if this will help us.
Now, real shooting is being done with the highest possible manual exposure.
As a result, we have that the brightness of the lamps in the field of view is 255, and the brightness of the terrain is 0.

I believe that it is necessary to disable the normalization of brightness and any noise reduction.
But I don’t know how to do it.

[6by9]

If you're using manual exposure and gain then everything is on you to sort out. The camera sets the exposure and gain to the value you set it to.
There is no normalisation of the brightness in manual mode - it's basically linear processing based on the input from the sensor (which is also linear with light level).
The processing pipe produces 8bit data, therefore the values are inherently clipped to 0-255. Saturate the output and you will get odd colours on the edges as the 3 channels saturate at slightly different places.

Pictures speak a thousand words. Show an example of what you're trying to do, preferably with the metadata for all the capture parameters.

[VAGrishin]

The processing pipe produces 8bit data, therefore the values are inherently clipped to 0-255.

This is the main problem.
During the clipping, information about areas of the scene with a low level of brightness is lost.
Saturation of channels and strange colors do not bother us at all.
For processing, only one channel is used. It is red channel.
The reason is that the longer the wavelength, the less the dissipation in the atmosphere.

I asked a colleague to pick up a night shot.
Can I attach large files (24 MB) to a message?
Or is it better to put it on any server and give a link?

[VAGrishin]

A colleague sent an image file.
After archiving, the file size turned out to be small, about 192 KB.
The archive contains the corresponding file with the parameters of shooting.
The command line is given there:
command=raspistill -e bmp -ss 8000000 -awbg 8,8 -drc off -o /mnt/RasPi4_1831203849.bmp
RasPi4.zip

[6by9]

If you are using raspistill. why are you raising the issue in the Picamera library which is a Python binding for the Pi camera? raspistill is part of userland - https://github.com/raspberrypi/userland/issues

You've fixed exposure time, but done nothing for analogue gain or digital gain. AE/AGC will therefore be setting those based on the expected overall exposure of the frame. Add -ag 8.0 and -dg 2.0 and you'll be boosting the signal further. Expect the noise to go up accordingly (digital gain moreso than analogue).

Exposure time can go up to 10seconds on IMX219, which gives you another 25% light.

Pushing AWB digital gains to x8 for red and blue will give very odd colour casts when green is still at x1.0 (fixed). Actually without -awb off I suspect it is doing nothing.

Capturing JPEGs also drops a load of more useful metadata in the EXIF. BMPs also include a further image conversion from YUV to RGB.

If you're really wanting to mess with images in this way then you may be better off dropping back to the raw Bayer data. See https://github.com/6by9/raspiraw. Be prepared to take full responsibility for all settings at that point.

[VAGrishin]

I am sorry. Now Python is actively used under Linux along with the Bash and in my understanding the Python and Linux were not divided. My colleague also works with the Picamera library too.
In addition, the question relates more to the camera functioning.

As for analog gain, as far as I know, we can only try to influence it indirectly, for example, through ISO. The final value is set by the camera processor. However, usage -ISO 800 improved the situation.

The best thing, of course, would be to totally eliminate all the processing and auto-tuning that the processor does. We are used to work with special developed cameras that do exactly what we need. Namely, they give out a signal in direct proportion to the number of photons. And nothing more. We do the rest of the processing ourselves. But the development and manufacture of such cameras are very expensive. For example, only one special developed lens can cost $20,000 or more. But there is no such money. So there was an idea to take commercial off-the-shelf (COTS) device and use it.

Thank you for the recommendations. We will work in the directions you indicate.

[6by9]

Direct control of analogue and digital gains was added to raspistill in Oct 2017. I develop the firmware, and PiCamera has dropped behind a bit as I know waveform80 has other priorities at present, and it hasn't been added to PiCamera.

Denoise operations can be disabled. Bayer denoise needs #571.
The YUV denoise stages can be disabled via https://picamera.readthedocs.io/en/latest/api_camera.html#picamera.PiCamera.image_denoise and https://picamera.readthedocs.io/en/latest/api_camera.html#picamera.PiCamera.video_denoise

[VAGrishin]

The Raspi4 camera did not respond to the -ag 8.0 and -dg 2.0 options. It looks like the old version of raspistill is there right now. Unfortunately, the camera is now located about 1200 km from us.
And the remote control over the camera by the Internet is lost. It is possible to reboot and change the command line only.

Physically, the camera will be available in two months. The camera itself was developed by my colleague Maslov I.A. The appearance of the camera can be seen in the attached figures.
These figures were published in http://iki.cosmos.ru/books/2018nazirov.pdf
In the picture you can see three cameras, but only one of them is working now. It produces about 1 GB of information per day.

These cameras were designed to analyze the processes of absorption and scattering of light in the atmosphere. Unfortunately, even for daytime shots, the camera processor irreversibly spoils the image.

Maslov I.A. will experiment with another camera for now.

Do I understand correctly that disabling noise reduction is only possible when using the Python Library?

[6by9]

And the remote control over the camera by the Internet is lost. It is possible to reboot and change the command line only.

So either update the Pi via apt, or rebuild userland for yourself to get the latest version.
You may wish to take a backup copy of /opt/vc/ before rebuilding userland to be able to restore the older version, although apt install --reinstall raspberrypi-bootloader will probably restore it for you.

These cameras were designed to analyze the processes of absorption and scattering of light in the atmosphere. Unfortunately, even for daytime shots, the camera processor irreversibly spoils the image.

Sorry, it's your job to ensure the camera module does what you want. The standard camera stack is written for taking standard webcam-type images.
It sounds like you need to switch to working on the raw Bayer output, and even then it's up to you to ensure that the data collected is of use, and that appropriate settings are being used.

Do I understand correctly that disabling noise reduction is only possible when using the Python Library?

No, but it's not exposed through raspistill/vid. See https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=175711

[VAGrishin]

As I wrote, now there is no SSH access to the camera. The camera Raspi4 read the command line from cloud storage. The command line is performed cyclically by the timer and serves only to change the shooting parameters of raspistill. What happens if we run the system update command in a loop is hard to say. Most likely, the camera will stop working at all.

My colleague Maslov I.A. is planning gradual moving from raspistill to the Picamera library, since it has more features.

Thank you very much for the consultations.
You have given us many links that contain much useful information.
We will study them carefully.