Cannot capture images

[WEIIEW97]

Hello @TimmHess ! Thank you so much for your outstanding work! I am currently working on generating synthesized RGB data with corresponding depth for my deep learning training. I am very new to the unreal engine and this area but I can follow your steps in readme and try to manage to capture materials. However, with my blueprint setting, I cannot see any log info and images cannot be saved in my directory. What am I doing wrong? Or how can I achieve this by capturing RGB and depth in pure c++?(I am ok with c++)

Attached are my blueprint settings.

Thanks in advance! Hope you will spare your time to save my silly question...

[TimmHess]

Hey,
I'm sorry for not getting back to you sooner.

I am currently not at my setup but comparing screenshots from the tutorial text and yours it seems that you (assuming that this is the LevelBlueprint) are not providing the CaptureManager references to the "capture"-functions. Will that be it?

[4ndr3aR]

Ok, I have the same problem, I think it's a sync issue because in the log I see that RunAsyncImageSaveTask() is always called with a PNG filename.

This is my LevelBlueprint trigger:

This is my scene layout:

And these are my CameraCaptureManager components:

I'm using UE 5.2.0, maybe something changed in the rendering pipeline?

EDIT: and another major problem is that captured masks are totally transparent (but I'm probably not understanding which components are considered background and which are considered foreground or "salient" objects).

[4ndr3aR]

This is my LevelBlueprint trigger:

Ok, my fault, I was using CameraCaptureManager_BP2 two times in the blueprint layout so no color image was captured. Now RGB images are captured as they should. Good, now I just need to understand why I get transparent masks...

[4ndr3aR]

Ok, I get transparent masks also using CaptureToDisk.uproject so I'm definitely missing something important (such as the way to define foreground and background).

[4ndr3aR]

They're not transparent! As you wrote here OpenCV correctly shows the segmentation in them! Why both image viewers and PIL don't? Because every pixel has alpha value set to zero you can tell me...

I've asked GPT-4 for a solution, it seems to work. Instead of:

imageWrapper->SetRaw(nextRenderRequest->Image.GetData(), nextRenderRequest->Image.GetAllocatedSize(), FrameWidth, FrameHeight, ERGBFormat::BGRA, 8);

just paste this code:

// Assuming nextRenderRequest->Image is a TArray<FColor>
TArray<FColor> colors = nextRenderRequest->Image;
TArray<uint8> imageData;
imageData.Reserve(colors.Num() * 4); // Reserve space for R, G, B, and A for each pixel

for (const FColor& color : colors)
{
    imageData.Add(color.R);
    imageData.Add(color.G);
    imageData.Add(color.B);
    imageData.Add(color.A);
}

// Set the alpha value to 255 (or 1.0f in normalized format)
for (int32 i = 0; i < imageData.Num(); i += 4) {
    imageData[i + 3] = 255;
}

// Set the modified image data
imageWrapper->SetRaw(imageData.GetData(), imageData.GetAllocatedSize(), FrameWidth, FrameHeight, ERGBFormat::BGRA, 8);

I can submit a pull request if you prefer.