How to Access Image Data in C++ (not save to disk)

[dilne]

Hi,

Apologies for the newby question. I have a 2DSceneCapture component that I have very simply made to follow an actor. I want to access the data from the scene capture in C++ so I can do some postprocessing with the image. I don't want to save to disk however.

Is this code under the Setup A ColorCapture Component section in your read me assigning the raw image data from the scene capture to the variable renderTarget2D? So my question is, is this render target basically holding the image data?:

// Assign RenderTarget
    captureComponent->GetCaptureComponent2D()->TextureTarget = renderTarget2D;

Thanks in advance

[TimmHess]

Hey,

I will try to answer your question, but I am not sure if I understand correctly what it is you are trying to achieve, which is why it may point towards a different direction...

is this render target basically holding the image data?:

yes, but there are two things to be kept in mind:

1. The image data stored in the RenderTarget is a copy not a pointer to the displayed image in your viewport
2. To actually gain useful access to the data, such that you could do manipulations of pixel values, you will still need the as detailed in the section on "Implement the image capturing function, because you will need to issue the rendering of the image data to the RenderTargetTexture.

So basically if you leave out all implementation from the "Save Image Data to Disk" section, you are left with the const TArray<uint8>& ImgData that holds your rendered images data.

I hope this somewhat clarifies :) Otherwise, please feel free to follow up 👍

[dilne]

Thanks for your response. You answered my question perfectly. For now, I am just trying to implement the full code from your repo to test out saving to disk, just to make sure everything is working before I make the changes I want to.

I get an error when I implement the code from the Override the Tick function of the CaptureManager section. Do you know what might be causing the error and how I can fix it?

Full code for CaptureManager.cpp: https://gist.github.com/dilne/715162238d4c3487ababc991f75fe483
Full code for CaptureManager.h: https://gist.github.com/dilne/f2c3ee67fdf5fd66f55752b93087663c

[TimmHess]

Good to hear!

Regarding the code - sorry, that should have been fixed a long time ago.. I must have missed that one in the readme.. I will update it right away.
You are using const TArray<uint8>& ImgData, where its const TArray64<uint8> ImgData. This is what your error message is trying to tell you but it is easy to miss.

[dilne]

Awesome! Thanks for your help! I also had to update the following code to TArray64<uint8> to get a working build.

protected:
	// Creates an async task that will save the captured image to disk
	void RunAsyncImageSaveTask(TArray<uint8> Image, FString ImageName);

public:
	AsyncSaveImageToDiskTask(TArray<uint8> Image, FString ImageName);
	~AsyncSaveImageToDiskTask();

AsyncSaveImageToDiskTask::AsyncSaveImageToDiskTask(TArray<uint8> Image, FString ImageName) {
    ImageCopy = Image;
    FileName = ImageName;
}

Now to test, how do I call the CaptureColorNonBlocking() from the level blueprints? What should I search for from within the blueprint to do this?

I think maybe your screenshot of the blueprint didn't upload under the bit where you say: "For testing purposes we can call the CaptureColorNonBlocking() from the LevelBlueprint by attaching it to a button pressed event."

Thanks again for all your help! Interestingly, even though my code wasn't working last night, I just checked my Saved folder and it seems I have an image saved that was created last night, I'm not sure how that happened!

[TimmHess]

Nice!

Unreal comes with a super convenient way of exposing c++ functionality to blueprint with Macro-Magic ;)
The UFUNCTION(BlueprintCallable, Category = "ImageCapture") decoration of the void CaptureColorNonBlocking(...) method is all you need.

UFUNCTION(BlueprintCallable, Category = "ImageCapture")
void CaptureColorNonBlocking(...)

In the LevelBlueprint you can drag and drop your CaptureManager that is placed in your scene to the LevelBlueprint editor. Drawing a new 'connection' you should see the CaptureColor functionality as a suggestion.
It will look something like this:

Of course, this is directly transferable to any other Blueprint (Actor, etc.) or C++ class as well, you just need access to the CaptureManager.

[TimmHess]

Note that there are 2 separate CaptureManagers used in the above example :D

[dilne]

Thanks, I'm learning lots! My project crashes completely when I press my save to disk key. I get this error:

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000308

UE4Editor_TimmHess_0942!ACaptureManager::CaptureColorNonBlocking() [C:\Users\dmiln\UE4\TimmHess\Source\TimmHess\CaptureManager.cpp:25]
UE4Editor_TimmHess_0942!ACaptureManager::execCaptureColorNonBlocking() [C:\Users\dmiln\UE4\TimmHess\Intermediate\Build\Win64\UE4Editor\Inc\TimmHess\CaptureManager.gen.cpp:99]

Line 25 in the cpp is the last line shown here:

void ACaptureManager::CaptureColorNonBlocking(ASceneCapture2D* CaptureComponent, bool IsSegmentation) {
    // Get RenderContext
    FTextureRenderTargetResource* renderTargetResource = CaptureComponent->GetCaptureComponent2D()->TextureTarget->GameThread_GetRenderTargetResource();

When I check the logs, it says this:

LogWindows: Failed to load 'aqProf.dll' (GetLastError=126)
LogWindows: File 'aqProf.dll' does not exist
LogWindows: Failed to load 'VtuneApi.dll' (GetLastError=126)
LogWindows: File 'VtuneApi.dll' does not exist
LogWindows: Failed to load 'VtuneApi32e.dll' (GetLastError=126)
LogWindows: File 'VtuneApi32e.dll' does not exist
LogConsoleResponse: Display: Failed to find resolution value strings in scalability ini. Falling back to default.
LogConsoleResponse: Display: Failed to find resolution value strings in scalability ini. Falling back to default.
LogInit: Display: Running engine for game: TimmHess

I'm not sure if this is an issue with the code, or builds or what. I'll try searching elsewhere on the web just in case, but if you've got any ideas please let me know. It might be something to do with why I couldn't get my blueprint looking the same as yours?

[TimmHess]

Yea that's the thing with C++ in unreal...
Maybe you can "shotgun" debug by adding a bunch of UE_LOG(LogTemp, Warning, TEXT("some debug message") lines in the code to see where exactly it crashes in the function.
You find these logs in the console or /Saved/Logs/ directory of your project. But do not restart the editor after the crash or else the log is overwritten

[dilne]

Thanks for the tip! I saw from your plugin code that you implemented the following:

if (!IsValid(CaptureComponent)) {
        UE_LOG(LogTemp, Error, TEXT("CaptureColorNonBlocking: CaptureComponent was not valid!"));
        return;
    }

I implemented this and now UE no longer crashes and I see this error message in the logs! So now I need to understand why the CaptureComponent is not valid, any ideas on where to start?

[TimmHess]

Alright, that is a start.

You do have an ASceneCapture2D (not an ACamera) in your scene? And it is connected to the CaptureManager?
Sounds a lot like that might be missing :/

[dilne]

I have dragged my CaptureManager into the scene and added a SceneCapture2D to the scene. I then click CaptureManager in the World Outliner and under Capture > Color Capture Components I select SceneCapture2D1, which is the Scene Capture 2D that I previously dragged into my scene. I'm assuming this is what you mean by connecting them?

I don't need to do something like create a render target do I? Or is there some kind of setting on the SceneCapture2D that needs to change?

[TimmHess]

Yes, that is what I meant.

All setup of the ASceneCapture2D is done in the SetupCaptureComponent() method...

Ok now I see.. sorry this is the result of me switching the functionality of the CaptureManager while writing the tutorial..
In your LevelBlueprint there is also a connection to a CaptureComponent needed.. You could actually replace that later (as I did in the PluginCode) but in 'your' implementation, a.k.a. the instruction in the tutorial 🙈, you will have to provide this parameter as well.

[TimmHess]

At least I think that will be it

[dilne]

Yes, you got it!!! My output is a bit janky though!

Changing to .png fixes it nicely however.

Thanks for all your amazing help! I'd like to thank you in some way for your kindness, patience and for putting this repo up. Often I'm speaking to coders/developers on LinkedIn and I 'endorse' some of their coding skills under their skills section when they've been a great help. I'd happily do the same for you if you'd like or something else, let me know. I'll be sure to give this repo a star either away!

[TimmHess]

Yes, awesome!

the 'janky' artifact you are seeing is/might be jpeg compression. Make sure to always use png (lossless compression) to store labels.

You are very welcome, this stuff working for you is enough for me. I remember the struggles to get this together very well and I am glad to help! 👍
Good luck with your project. If there is anything I might be able to help with, do not hesitate to ask.

[dilne]

Hey, rather than selecting a SceneCapture2D that I drag into the scene, do you know how I can select the SceneCapture2D if it is a component inside a blueprint class? I'd like do to this because the blueprint class contains blueprints that make the SceneCapture2D follow the player character.

[TimmHess]

In that case, maybe I would make an actor reference in the actor (blueprint?) that your SceneCapture is a component of, and pass it the CaptureManager actor. This is almost always possible because many classes inherit from AActor eventually.
Now your, e.g. PlayerCharacter, has a reference to the CaptureManager and you can proceed as before. You will, however, need to cast your actor variable to CaptureManager to access its methods of course.
Also, all of that should be possible within Blueprint.

[dilne]

So I've create an actor object reference in my actor BP (called CapManRef) that my SceneCapture is a component of. I then pass in the CaptureManager actor via the Details pane. Therefore, I believe the casting should work in theory, but then when trying to set the Color Capture Component I see that because my SceneCapture is a component it therefore isn't compatible. Do you know how I can fix this please?

[TimmHess]

You should be able to access the component with this, after all, it seems actors are merely containers of components