Image Capturing With UnrealEngine 5.5 (for deep learning)

| Color | Segmentation | |---------|---------| | | |

Changelog

Updated to UE5.5.4
Streamlined capture code to not block the render thread, and further reduce the load on the main game thread.
Fixed alpha channel for pixel annotation in the post-process material. Images are no longer appearing as "empty" because of their alpha mask being 0.
Removed plugins from repository - reduce the mess of files
Slightly updated the tutorial text

Outline

Introduction
MAIN: How to Save Images to Disk (without blocking the main threads)
Capturing Object Pixel-Annotations (segmentation mask)
Capturing Scene Depth
Enable Lumen on SceneCapture2D
Known Issues

TLDR

Use these links to the FrameCaptureManger.h and FrameCaptureManger.cpp file. They are the only source needed. Plus, make sure to link the correct unreal-libs to your project - check the prerequisite or CaptureToDisk.Build.cs.

Kudos to the UE4 and UE5 community!

Special thanks to Panakotta00, for pointing to an even better GPU readback!

Using the source as is you should get an AFrameCaptureManager (Actor) in your scene with the following settings.

Maybe most important is the ImageFormat setting. You want to use PNG for lossless compression when storing object annotation. For color rendered images most likely JPEG gives you better file-sizes. EXR is the float format - the code automatically stores .exr images when used.

Known Issues

Capturing per-pixel annotations is done using the CustomDepth feature. The CustomDepthStencil is of type uint8 which allows a range of 0-255, i.e. we can handle at most 256 different annotations per image!

A Small Introduction

In this repository I condense my findings on how to implement a component to capture images to disk from an arbitrary UE5 (former UE4) scene from scratch lowering the bar for UE novices (and potentially bypassing the need for large frameworks that don't fit ones own particular needs). This will include:

Capturing rendered images to disk at high FPS, without blocking the UE rendering thread or the main game thread
Rendering pixel annotations (or other graphics buffers, such as depth) at the same time

UnrealEngine (UE) is a powerful tool to create virtual worlds capable of AAA productions. Generating temporally consistent data with automatic pixel-wise annotations from complex scenes, such as traffic scenarios, is a capability worth leveraging. Especially for training and validation of machine learning- or deep learning applications it has been explored in a variety of projects. Already, there are plugins available that allow rendering images from UE to disk at runtime, such as prominently Carla, UnrealCV, or AirSim. This repository aims to be a tutorial that demonstrates such an 'image capturing' mechanism in detail for you to understand its inner workings, and in turn enable you to reuse it in a custom fashions that suit the needs of your project.

When I was setting up scenes for my research the plugins mentioned above were just not yet supporting the latest engine versions that I wanted/needed to use. Also, I was missing a place where the knowledge of how to render images to disk was explains for non-advanced graphics-programmers. Of course, there are lots of sources for code available online and also there are community blog-entries scattered across multiple platforms explaining parts of the problem and possible solutions, even though they typically are targeting very particular scenarios.

Disclaimer: I do not claim to own any of the code. Merely, I condensed the sources already available online for easier use and provide an overview to the general functionality of this particular approach!

How to Save Images to Disk In UE5 (without blocking the rendering or main thread)

I will go through the main components of the code step-by-step so that hopefully it will be easier to implement each step as you are following along. However, I recommend looking at the source that is merely a single class (here).

In the explanations I skip certain quality-of-life-like aspects for sakes of readability, for example exposing image resolution settings to the editor instead of hardcoding them. Make sure to check out the sources linked in TLDR

Prerequisite

You will need a UE5 C++ project.

Also, you will have to add a few packages to your 'YourProjectName'.Build.cs file. These are part of UnrealEngine, however, sometimes they are not added automatically resulting in unpleasant (linker) errors. Find the 'YourProjectName'.Build.cs file in the Source/'YourProjectName/ directory, and add or extend it to include the modules: "ImageWrapper", "RenderCore", "Renderer", "RHI", for example like this:

PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "EnhancedInput" , "ImageWrapper", "RenderCore", "Renderer", "RHI"});

Setup A FrameCapture Component

I am using SceneCaptureComponent2D as the basis for capturing images. Placing one of these into your scene will give you an ASceneCaptureComponent which is its Actor instance. It basically behaves like any other camera component, but its viewport is not restricted by your computer's monitor or main camera viewport. This provides us the possibility to render images of arbitrary resolution independent from the actual screen resolution.

Add a FrameCaptureManager class of type Actor to your project.

All functionality to request the capturing of a frame, as well as receiving the rendered image back, and storing the frame to disk will be handled by the FrameCaptureManager.

In the FrameCaptureManager.h we add the following:
FrameCaptureManager.h

#pragma once
class ASceneCapture2D; // forward declaration

#include ... // the stuff that is already there

and to our public variables:

// Color Capture  Components
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category="Capture")
ASceneCapture2D* CaptureComponent;

This enables you to assign a CaptureComponent2d to your FrameCaptureManager inside the UE5 Editor.

Compile and place a FrameCaptureManager in your scene.

As it does not have any primitive to render you will only see it in the editor's outline. In the details panel of the placed FrameCaptureManager you can now see the CaptureComponent assigned to None. From the drop down menu select the CaptureComponent2D you already placed in the scene.

Back to code: We will now prepare our yet "naked" CaptureComponent2D class for capturing images. This includes creating and assigning a RenderTarget - which is basically a Texture to store our image data to - and setting the camera properties.

Note: You could also do this in the Editor but if you deal with, i.e. multiple capture components, you may find it handy not to worry about creating and assigning all the components by hand!

Create a setup function to put all your setup code for the CaptureComponents in the CaptureManger:

FrameCaptureManager.h

protected:

    void SetupCaptureComponent();

FrameCaptureManager.cpp

#include ...

// A bunch of includes we need
#include "Engine/SceneCapture2D.h"
#include "Components/SceneCaptureComponent2D.h"
#include "ShowFlags.h"

#include "Engine/TextureRenderTarget2D.h"
#include "Materials/Material.h"
#include "RHICommandList.h"
#include "IImageWrapper.h"
#include "IImageWrapperModule.h"
#include "ImageUtils.h"

#include "Modules/ModuleManager.h"
#include "Misc/FileHelper.h"


void AFrameCaptureManager::SetupCaptureComponent(){
    if(!IsValid(CaptureComponent)){
        UE_LOG(LogTemp, Error, TEXT("SetupCaptureComponent: CaptureComponent is not valid!"));
        return;
    }

    // Create RenderTargets
    UTextureRenderTarget2D* renderTarget2D = NewObject<UTextureRenderTarget2D>();
    renderTarget2D->InitAutoFormat(256, 256); // some random format, got crashing otherwise
    
    renderTarget2D->RenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA8_SRGB; //ETextureRenderTargetFormat::RTF_RGBA8; //8-bit color format
    renderTarget2D->InitCustomFormat(FrameWidth, FrameHeight, PF_R8G8B8A8, true); // PF... disables HDR, which is most important since HDR gives gigantic overhead, and is not needed!
    renderTarget2D->bForceLinearGamma = true; // Important for viewport-like color reproduction.

    renderTarget2D->bGPUSharedFlag = true; // demand buffer on GPU

    // Assign RenderTarget
    CaptureComponent->GetCaptureComponent2D()->TextureTarget = renderTarget2D;
    // Set Camera Properties
    CaptureComponent->GetCaptureComponent2D()->CaptureSource =  ESceneCaptureSource::SCS_FinalColorLDR;
    CaptureComponent->GetCaptureComponent2D()->TextureTarget->Target

UnrealImageCapture

Install / Use

README