EasySynth

EasySynth is an Unreal Engine plugin for easy creation of image datasets from a moving camera or a multi-camera rig, requiring no C++ or Blueprint knowledge. It leverages native Unreal Engine components to create compelling machine learning datasets, without relying on 3rd party tools.

The plugin works by automatically starting the rendering of a user-defined level sequence, with different camera post-process settings. The outputs are camera poses, including position, rotation, and calibration parameters, together with the following image types:

||| |-|-| |Standard color images, as seen while creating the sequence in the editor|<img src="ReadmeContent/ColorImage.gif" alt="Color image" width="250" style="margin:10px"/>| |Depth images, representing the depth of a pixel using a grayscale value|<img src="ReadmeContent/DepthImage.gif" alt="Depth image" width="250" style="margin:10px"/>| |Normal images, representing pixel normals using X, Y, and Z color values|<img src="ReadmeContent/NormalImage.gif" alt="Normal image" width="250" style="margin:10px"/>| |Optical flow images, representing pixel movement between frames using X, Y, and Z color values|<img src="ReadmeContent/OpticalFlowImage.gif" alt="Optical flow image" width="250" style="margin:10px"/>| |Semantic images, with every object rendered using the user-defined semantic color|<img src="ReadmeContent/SemanticImage.gif" alt="Sematic image" width="250" style="margin:10px"/>| ||Model credits: Art Equilibrium|

Installation

The current main branch of this repo is compatible with Unreal Engine 5.5. For source code or binary builds, targeting older engine versions, please consult previous releases.

Major releases also include example projects amongst the release assets.

Install from the marketplace

This is the easiest method if using Windows and you are not interested in modifying the plugin source code.

• Install a compatible Unreal Engine version through the launcher
• Find the EasySynth page inside the Marketplace tab and hit Install to Engine
• Create a new project and activate the plugin inside the Plugins menu

Install inside a specific project

This method works with both engines installed from the Epic launcher and engines built from source code.

• Create a new Unreal Engine project with the path <UEProject>
  • <em>Optional</em>: Use some of the provided Unreal Engine templates that come with a prebuilt level layout
• Exit the editor
• Create and navigate to the <UEProject>/Plugins directory
• Clone this repo, or unzip the appropriate release inside the <UEProject>/Plugins directory
• Reopen your project

Install inside the engine using prebuilt binaries

This method works with both engines installed from the Epic launcher and engines built from source code but requires using appropriate binaries.

• Navigate to the <UEPath>/Engine/Plugins directory
• Unzip the appropriate release
• Restart the engine
• Create a new project and activate the plugin inside the Plugins menu

Install inside the engine by building from source code

This method only works with engines built from source code.

• Navigate to the <UEPath>/Engine/Plugins directory
• Clone this repo
• Navigate to the <UEPath> directory
• Run the make command
• Restart the engine
• Create a new project and activate the plugin inside the Plugins menu

How to use

Check out our demonstration video showcasing everything from setting up the project to rendering output images.

Setup

• Create a project as described in the Installation section
• Create a level layout yourself, or look online for available levels that suit your needs
• Click the EasySynth button in the toolbar to open the EasySynth widget

Semantic annotation

You can skip this step if you don't need semantic rendering.

The first step is to define the needed semantic classes, which you can modify at any point. To open the semantic classes editor, click the Manage Semantic Classes button. All modifications execute immediately. To close the editor, click the Done button.

This editor allows you to:

• Add a new semantic class
• Remove an existing semantic class
• Modify an existing semantic class name
• Modify an existing semantic class color

Next, you should assign semantic classes to all of the level actors. To do this:

• Select one or more actors of the same class in the editor window or by using the World Outliner
• Supported mesh types are static mesh, skeletal mesh and landscapes
• Assign them a class by clicking on the Pick a semantic class button and picking the class

To toggle between original and semantic color, use the Pick a mesh texture style button. Make sure that you never save your project while the semantic view mode is selected.

A CSV file including semantic class names and colors will be exported together with rendered semantic images. This file can be used for later reference or can be imported into another EasySynth project.

Sequence rendering

Image rendering relies on a user-defined Level Sequence, which represents a movie cut scene inside Unreal Engine.

<b>IMPORTANT:</b> Camera(s) used with the sequencer have to be manually added to the level and then linked with the sequencer. Adding a camera through the sequencer itself will cause unexpected behavior or crashes, due to issues caused by spawnable/possessable sequence actors. For more information see:

https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/Sequencer/Overview/SpawnablesPossessables/

You only need the LevelSequence asset for rendering. Skip anything that has to do with the LevelSequenceActor. Here are some materials on how to get started:

• https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/Sequencer/Overview/
• https://youtu.be/-NmHXAFX-3M

The following tutorial is a good starting point for working with skeletal meshes inside the sequencer:

• https://youtu.be/Xbts4YbhwbE

Setting up rendering options inside the EasySynth widget:

• <em>Optionally</em> import semantic classes from a CSV file (or create them manually)
• <em>Optionally</em> import camera rig from a ROS format JSON file (or create it manually)
• Pick the created level sequence
• Choose the desired rendering targets using checkboxes
• Choose the output image format for each target
  • jpeg - 8-bit image output intended for visual inspection due to lossy jpeg compression,
  • png - 8-bit image output with lossless png compression
  • exr - 16-bit image output with lossless exr compression, to open them with OpenCV in Python use cv2.imread(img_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)
• Choose the output images width and height
  • The aspect ratio of the camera will be updated according to the chosen output size
• Choose the depth infinity threshold for depth rendering
• Choose the appropriate scaling coefficient for increasing optical flow image color saturation
• Choose the output directory

Start the rendering by clicking the Render Images button.

<b>IMPORTANT:</b> Take recorder, unfortunately, cannot be used to record sequences, as we could not find a way to integrate sequences it produces with our plugin. If you have an idea how this can be done feel free to leave your suggestions in the issues.

<b>IMPORTANT:</b> If you need anti-aliasing applied on output images, set Anti-Aliasing Method in project settings to FXAA. Other options will have no effect and output images will have jagged edges.

<img src="ReadmeContent/AntiAliasingSetting.png" alt="Anti-aliasing setting" width="250" style="margin:10px"/>

<b>IMPORTANT:</b> Do not close a window that opens during rendering. Closing the window will result in the successful rendering being falsely reported, as it is not possible to know if the window has been closed from the plugin side.

Multi-camera rigs

EasySynth seamlessly supports rendering using rigs that contain multiple cameras. To create a rig, add an empty actor to the level, and then add any number of individual camera components to the actor, position them as desired relative to the actor position. Then, add the actor to the level sequence and assign it to the camera cut track. When you start rendering, outputs from all of the rig cameras will be created in succession.

Camera rig information can be imported and exported using ROS format JSON files with a specific structure. Clicking on the button Import camera rig ROS JSON file and choosing a valid file will create an actor that represents the described rig inside the level. The camera rig file is also exported during rendering to the selected output directory. Its structure will be described below.

Workflow tips

• You can use affordable asset marketplaces such as Unreal Engine Marketplace or CGTrader to obtain template levels. Ones that provide assets in the Unreal Engine .uasset format are preferred. Formats such as FBX or OBJ can lose their textures when imported into the UE editor.
• When semantic labeling levels with many assets, utilize the World Outliner as much as possible. Use the search option to filter similar actors and bulk-select them so that all of them are labeled in one click.
• If you want images to be spaced out more inside the level sequence (instead of being very close in order to provide a smooth video), you can set the custom level sequence FPS to a small value (1 FPS or lower) inside the Sequencer editor.
• To avoid adding keyframes manually, select the <img src="ReadmeContent/KeyButton.png" alt="Color image" width="20"/> button inside the sequence editor toolbar. After you create the first keyframe, this will automatically add one at the current time fo