CV4UE - Computer Vision for Unreal Engine

Description

CV4UE is a plugin for Unreal Engine (>=5.1) that aims to simplify generation of synthetic data for computer vision applications. Initially designed for Crowd simulation, it could be used with Mass Entity System or any predefined object in the scene as well.

Kudos to Unreal Engine Community (espesially TimmHess and Viperdk UE5) and Denisoidd for editing

Demo

Video example

Instalation

• Clone the repo.

• Open Engine/Build/BatchFiles in Unreal Engine location.

• Run RunUAT script to build the plugin:

  • Windows: RunUAT.bat BuildPlugin -plugin="[uplugin file path]" -package="[temporary location]"
  • Linux/OSX: RunUAT.sh BuildPlugin -plugin="[uplugin file path]" -package="[temporary location]"

  Remark Even though the script finished with the message RunUAT ERROR: AutomationTool was unable to run successfully. Exited with code: 6 and something about android, you can go to the next step.

• Copy the plugin from a temporary location to the Engine or your project location.

How to use

• Enable the plugin in Edit > Plugins.
• Select Enabled with Stencil in Edit > Project Settings > Postprocessing > Custom Depth-Stencil Pass.
• Enable Render CustomDepth Pass and set CustomDepth Stencil Value (from 0 to 255) for objects in the scene you would like to register. Different objects should have different CustomDepth Stencil Values.
• Put BP_MultiCapturing from Plugins/cv4ue Content/Blueprints. This Blueprint class contains three instances of BP_Capturing that capture Scene, Depth and Masks.
• Now you can play directly the level in the editor viewport.* The results are located in ProjectFolder/Saved/{img, depth, mask}.

The scene and depth capturings produce .jpeg images, while masks are converted into COCO-like .json annotation files. In the annotation file you find:

• images - list of images data, always contains just one element.
  • id - image ID,
  • width - image width
  • height - image height
  • file_name - image file name
  • license - always 0, not used, license ID
  • fov - field of view of the camera used for capturing
  • offsets - camera offset
  • camera_translation - camera translation
  • camera_rotation - camera rotation
• annotations - list of annotations
  • id - instance ID
  • image_id - image ID
  • category_id - always 1 (person), the id of the current stuff category
  • area - area of the bounding box
  • iscrowd - always 1, the instance represents a single object (0) or a collection of objects (1)
  • segmentation - instance segmentation in RLE format
  • bbox - bounding box in flatted format (upper left corner, lower right corner)

Remark: Firstly, the plugin was created for Crowd Simulation. This is the reason why several values are hard coded. Furthermore, the dataset with the structure of COCO dataset can't be directly created in Unreal Engine without external data post processing.

Useful information

• You can find jupyter notebook with useful Python functions (RLE encoding/decoding, data loading and visualization) in Demo/demo.ipynb
• The tutorial about MassEntity system, the plugin and Unreal Engine is also available in TUTORIAL.md