Optical-Flow-And-Depth-Estimation-From-Event-Cameras-Based-On-Deep-Learning-Techniques

Data from event cameras have become readily available due to new camera products and public databases. These systems offer high temporal resolution, high dynamic range, low power consumption and high pixel bandwidth making them attractive for image-based navigation solutions. However, new computational techniques are required for this completely different data format. This research explores the use of deep learning techniques for the estimation of optical flow and depth from this data.

Fig: Optical Flow Example

Fig: Depth Estimation Example

1. Datasets

DSEC

The DSEC dataset can be downloaded here. The script download_dsec_test.py can be used to extract the data. It downloads the dataset directly into the OUTPUT_DIRECTORY with the expected directory structure.

download_dsec_test.py OUTPUT_DIRECTORY

MVSEC

The MVSEC dataset can be downloaded here:

MVSEC Outdoor Day 1 for 20 Hz evaluation

MVSEC Outdoor Day 1 for 45 Hz evaluation

DENSE

The Depth Estimation oN Synthetic Events (DENSE) Dataset can be downloaded here:

• DENSE

MVSEC	DSEC	DENSE

2. Optical Flow Estimation

Download

Download the network checkpoints and place them in the folder checkpoints/:

Checkpoint trained on DSEC

Checkpoint trained on MVSEC 20 Hz

Checkpoint trained on MVSEC 45 Hz

Running

This part consists in getting the optical flow predictions for each frame. To do that, the python file 'main.py' can be used.

Fig: Event Source

Fig: Optical Flow Estimation

3. Depth Estimation

The python code 'run_depth.py' is used in this part to get the depth estimations for each frame of a sequence.

Running

• Download the pretrained model:

wget "http://rpg.ifi.uzh.ch/data/E2DEPTH/models/E2DEPTH_si_grad_loss_mixed.pth.tar" -O pretrained/E2DEPTH_si_grad_loss_mixed.pth.tar

• Download the test sequence in the DENSE dataset:

wget "http://rpg.ifi.uzh.ch/data/E2DEPTH/dataset/test_sequence_00_town10.zip" -O data/test_sequence_00_town10.zip

• Extract the data sequence:

unzip -q data/test_sequence_00_town10.zip -d data/test

Before running the depth prediction, make sure the conda environment is sourced:

conda activate E2DEPTH

• Run reconstruction:

python run_depth.py -c pretrained/E2DEPTH_si_grad_loss_mixed.pth.tar \
  -i data/test/events/voxels \
  --output_folder /tmp \
  --save_numpy \
  --show_event \
  --display \
  --save_inv_log \
  --save_color_map

Fig: Event Source

Fig: Depth Estimation