ofxDVS

An openFrameworks addon for interfacing Dynamic Vision Sensor (DVS / event) cameras with built-in neural-network inference pipelines for real-time object detection and gesture recognition.

Features

• Camera I/O via dv-processing (auto-detects connected camera)
• YOLO object detection on a 5-channel VTEI (Visual-Temporal Event Image) representation, with asynchronous inference on a worker thread
• TSDT gesture recognition using temporal-spatial binary event tensors with EMA-smoothed logits
• Rectangular Cluster Tracker for event-driven multi-object tracking
• ONNX Runtime inference backend with multi-threaded execution and float16 support
• Event reconstruction with per-pixel exponential decay, spatial spread, and ON/OFF color coding (yellow/blue)
• MP4 video recording of the viewer output via ofxFFmpegRecorder (pipes to system ffmpeg)
• 2D and 3D event visualization, APS frame display, IMU overlay
• AEDAT 3.1 and AEDAT4 file recording and playback with real-time speed control
• Hot-pixel suppression via startup calibration mask, refractory period, and rate-based filtering
• Full GUI controls via ofxDatGui

Supported Cameras

Family	Models	Resolution
DVS128	DVS128	128 x 128
DAVIS	DAVIS128, DAVIS208, DAVIS240A/B/C, DAVIS346A/B/Cbsi, DAVIS640, DAVISHet640	varies
DVXplorer	DVXplorer	640 x 480

Select the camera family by setting the #define flags in src/ofxDVS.hpp (DAVIS, DVS128, or DVXPLORER).

Dependencies

Bundled (in libs/)

• libcaer — C driver library for iniVation cameras
• onnxruntime — ONNX Runtime C++ inference engine

External (system / OF addons)

Dependency	Type	Notes
openFrameworks 0.12.0	Framework	Linux 64-bit tested
dv-processing	System library	Camera I/O (dv::io::camera)
ofxDatGui	OF addon	GUI panels
ofxFFmpegRecorder	OF addon	MP4 video recording (requires system ffmpeg)
ofxGui	OF addon (core)	Additional GUI elements
ofxPoco	OF addon (core)	Networking utilities
ofxNetwork	OF addon (core)	Network I/O

Installation

1. Clone or copy this addon into openFrameworks/addons/ofxDVS/.
2. Install dv-processing system-wide (follow their build instructions).
3. Install ofxDatGui into openFrameworks/addons/.
4. Install ofxFFmpegRecorder into openFrameworks/addons/.
5. Make sure ffmpeg is installed system-wide (sudo apt install ffmpeg on Ubuntu).
6. The bundled libcaer and onnxruntime libraries are in libs/. If they don't match your architecture, rebuild and copy them there.

Building the Example

cd example_dvs
make
make run

Or use the openFrameworks Project Generator to create an IDE project that includes ofxDVS and ofxDatGui.

Architecture

src/
  ofxDVS.hpp / .cpp              Core addon class (camera I/O, event processing, draw, GUI)
  dvs_yolo_pipeline.hpp / .cpp   YOLO detection pipeline (VTEI build, inference, NMS, drawing)
  dvs_tsdt_pipeline.hpp / .cpp   TSDT gesture pipeline (event history, tensor build, inference)
  dvs_nn_utils.hpp               Shared NN utilities (sigmoid, NMS, letterbox transforms)
  dvs_gui.hpp / .cpp             NN and tracker GUI panel creation + event handlers
  dvs_inference_worker.hpp       Thread-safe async inference worker (template)
  onnx_run.hpp / .cpp            ONNX Runtime wrapper (load, run, FP16 support)
  RectangularClusterTracker.hpp / .cpp   Event-driven cluster tracker
  ofxDvsPolarity.hpp             Polarity event data structures

Event Reconstruction

The addon includes real-time event-driven image reconstruction. Since DVS cameras only output per-pixel brightness changes (not absolute intensity), this feature integrates events over time to reconstruct a continuous image of the scene.

How it works:

• Each pixel maintains a signed intensity value (-1.0 to +1.0)
• Every frame, all pixels decay towards zero (exponential decay)
• Incoming ON events add intensity; OFF events subtract it
• Events spread to neighboring pixels within a configurable radius, weighted by distance
• Positive values render as yellow (ON activity), negative as blue (OFF activity), fading to black

GUI controls:

Slider	Range	Description
Recon Decay	0.90 – 1.0	Per-frame decay factor. Lower = faster fade, higher = longer trails
Recon Contrib	0.01 – 0.5	Intensity added per event. Higher = brighter pops
Recon Spread	1 – 8	Spatial spread radius in pixels. Higher = softer bleed

Toggle Recon Image in the GUI to enable/disable.

Video Recording

The addon can record the viewer output to MP4 video using ffmpeg. Controls are in the >> Video Output folder on the NN panel.

Button / Slider	Description
START / STOP RECORDING MP4	Begin or end recording. Output: bin/data/dvs_video_<timestamp>.mp4
PAUSE / RESUME RECORDING	Pause recording without starting a new file
REC FPS	Frame rate for the output video (10 – 60, default 30)

Workflow for stitching multiple AEDAT files into one video: start recording, load the first file, pause, load the next file, resume, and so on. Click STOP when done.

Hotkeys

Key	Action
c	Toggle GUI visibility

GUI toggles and sliders control all other functionality (DVS, APS, IMU, tracker, NN, recording, etc.) via the panel interface.

ONNX Models

Place model files in example_dvs/bin/data/. The code loads:

• YOLO: ReYOLOv8m_PEDRO_352x288.onnx (object detection)
• TSDT: spikevision_822128128_fixed.onnx (gesture recognition)
• TPDVSGesture: tp_gesture_paper_32x32.onnx (gesture classification)

Build Notes for Ubuntu 20.04 + OF 0.12.1

Building on Ubuntu 20.04 requires several adjustments due to toolchain and library version differences.

Prerequisites

# g++-13 (dv-processing 2.0.3 requires C++20)
sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt-get install -y g++-13

# Inivation PPA for dv-processing and boost-inivation
sudo add-apt-repository ppa:inivation-ppa/inivation -y
sudo apt-get install -y dv-processing boost-inivation

# OF dependency fix: libssl3 does not exist on 20.04
# In scripts/linux/ubuntu/install_dependencies.sh, replace libssl3 with libssl1.1

# Install onnxruntime (bundled in libs/onnxruntime/)
sudo cp libs/onnxruntime/libonnxruntime*.so* /usr/local/lib/
sudo rm /usr/local/lib/libonnxruntime.so /usr/local/lib/libonnxruntime.so.1
sudo ln -s libonnxruntime.so.1.20.0 /usr/local/lib/libonnxruntime.so.1
sudo ln -s libonnxruntime.so.1 /usr/local/lib/libonnxruntime.so
sudo ldconfig

C++ Standard Conflict

OF 0.12.1 detects the system gcc (v9) and sets -std=c++17, but dv-processing requires C++20. The example_dvs/Makefile appends CXXFLAGS += -std=gnu++20 after including the OF build system so the last -std= flag wins.

fmt Version Mismatch

OF 0.12.1 bundles fmt v11 headers but Ubuntu 20.04 only has libfmt.so.9. This causes linker errors for fmt::v11::* symbols. Fixed by adding -DFMT_HEADER_ONLY to PROJECT_CFLAGS in example_dvs/config.make.

Inivation Boost

dv-processing depends on boost-inivation (>= 1.80) installed at /opt/inivation/boost/include. The system boost 1.71 lacks required headers (boost/endian.hpp, boost/nowide/). Added -I/opt/inivation/boost/include to PROJECT_CFLAGS.

C++20 Namespace Fixes

g++-13 with C++20 is stricter about unqualified names from std::. The following were fixed:

• ofxDVS.hpp: ios:: → std::ios::, ifstream → std::ifstream, fstream → std::fstream
• ofxDatGui: unique_ptr → std::unique_ptr, make_unique → std::make_unique, min → std::min
• ofTypes.h: added #include <memory>
• ofxFFmpegRecorder.h: added #include <list>, added setInputPixelFormat(const std::string&) method

License

MIT License — Copyright (c) 2017 Federico Corradi