🚀 Altera FPGA-based NVIDIA Holoscan Sensor Bridge

Introduction

This repository provides driver code and example applications for integrating Altera FPGAs with NVIDIA's Holoscan Sensor Bridge.

Holoscan Sensor Bridge enables low-latency sensor data processing with GPUs, using an FPGA to acquire peripheral device data and transmit it via UDP to the host system. ConnectX devices can write UDP data directly into GPU memory, facilitating seamless integration with Holoscan pipelines.

Example applications include video processing and inference using an IMX678 MIPI sensor with the Agilex™ 5 FPGA E-Series 065B Modular Development Kit.

Hardware Requirements

The following equipment is needed to test the MIPI Holoscan Sensor Bridge on the Agilex™ 5 FPGA E-Series 065B Modular Development Kit:

• Agilex™ 5 FPGA E-Series 065B Modular Development Kit.
• 1 Framos FSM:GO IMX678C Camera Modules, with either:
  • Wide 110deg HFOV (Horizontal Field of View) Lens.
  • Medium 100deg HFOV Lens.
  • Narrow 54deg HFOV Lens.
  • 150mm flex-cable, or
  • 300mm micro-coax cable.
• USB Micro B JTAG Cable.
• 10GBase-T SFP+ to RJ45 module
• Cat6 Ethernet Cable
• nVidia Jetson Orin AGX Devkit

Agilex™ 5 FPGA E-Series 065B Modular Development Kit setup

Refer to the Agilex™ 5 FPGA E-Series 065B Modular Development Kit project documentation for FPGA setup instructions. (Link TBD)

nVidia Host system setup

Follow nVidia Holoscan Sensor Bridge setup instructions here.

When cloning the Holoscan Sensor Bridge repository, use the Altera mirror:

git clone -b altera-2.3.0 --single-branch https://github.com/altera-fpga/holoscan-sensor-bridge

Demo Applications

The following example applications are available to demonstrate the holoscan sensor bridge working on an Agilex™ 5 FPGA E-Series 065B Modular Development Kit.

Build the Demo Docker Container

cd <holoscan sensor bridge>
sh docker/build.sh

To run a demo application, start the demo container

cd <holoscan sensor bridge>
sh docker/demo.sh

From within the Demo container run one of the following demos:

• Simple Playback Example

python examples/linux_agx5_player.py

• YOLOv8 Body Pose Example

Follow instructions to download YOLOv8 ONNX Model

python examples/linux_body_pose_estimation_agx5.py

• TOA PeopleNet Example

Follow instructions to download toa-peoplenet model

python examples/linux_tao_peoplenet_agx5.py