Add nav2 perception pipeline (RGB → depth → point cloud) by Vidyadharan98 · Pull Request #1 · ros-navigation/navigation2.ai

Vidyadharan98 · 2026-03-14T10:12:06Z

Summary

This PR adds the nav2_perception_pipelines package, which provides a configurable perception pipeline that converts RGB images into depth maps and point clouds.

The pipeline is modular and configurable, allowing users to swap components such as image sources and depth estimation models through a YAML configuration file.

All components run as ROS 2 composable nodes inside a single container with intra-process communication enabled for improved efficiency.

Key Features

End-to-end pipeline:
image source → preprocessing → depth estimation → point cloud projection
Configurable image source and depth estimator to allow swapping with alternative implementations
Preprocessing parameters fully configurable via YAML
Launch file composes all nodes into a single component container with intra-process communication enabled

Demo

nav2_perception_pipe_demo_1.mp4

Issue reference

Implements the perception pipeline discussed in ros-navigation/navigation2#5536

…d config Signed-off-by: Vidyadharan98 <vidyadharan98@gmail.com>

SteveMacenski

So this generally looks good, but where's the example of it integrated into Nav2 (configuration file, tutorial launch file, etc) and video of it all working? This is a good first step to get it going, now its about actually showing it in use in an example configuration/launch file/short video

Overall, a good first run!

SteveMacenski · 2026-04-08T18:26:06Z

@sachinkum0009 @Vidyadharan98 Any update? I'd LOVE to have this in before Lyrical!

Vidyadharan98 · 2026-04-09T02:59:08Z

@sachinkum0009 @Vidyadharan98 Any update? I'd LOVE to have this in before Lyrical!

Hello @SteveMacenski , I have made some updates but yet to complete. Occupied with some personal commitments lately 😅. But we will be able to complete this before Lyrical. Will update soon. Thank you.

sachinkum0009 · 2026-04-20T18:36:24Z

Hi, @SteveMacenski @Vidyadharan98

I have setup the Turtlebot3 with camera and tried to run basic implementation
RGB->Depth->Pointcloud->Voxel Costmap Layer (Nav2)

I have attached image for Rviz containing the RGB, Depth images, with Pointcloud and Nav2 costmap for voxel layer.

SteveMacenski · 2026-04-20T19:53:30Z

@sachinkum0009 any interest in helping drive this to the finishline? There's a few open comments and then obviously using your videos and images. Thanks for the images (and hopefully video navigating using it)!

Vidyadharan98 · 2026-04-21T02:04:52Z

Hello @sachinkum0009 , thank you for sharing the demo 💙.

Hello @SteveMacenski , apologies for the delay. Could you please give me until April 27? I’d like to complete it by then.

SteveMacenski · 2026-04-21T03:33:01Z

OK!

sachinkum0009 · 2026-04-21T07:58:57Z

@sachinkum0009 any interest in helping drive this to the finishline? There's a few open comments and then obviously using your videos and images. Thanks for the images (and hopefully video navigating using it)!

Thanks, yes. I will do the navigating test today and will record some videos.

@Vidyadharan98 Can you please invite me to your repo? I will take care of the comments and will push to finishline. 😄

sachinkum0009 · 2026-04-21T16:13:41Z

Hi @SteveMacenski @Vidyadharan98

Please find the video attached of TB3 Navigation2.

tb3_nav2_dep_any.mp4

There are two issues I want to investigate.

The Pointcloud data is flickering a little, it could be because of Camera auto focus, exposure or white balance. (will investigate in upcoming days)
I am thinking of post processing the PC to remove the ground plane and remove noise using PointCloud filters

LMK, what are your suggestions for this?

SteveMacenski · 2026-04-21T16:53:36Z

I think thoes are good suggestions! Maybe not removing the ground plane (the minimum height filter should take care of that in the voxel layer - maybe just needs to be increased?) but removing noise may be good.

- End-to-end pipeline: image source → preprocessing → depth estimation → point cloud projection - Image source and depth estimator are configurable to allow swapping with alternative implementations - Preprocessing parameters are fully configurable via YAML - Launch file composes all nodes into a single component container with intra-process communication enabled Signed-off-by: Vidyadharan98 <vidyadharan98@gmail.com>

Vidyadharan98 · 2026-04-26T04:48:57Z

Hello @SteveMacenski ,
I’ve pushed my changes, but there are still a few comments that need to be addressed.

Hello @sachinkum0009 ,
Thank you for sharing the navigation demo video. I’ve sent you a collaboration invite—please check and accept it when you have a moment.

Thank you for your understanding!

Signed-off-by: Vidyadharan98 <vidyadharan98@gmail.com>

- params launch file for all nodes - CMakelist updated to include params folder Co-authored-by: Copilot <copilot@github.com>

…s in perception pipeline launch file

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

SteveMacenski · 2026-04-27T21:35:22Z

Check out the still open comments - there are a few :-) I think the launch file needs to be redone from scratch using standard ROS 2 launch formatting. The tutorial README / configuration yaml then updated respectively.

…ch file Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

…rmatting

…for depth_anything_v3 Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

sachinkum0009 · 2026-04-28T16:08:51Z

To Do

Add license
Add launch file to use costmap
Update readme with instructions for costmap
Add PC filter to remove noise

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

- removed commented code Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

SteveMacenski · 2026-04-28T18:48:49Z

+### Image Source
+
+Defines the node responsible for providing the **input image stream** to the perception pipeline.
+
+Example configuration:
+
+```yaml
+image_source:
+  type: rgb
+  package: usb_cam
+  plugin: usb_cam::UsbCamNode
+  parameters:
+    video_device: /dev/video0
+    image_width: 640
+    image_height: 480
+    pixel_format: mjpeg2rgb
+    frame_rate: 30.0
+  topics:
+    output_topic: /image_raw
+    camera_info_topic: /camera_info
+```
+
+| Parameter                  | Description                                                    |
+| -------------------------- | -------------------------------------------------------------- |
+| `type`                     | Specifies the input image type used by the pipeline. Supported types: `rgb` or `depth`           |
+| `package`                  | ROS 2 package that provides the image source node.             |
+| `plugin`                   | Fully qualified composable node plugin used to start the node. |
+| `parameters`               | Configuration parameters passed to the image source node.      |
+| `topics.output_topic`      | Topic where the node publishes the image stream.               |
+| `topics.camera_info_topic` | Topic where the node publishes camera calibration information. |


Old I think?

Yes, I don't think, we need these explanation for these params now, as they are self explanatary. Should I remove them?

I don't think we still use this old launch file method, is there a "package" and "plugin" "type" parameter?

no, these params were from old yaml file.

I don't understand... The launch file now uses a consistent node, why does the README tutorial still have things like the following:

image_source: type: rgb package: usb_cam plugin: usb_cam::UsbCamNode ...

SteveMacenski · 2026-04-28T18:50:30Z

Thanks @sachinkum0009 - looks better already :-) Let me know on the costmap parts, but nit-pick tweaks at this point from my review only.

Co-authored-by: Steve Macenski <stevenmacenski@gmail.com> Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

- fixed the typo for remapping Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

- nav2 params updated with voxel costmap layer for tb3 - rviz config added to visualize the pointcloud - exec dependencies added for the - removed config dir and added rviz dir in cmakelists - readme updated with instructions for costmap layer Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

sachinkum0009 · 2026-04-29T14:58:44Z

Hi @SteveMacenski

Almost most of the things have be fixed. Would like to ask for another review with suggestions.

will add and test the PC filter to remove noise tomorrow.
plan to test different camera to see if flickering still happens

Also, would like to ask which License should be added to the launch files?

Thanks

SteveMacenski · 2026-04-29T19:25:18Z

Also, would like to ask which License should be added to the launch files?

Apache2.0 in general unless you have a reason to do otherwise, is my preference to be consistent with other Nav2 code

Co-authored-by: Steve Macenski <stevenmacenski@gmail.com> Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

- Removed the launch file for the waffle as it doesn't fit - Updated package and cmakelist - Updated readme for users to integrate with their nav2 requirements. Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

sachinkum0009 · 2026-05-03T10:04:45Z

Hi @SteveMacenski

Thanks for the review. I have pushed the changes.

SteveMacenski

More or less looks good to me. Do you think you could record a video where there isn't so much costmap noise in the scene before navigating (i.e. clearing the costmap from the previous test)? There's some noise in there I'd love to not have so that we can see the navigation more clearly without distracting noise.

I think with the few comments below, this is good to merge.

The next steps here would be to convert the README into a Nav2 tutorial and probably expand a little on the explanations. Imagine you're starting without knowing anything; explaining each of the technology items, how we put them together, and the steps. Maybe explain a bit more on the model / you can use any RGB camera / why we might want to do some pre- or post-processing / some of the key costmap configurations for this / etc.

SteveMacenski · 2026-05-05T16:44:45Z

+
+This package provides a **perception pipeline** using AI-based depth estimation using DepthAnything V3 from RGB images for use in navigation and mobility tasks.
+
+The pipeline is designed to be **modular and configurable**, allowing users to swap components such as image sources and depth estimation models using a YAML configuration file.


Suggested change

The pipeline is designed to be **modular and configurable**, allowing users to swap components such as image sources and depth estimation models using a YAML configuration file.

The pipeline is designed to be **modular and configurable**, allowing users to swap components such as image sources, pre- and post-processing nodes, and depth estimation models by modifying the launch file or configuration file.

SteveMacenski · 2026-05-05T16:48:10Z

This is still open #1 (comment). I think some of these config parameters are from the old version that aren't used anymore

sachinkum0009 · 2026-05-06T13:14:17Z

Thanks again for the review. I will update the old version config parameters and apply the filter to remove the noise for the pointcloud to create a map.

Add nav2_perception_pipelines package skeleton with initial launch an…

5690ee6

…d config Signed-off-by: Vidyadharan98 <vidyadharan98@gmail.com>

SteveMacenski reviewed Mar 15, 2026

View reviewed changes

Vidyadharan98 force-pushed the feature/nav2-perception-pipelines branch from 1f27d65 to 9705f0a Compare April 26, 2026 03:38

Rename package to nav2_depth_estimation_ai

0f839ca

Signed-off-by: Vidyadharan98 <vidyadharan98@gmail.com>

Vidyadharan98 force-pushed the feature/nav2-perception-pipelines branch from e3f9aac to 0f839ca Compare April 26, 2026 04:52

sachinkum0009 and others added 3 commits April 27, 2026 22:23

perception pipeline launch file added

d3584d2

- params launch file for all nodes - CMakelist updated to include params folder Co-authored-by: Copilot <copilot@github.com>

Update PointCloudXyzNode to PointCloudXyzrgbNode and adjust remapping…

2e4730a

…s in perception pipeline launch file

launch file formated

479996d

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

SteveMacenski reviewed Apr 27, 2026

View reviewed changes

Comment thread nav2_depth_estimation_ai/launch/perception_pipeline_launch.py Outdated

Comment thread nav2_depth_estimation_ai/launch/perception_pipeline_launch.py Outdated

Comment thread nav2_depth_estimation_ai/README.md Outdated

sachinkum0009 added 4 commits April 28, 2026 16:29

Add USB camera and image processing nodes to perception pipeline laun…

2afed74

…ch file Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

Refeactored the perception pipeline as per standard of ROS2 launch fo…

7edf501

…rmatting

navigation demo video updated

e46753c

Update README.md with build instructions and model preparation steps …

def76f0

…for depth_anything_v3 Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

sachinkum0009 added 2 commits April 28, 2026 19:47

Disable debug mode in depth_anything_v3 parameters

3efc2e1

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

Update image remapping from compressed

82a5c82

- removed commented code Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

SteveMacenski reviewed Apr 28, 2026

View reviewed changes

sachinkum0009 and others added 8 commits April 28, 2026 20:59

Update nav2_depth_estimation_ai/README.md

1157ba0

Co-authored-by: Steve Macenski <stevenmacenski@gmail.com> Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

Comment added to replace your sensor driver

e7d175e

Co-authored-by: Steve Macenski <stevenmacenski@gmail.com> Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

Comment added to replace sensor driver for composable node

2cc5ff2

Co-authored-by: Steve Macenski <stevenmacenski@gmail.com> Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

update model path to use package share directory

b544615

- fixed the typo for remapping Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

instructions updated to add model configuration

5d28630

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

Remove outdated parameter descriptions from README.md

138142a

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

update model path in README to use package share directory

153270a

Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

SteveMacenski reviewed Apr 29, 2026

View reviewed changes

sachinkum0009 and others added 2 commits May 3, 2026 11:43

Add instructions for using a camera with ROS driver.

c8279fc

Co-authored-by: Steve Macenski <stevenmacenski@gmail.com> Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

- Add license header to launch file

9d0dae1

- Removed the launch file for the waffle as it doesn't fit - Updated package and cmakelist - Updated readme for users to integrate with their nav2 requirements. Signed-off-by: Sachin Kumar <sachinkum123567@gmail.com>

SteveMacenski reviewed May 5, 2026

View reviewed changes


		This package provides a perception pipeline using AI-based depth estimation using DepthAnything V3 from RGB images for use in navigation and mobility tasks.

		The pipeline is designed to be modular and configurable, allowing users to swap components such as image sources and depth estimation models using a YAML configuration file.

Conversation

Vidyadharan98 commented Mar 14, 2026

Summary

Key Features

Demo

Issue reference

Uh oh!

SteveMacenski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SteveMacenski commented Apr 8, 2026

Uh oh!

Vidyadharan98 commented Apr 9, 2026

Uh oh!

sachinkum0009 commented Apr 20, 2026

Uh oh!

SteveMacenski commented Apr 20, 2026

Uh oh!

Vidyadharan98 commented Apr 21, 2026

Uh oh!

SteveMacenski commented Apr 21, 2026

Uh oh!

sachinkum0009 commented Apr 21, 2026

Uh oh!

sachinkum0009 commented Apr 21, 2026

Uh oh!

SteveMacenski commented Apr 21, 2026

Uh oh!

Vidyadharan98 commented Apr 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SteveMacenski commented Apr 27, 2026

Uh oh!

sachinkum0009 commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To Do

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SteveMacenski Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

sachinkum0009 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

SteveMacenski Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

sachinkum0009 Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

SteveMacenski Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SteveMacenski commented Apr 28, 2026

Uh oh!

sachinkum0009 commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SteveMacenski commented Apr 29, 2026

sachinkum0009 commented Apr 28, 2026 •

edited

Loading

SteveMacenski Apr 29, 2026 •

edited

Loading

SteveMacenski left a comment •

edited

Loading