Vision Pipeline Tasks

Vision Pipeline Tasks

The pipeline exposes 6 tasks, each triggered by name via the Brain Client or directly via the ROS 2 action interface. Below is a detailed description of each.

---

Task Overview

Task Name	Description	Output Topic	Source File
car_objects	YOLO detection + 3D pose of car parts	/car/pose	action_yolo_object_detection.py + action_pose_pca.py
place_object	Finds a safe empty zone on the table to place an object	/perception/target_place_pose	place_object.py
subdoor_pose	Estimates 3D corner poses of a subdoor panel	/body_markers	subdoor_pose_estimator.py
detect_screws	YOLO detection + 3D pose of screws	/screw/pose	action_yolo_object_detection.py + action_pose_pca.py
detect_screwdriver	OBB detection + 6D pose of screwdriver	/object_pose_screwdriver	action_obb_object_detection.py + action_obb_pose.py
table_height	RANSAC-based table surface height estimation	/perception/table_pose	table_height_ransac.py

---

1. Car Object Detection

Task name: car_objects

Detects car parts (e.g. motors) using a fine-tuned YOLOv8 model and computes the 3D centroid pose from the depth point cloud using PCA.

Nodes activated: yolo_car_node → cropper_car_node

Parameters:

• object_class: optional class filter (e.g. motor, motor_grip). Selecting motor or motor_grip switches to the fine-tuned model; other classes use the default model.

Outputs:

• /car/pose — PoseStamped of the detected object centroid
• /car/radius — Float32 bounding sphere radius (used by Place Object)

Trigger:

ros2 run perception brain_client car_objects
ros2 run perception brain_client car_objects motor

---

2. Place Object

Task name: place_object

Segments the table surface from the 3D point cloud, detects occupied regions, and finds the nearest safe empty zone large enough to place the previously detected object. The required placement radius is received from the car/screw pipelines.

Node activated: place_object_node

Modes:

Mode	Description
test	Uses a fixed test_radius parameter (default 7 cm)
actual	Listens to /car/radius and /screw/radius for the real object size

Key parameters:

• mode: test or actual
• test_radius: float (metres), default 0.07
• safety_margin: float (metres), default 0.02
• debug_viz: bool — publishes debug point clouds and markers to RViz

Outputs:

• /perception/target_place_pose — PoseStamped of the safe placement point

Trigger:

ros2 run perception brain_client place_object

---

3. Subdoor Poses

Task name: subdoor_pose

Uses YOLOv8-Pose to detect human body keypoints repurposed for subdoor panel corner estimation. Returns up to 4 corner PoseStamped markers.

Node activated: subdoor_pose_estimator

Inputs: Aligned RGB image + point cloud

Outputs:

• /body_markers — MarkerArray (namespace corners) with one marker per detected corner

Result message format:

Subdoor estimated successfully with 4 corner markers.
Pose 0: x=0.512, y=-0.034, z=0.891
...

Trigger:

ros2 run perception brain_client subdoor_pose

---

4. Screw Detection

Task name: detect_screws

Detects screws using a dedicated fine-tuned YOLOv8 model (low confidence threshold: 0.3) and estimates their 3D positions via point cloud cropping and PCA.

Nodes activated: yolo_screw_node → cr![alt](url)opper_screw_node

Model: Screw-specific fine-tuned weights, conf_threshold: 0.3

Outputs:

• /screw/pose — PoseStamped centroid of detected screw
• /screw/radius — Float32 bounding sphere radius

Trigger:

ros2 run perception brain_client detect_screws

---

5. Screwdriver Detection

Task name: detect_screwdriver

Uses YOLOv8-OBB (Oriented Bounding Box) to detect the screwdriver's precise orientation. The point cloud is then cropped to the OBB region and a 6D pose (position + quaternion) is estimated.

Nodes activated: obb_detector_node → point_obb_cloud_cropper_node

Why OBB? Standard axis-aligned bounding boxes cannot capture tool orientation. OBB gives a rotated box that aligns with the object's principal axis, enabling accurate 6D pose estimation.

Outputs:

• /object_pose_screwdriver — PoseStamped with full 6D pose (position + quaternion orientation)

Result message format:

Screwdriver detected at [x: 0.312, y: -0.045, z: 0.723]
Orientation: [qx: 0.012, qy: 0.034, qz: 0.707, qw: 0.706]

Trigger:

ros2 run perception brain_client detect_screwdriver

---

6. Table Height

Task name: table_height

Estimates the height of the table surface using RANSAC plane fitting on the 3D point cloud. The result is transformed into the robot's base frame (eddie_base_footprint) via TF2.

Node activated: table_height_estimator

Method: RANSAC plane segmentation with Open3D, followed by TF2 transform to the robot base frame.

Accuracy: ±5 cm. A safety margin should be applied by the robot controller.

Outputs:

• /table_height_value — Float32 height in metres
• /perception/table_pose — PoseStamped of the table plane origin in base frame

Result message format:

Table Pose Found: [x: 0.000, y: 0.000, z: 0.583 (Height)]

Trigger:

ros2 run perception brain_client table_height