Can't solve how to interact with a mouse during a non blocking visualization with point clouds

[alessandrocapialbi]

Hi everyone, I have this issue with interactions. I have 50 scans of a selected number of LiDAR sensors. On these data I transform all the coordinates from local to global, then I implement clustering with bounding boxes and tracking. These 50 scans are in a loop, so when it finishes to process one of them, it continues with the next. Using the visualizer allows me to have some sort of animation of these point clouds during time. The problem is that I cannot zoom nor rotate freely into my scenario with the mouse, something that you can do in the static contest. I tried with VisualizerWithKeyCallback or VisualizerWIthEditing, but they don't help me much. Any suggestions? Here's the code:

for i in range(20, 71):

point_clouds = []
combined_geometries = []
bounding_boxes = []
all_transformed_xyz = []
all_object_ids = []

# Load the point clouds from the selected sensors
sensors_scans = load_point_clouds_from_sensors(point_cloud_directory, selected_sensors, i)
print(sensors_scans)

# Load and transform scans for each selected sensor
for sensor_id, sensor_scan in zip(selected_sensors, sensors_scans):
    print(f"Loading scan {i} for sensor {sensor_id}")
    transformed_xyz, object_ids = load_and_transform_scan(sensor_scan, sensors_positions_df, centroid,
                                                          sensor_id)
    all_transformed_xyz.append(transformed_xyz)
    all_object_ids.extend(object_ids)

# Convert to numpy arrays for easier handling
all_transformed_xyz = np.vstack(all_transformed_xyz)

if all_transformed_xyz is not None:
    pcd_combined = o3d.geometry.PointCloud()
    pcd_combined.points = o3d.utility.Vector3dVector(all_transformed_xyz)
    print("Number of points before downsampling: ", len(pcd_combined.points))
    pcd_combined = pcd_combined.voxel_down_sample(voxel_size=0.3)
    print("Number of points after downsampling: ", len(pcd_combined.points))

    # Perform DBSCAN clustering and create bounding boxes
    clusters, labels = dbscan_clustering(pcd_combined)
    bounding_boxes, bbox_centroids = create_bounding_boxes(clusters)
    bbox_ids = associate_ids_to_bboxes(bbox_centroids, all_object_ids, all_transformed_xyz)

    # Track vehicles between scans considering the bounding box centroids
    if prev_bbox_centroids:
        matches, exited_vehicles, entered_vehicles = track_vehicles(prev_bbox_centroids, bbox_centroids,
                                                                    prev_ids,
                                                                    bbox_ids,
                                                                    tracking_threshold, frequency)
        print("Matches:\n", pd.DataFrame(matches))
        print("Exited vehicles:", pd.DataFrame(exited_vehicles))
        print("Newly entered vehicles:", pd.DataFrame(entered_vehicles))

    # Update the visualization including trajectories
    update_visualization(vis, pcd_combined, bounding_boxes)

    # Update the previous bounding boxes and centroids
    prev_ids = bbox_ids
    prev_bbox_centroids = bbox_centroids

time.sleep(0.1)

update_visualization function:

def visualize(vis, pcd_combined, bounding_boxes):
# Clear previous geometries
vis.clear_geometries()

# Add new geometries
vis.add_geometry(pcd_combined)
for bbox in bounding_boxes:
    vis.add_geometry(bbox)

# Update the visualization
vis.poll_events()
vis.update_renderer()

def update_visualization(vis, pcd_combined, bounding_boxes):
# Update the visualization in the main thread
visualize(vis, pcd_combined, bounding_boxes)