Questions on Reproducing the Experimental Results

[Laura-Ting]

Dear Developers,

first and foremost, I would like to express my gratitude for your remarkable work, which has greatly piqued my interest. However, I am encountering some challenges in reproducing the experimental results as described in your paper, and I would appreciate your assistance in clarifying a few points.

Queries:

1. It is evident that the number of segment nodes and corresponding bounding boxes in Clio online is greater than in the Pre-built Scene Graph (the node count in Figures 1 and 2 is significantly higher than in Figures 3 and 4). I am unsure of the reason behind this discrepancy.

2. The number of objects in the Clio online backend results is considerably higher than after running object clustering on the Pre-built Scene Graph (Figure 5 has significantly more nodes than Figure 7).

3. The node count in the Clio online frontend is much lower than the number of objects in the Pre-built Scene Graph, with the frontend count even being zero (Figure 6 has significantly fewer nodes than Figure 8, and in Figure 6 the count is zero). I wonder is this expected behavior? And the evaluation results differ from those in the paper(see attached table).

4. Addition: The generated mesh displays horizontal or vertical stripes, a phenomenon also visible in the mesh images presented in your repo. I am keen to know if you are aware of what causes this issue and how it might be resolved to reconstruct a complete mesh.

Attached:

Observed Node Counts:

Model/Numbers	Segments	Objects
Raw	488	99
D3	488	51
Online Backend	883	70

Online Results Evaluation:

System Configuration:

• OS: Ubuntu 20.04
• GPU: NVIDIA RTX 4080 16GB
• ROS Version: Noetic
• CLIP Model: ViT-L/14
• Dataset: Apartment
• Parameters: Default settings as per the repository

1. Segments and Objects Windows:

   • Segments: Extracted from the DSG's SegmentsLayer.
   • Objects: Extracted from the DSG's ObjectsLayer.

2. Data Sources:

   • Figure 4 and Figure 8: Pre-built Scene Graphs from the GitHub repository (clio_logs/apartment/trial_0/dsg.json).
   • Figure 3 and Figure 7: Results post run_3d_object_ablations.py execution.
   • online backend and frontend: Results after running clio_online, generated in the hydra/output/realsense dir.

I would appreciate any guidance or suggestions you can provide to help resolve these issues. Thank you for your assistance and for your continued contributions to this field.

[nathanhhughes]

Thanks for your interest in our work! Responses to your questions are inlined below!

1. It is evident that the number of segment nodes and corresponding bounding boxes in Clio online is greater than in the Pre-built Scene Graph (the node count in Figures 1 and 2 is significantly higher than in Figures 3 and 4). I am unsure of the reason behind this discrepancy.

Yes, we've also observed this on our side. It appears to be due to changes in Khronos and Hydra when we updated versions when preparing the release for Clio.

2. The number of objects in the Clio online backend results is considerably higher than after running object clustering on the Pre-built Scene Graph (Figure 5 has significantly more nodes than Figure 7).

This is a side-effect of the changes in number and quality of the segments. The underlying graph of segments that gets clustered by IB is more densely connected than in the pre-built scene graphs (and the online scene graphs that generated the paper results), which affects the stop condition and other parameters of Clio. Rather than trying to retune the parameters of Clio, we've been working at isolating the issue affecting the segments (please see below).

3. The node count in the Clio online frontend is much lower than the number of objects in the Pre-built Scene Graph, with the frontend count even being zero (Figure 6 has significantly fewer nodes than Figure 8, and in Figure 6 the count is zero). I wonder is this expected behavior?

Yes, this is expected behavior. The frontend scene graph is not expected to have any objects in it when produced by the online version of Clio (the frontend scene graph is expected to contain the same segments as the backend scene graph, and only the backend scene graph should contain objects).

4. Addition: The generated mesh displays horizontal or vertical stripes, a phenomenon also visible in the mesh images presented in your repo. I am keen to know if you are aware of what causes this issue and how it might be resolved to reconstruct a complete mesh.

This has to do with how the Khronos active window interacts with Hydra's frontend (note that it is unrelated to the issues you've observed with the objects). At a high-level the missing strips in the mesh are due to faces being dropped when integrating the active background mesh from Khronos into the full background mesh of Hydra. We had a way of synchronizing the two in the past that didn't integrate well with the current versions of Hydra and Khronos, and we're working on a longer term fix.

---

Regarding the issues with the object metrics being worse than the results reported in the paper, we've added a link to an archive of the version of the code used to generate the paper here. We're still working on isolating exactly what caused the change in quality of the segments and will update Clio and other dependencies once we've figured out the appropriate fix.

[Laura-Ting]

Hi Nathan,

Thank you for your detailed response! I now understand the reasons behind my questions. I appreciate the link to the archive of the code version used in your paper—this will be very helpful.

Thanks again for your assistance and look forward to any future updates you might have.