Mismatch between Frame Count and Depth Data Batch Dimension

[jamesjin0516]

Hi authors, thanks for this new diverse SLAM dataset! I was downloading some validation sequences with intention to reconstruct a scene point cloud using the depth and camera pose (ie. gt_trajectory.txt) information. However, I found that the depth data from its .h5 files have a different batch dimension than that of the camera pose data, the latter of which is consistent with the number of frames in the user and GT view videos.

For example, for sequence new_indoor_close_14 validation's 000003.tar:

• GT and user videos have 5206 frames, and gt_trajectory.txt has shape: (5206, 8)
• depth_frame_0.h5 and confidence_frame_0.h5's data has shape: (4841, 135, 240)

Also for sequence new_scanning_54 validation's 000021.tar:

• GT and user videos have 3480 frames, and gt_trajectory.txt has shape: (3480, 8)
• depth_frame_0.h5 and confidence_frame_0.h5's data has shape: (3941, 135, 240)

For this sequence I noticed that the first 460 frames of confidence and depth data are all zeros.

I wasn't able to find information about the mapping between depth data and video frames in the documentation. Could the proper utilization of depth be clarified? Thanks.

[karhankayan]

Hi, thanks for your interest in our dataset.

This is unfortunately expected since the ZED X Stereo camera drops frames when you also try to get IMU info from it. So, the frame rate of the depth is not exactly 60 fps in contrast to the monocular video because certain frames drop. I don't think the dropping is uniform, so the mapping between depth and monocular video is not straightforward. However,

1. The first and last frames of depth and monocular are aligned.
2. ZED X provides timestamp information, so you can infer exactly which frames where dropped.

So, given the timestamp information, you should be able to get the mapping between monocular and depth.

@stamatisalex can you provide this timestamp information?

We will provide files for the mapping itself at some point in the near future, but timestamps should work in the meantime.

[jamesjin0516]

Thanks for clarifying the misalignment. I have actually seen a file with stereo camera's timestamp information mentioned in the Hugging Face page (sequence_id.*_stereo_mapping.json). However, that file is a pretty cryptic dictionary with a lot of keys like "1": "1739939278305718000.jpg". Would I need some extra information to make use of the data in that file? Or is the timestamp that you referred to something else entirely?

Yep I think once I have the timestamp I will be able to find the right pose for each depth frame. Also just to confirm, the depth is intentionally stored at a lower "resolution" (ie. only 240x135) right?