We analyse, for the first time, the popular loop closing module of a well known and widely used open-source visual SLAM (ORB-SLAM) pipeline. Investigating failures in the loop closure module of visual SLAM is challenging since it consists of multiple building blocks. Our meticulous investigations have revealed few interesting findings. Contrary to reported results, ORB-SLAM frequently misses large fraction of loop closures on public (KITTI, TUM RGB-D) datasets. One common assumption is, in such scenarios, the visual place recognition (vPR) block of the loop closure module is unable to find a suitable match due to extreme conditions (dynamic scene, viewpoint/scale changes). We report that native vPR of ORB-SLAM is not the sole reason for these failures. Although recent deep vPR alternatives achieve impressive matching performance, replacing native vPR with these deep alternatives will only partially improve loop closure performance of visual SLAM.
Our findings suggest that the problem lies with the subsequent relative pose estimation module between the matching pair. Surprisingly, using off-the-shelf SIFT based relative pose estimation (non real-time) manages to close most of the loops missed by ORB-SLAM. This significant performance gap between the two available methods suggests that ORB-SLAM's pipeline can be further matured by focussing on the relative pose estimators, to improve loop closure performance, rather than investing more resources on improving vPR. We also evaluate deep alternatives for relative pose estimation in the context of loop closures. Interestingly, the performance of deep relocalization methods (e.g. MapNet) is worse than classic methods even in loop closures scenarios. This finding further supports the fundamental limitation of deep relocalization methods recently diagnosed.
Finally, we expose the bias in the well known public dataset (KITTI) due to which these commonly occurring failures have alluded the community. We augment the KITTI dataset with detailed loop closing labels. In order to compentsate for the bias in the public datasets, we provide a challenging loop closure dataset which contains challenging yet commonly occurring indoor navigation scenarios with loop closures. We hope our findings and the accompanying dataset will help the community in further improving the popular ORB-SLAM's pipeline.
Load the .mat files (e.g. scene-graph-kitti-00) to your MATLAB (or whatever you want) workspace. The data would be loaded to variable 'SG_00'. Values at locations e.g. SG_00(4,5), would indicate number of SIFT feature matches surviving RANSAC between images 4, and 5 when the sequence is sampled at 10 fps. For other sequences, only the sequence number changes e.g. SG_00 changes to SG_02 for sequence KITTI-02.
ORB-SLAM.Kitti.mp4
TUM-RGBD.mp4
NUST.CLC.-.Dynamic.Scene.mp4
NUST.CLC.Viewpoint.Change.mp4
Email for Dataset, if following link is not working [email protected] [email protected] [email protected]
Click here to download Dynamice Sequences. To run with ROS, rosbags are also available.
Click here to download Viewpoint Sequences. To run with ROS, rosbags are also available.
Geometric.Check.mp4
ORB-SLAM3.Indoor.mp4
ORB-SLAM3.outdoor.mp4
Loop closure and better alignment with ground truth with our approach is evident.
