LightGlue on cases with either 90 or 180 degree rotation

[Abdi-Ghasem]

Thanks for sharing this cool repo! I've been getting great results with image pairs where "up is up." However, I'm curious about cases where there's a 90-degree or even 180-degree rotation. It seems to work when we manually rectify those images before applying the technique. But I'm wondering if there's a way to do this without the manual step.

"up is up" scenario:

"up is not up" scenarios:

[sarlinpe]

Yes this is a known limitation of learned local features (vs SIFT). This does not matter much in practice thanks to a pretty common trick: simply try to rotate the image by {90,180,270}° if it consistently has few matches with other images (assuming that most images in your set are upright). This is not yet part of hloc but could be a nice addition.

[Abdi-Ghasem]

Thanks for the reply. I think adding a training stream to find this rotation automatically or even detect features in different angles followed by concatenating them together would be a temporary solution to this limitation.

[Phil26AT]

Hi @Abdi-Ghasem

The underlying problem is that the descriptors (SuperPoint, DISK) are not rotation-invariant. There have been great works adressing this issue with steerable CNNs in deep matchers, like SE2-LoFTR. The same could be done with LightGlue.

[Abdi-Ghasem]

Hello @Phil26AT,

Thanks for your response and for sharing those helpful links. Since the training code for LightGlue hasn't been released yet, does that mean we can't make use of Steerable CNNs? I'm curious if there are alternative rotation invariant descriptors that can be used with LightGlue. As far as I know, the interface currently only accepts DISK and SuperPoint.

[sunset326]

Hello @Phil26AT,

Thanks for your response and for sharing those helpful links. Since the training code for LightGlue hasn't been released yet, does that mean we can't make use of Steerable CNNs? I'm curious if there are alternative rotation invariant descriptors that can be used with LightGlue. As far as I know, the interface currently only accepts DISK and SuperPoint.

您好@Phil26AT，

感谢您的回复和分享这些有用的链接。由于LightGlue的训练代码尚未发布，这是否意味着我们不能使用可操纵CNN？我很好奇是否有其他旋转不变描述符可以与LightGlue一起使用。据我所知，该接口目前只接受DISK和SuperPoint。

Hi, have you found a method to use rotation invariant descriptors with LightGlue?

[qiao89757]

Hi, have you found a way to use rotation invariant descriptors in LightGlue?Looking forward to your reply.