Feature Request: Multi-Object Inference Script

[zakroum-hicham]

Hello,

First of all, thank you for sharing this incredible work on SAMURAI! It’s an impressive adaptation of the Segment Anything Model (SAM) for zero-shot visual tracking, and I’m excited to explore its capabilities further.

I’m reaching out to request a feature that would be incredibly useful for many users: a multi-object inference script. Currently, the repository provides excellent support for single-object tracking, but many real-world applications require tracking multiple objects simultaneously in the same video.

Questions

• Is multi-object tracking something that could be supported by the current architecture of SAMURAI?
• If so, would you consider adding a multi-object inference script to the repository?
• If this is outside the scope of the current project, could you provide guidance on how to modify the existing code to support multi-object tracking?

I’d be happy to assist with testing or even contributing to the implementation if needed. Thank you again for your work on this project, and I look forward to hearing your thoughts!

[yangchris11]

You can check following issues #3 #61 #101.

A multi-object inference script will require some engineering effort to be made on top of any sam-based tracking/segmentation framework including:
(1) Tracklet/Masklet management (e.g., handling newborn, disappearing, re-entry of the objects)
(2) Association problem (e.g., many-to-one matching)

One possible reference you can refer to is https://github.com/IDEA-Research/Grounded-SAM-2?tab=readme-ov-file#grounded-sam-2-video-object-tracking-with-continuous-id-with-grounding-dino.