Training an EdgeTAM-style SOT model on top of SAM3 weights while keeping the parallel detector — any training recommendations?

[yinnengzhong]

Hi, I'm trying to build a lightweight single-object tracker (SOT) similar to EdgeTAM, initialized from SAM3's pretrained weights. My plan is roughly as follows:

• Primary goal: a SAM2-style tracker for SOT, initialized from the SAM3 checkpoint
• Detector: keep SAM3's DETR-based detector running in parallel for downstream engineering use cases (e.g., triggering re-identification, providing initial boxes for the tracker)
• Text encoder: not a priority. For the limited set of categories I care about, I plan to pre-cache text embeddings offline and skip the language backbone entirely at inference time

Based on this setup, I have a few questions:

1. Tracker fine-tuning strategy: Would you recommend freezing the detector entirely and only fine-tuning the tracker components (memory encoder, memory attention, mask decoder), similar to how SAM2 is
   trained on video datasets? Or does the shared vision backbone create too much coupling to freeze the detector side independently?
2. Vision-only distillation (skipping text distillation) feeding into tracker training: If Stage 1 only does vision feature distillation and skips text distillation entirely, then proceeds directly to
   tracker fine-tuning, is this a reasonable path? Are there any known issues with FPN features flowing into the tracker when the text branch doesn't exist? Can the training pipeline fully follow SAM2's
   recipe?
3. During the video training stage, should the detector also be trained, or should it remain fully frozen?

Looking forward to any recommendations on this geometry-first, tracker-focused training setup. Thanks!