Unsatisfactory performance

[xiao10ma]

It seems not good to inference on the autonomous driving scene with Grounded-SAM-2 Video Object Tracking with Continuous ID (with Grounding DINO) or reverse tracking. Is using the API the only solution?

The prompt: 'car. suv. bus.'

nusc1.mp4

As we can see, the model ignores the black suv :(

[rentainhe]

Hello, it seems like the file was broken, maybe you can try to use the stronger Grounding DINO 1.5 or DINO-X model for better performance.

[xiao10ma]

Here's two adjacent frames, i think may be it's not hard to track?

And i use DINO-X to track after you update the API usage to V2(btw, thank you for your update!), it didn't get any tracks:

---

(sam2) (base) mazipei@nvidia:/HDD_DISK/users/mazipei/Grounded-SAM-2$ python grounded_sam2_tracking_demo_custom_video_input_dinox.py
VideoInfo(width=1600, height=900, fps=10, total_frames=31)
Saving Video Frames: 31it [00:00, 124.58it/s]
frame loading (JPEG): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:01<00:00, 29.97it/s]
[]
[]
Traceback (most recent call last):
  File "/HDD_DISK/users/mazipei/Grounded-SAM-2/grounded_sam2_tracking_demo_custom_video_input_dinox.py", line 147, in <module>
    masks, scores, logits = image_predictor.predict(
  File "/HDD_DISK/users/mazipei/Grounded-SAM-2/sam2/sam2_image_predictor.py", line 291, in predict
    masks, iou_predictions, low_res_masks = self._predict(
  File "/HDD_DISK/users/mazipei/conda_envs/sam2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/HDD_DISK/users/mazipei/Grounded-SAM-2/sam2/sam2_image_predictor.py", line 420, in _predict
    low_res_masks, iou_predictions, _, _ = self.model.sam_mask_decoder(
  File "/HDD_DISK/users/mazipei/conda_envs/sam2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/HDD_DISK/users/mazipei/conda_envs/sam2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/HDD_DISK/users/mazipei/Grounded-SAM-2/sam2/modeling/sam/mask_decoder.py", line 136, in forward
    masks, iou_pred, mask_tokens_out, object_score_logits = self.predict_masks(
  File "/HDD_DISK/users/mazipei/Grounded-SAM-2/sam2/modeling/sam/mask_decoder.py", line 203, in predict_masks
    assert image_embeddings.shape[0] == tokens.shape[0]
AssertionError

DINO-X is indeed better than the normal model, I tried it for the FRONT camera. But it still failed for the FRONT_LEFT camera as the normal model, as you can see above.