Repository navigation
Use depth image as additional input #46
Description
Thanks for your great project!
The paper mentioned that SAM 3D can optionally take in point map as input. I notice in Inference class it supports pointmap, but there is no guidance on how to format the pointmap input.
In my case, I have a RGB-D image where the depth and RGB image are well aligned with known camera intrinsics. Could you please help me with more guidance on how to process the depth image to acquire the pointmap? Thank you very much!
Activity
gleize commentedon Nov 24, 2025
Hi @xuxw98,
Regarding the exact pointmap shape and format, you can set a breakpoint here, run the
demo.pyfile and figure out what shape / format it is from there. A pointmap encodes the x, y, z coordinates for each pixel, but I do not remember the exact dimension order.Regarding the "depth-to-pointmap" conversion, as you mentioned, you would need the camera depth intrinsics to unproject each pixel in 3D (i.e. convert pixel (i, j) + depth (d) to 3D coordinates (x, y, z) per pixel).
xuxw98 commentedon Nov 25, 2025
Thank you for your reply!
One more question is about the coordinate system and scale of the pointmap / pointcloud. After I use camera intrinsics to convert depth image to pointcloud in camera coordinate system, should I do any other operation, like normalizing the pointcloud?
This is not about the format, but about which distribution of pointcloud (x,y,z, and maybe additional channels) SAM 3D is trained on.
xuxw98 commentedon Nov 25, 2025
I notice in the InferencePipelinePointMap method, even pointmap is not provided, a depth estimator will be adopted for depth estimation followed by intrinsics inferring. So I can replace the estimated depth and intrinsics with the real ones, is that right? Thank you!
eternity123-null commentedon Nov 25, 2025
Hi @xuxw98, I also noticed that. So if I replace the estimated intrinsics with the real one and also input depth from my RGBD, does the "translation" and "rotation" in output denote the 6D pose of object? Thank you!
alexsax commentedon Nov 25, 2025
@eternity123-null Yes the translation and rotation correspond to the 6D pose of the object, using Pytorch3d camera conventions
alexsax commentedon Dec 3, 2025
@xuxw98
Sorry that I missed the main point of the issue you raised last week. To see how to provide your own pointmap/depth, you could look at the notebook @codepk37 contributed in #79