Project 3D mesh on 2D image using camera intrinsic matrix

[RafayAK]

I've been trying to use the HOnnotate dataset to extract perspective correct hand and object masks as shown in the images of Task-3 of the Hands-2019 challenge.

The data set comes with the following annotations:

annotations:
	The annotations are provided in pickled files under meta folder for each sequence. The pickle files in the training data contain a dictionary with the following keys:
	objTrans: A 3x1 vector representing object translation
	objRot: A 3x1 vector representing object rotation in axis-angle representation
	handPose: A 48x1 vector represeting the 3D rotation of the 16 hand joints including the root joint in axis-angle representation. The ordering of the joints follow the MANO model convention (see joint_order.png) and can be directly fed to MANO model.
	handTrans: A 3x1 vector representing the hand translation
	handBeta: A 10x1 vector representing the MANO hand shape parameters
	handJoints3D: A 21x3 matrix representing the 21 3D hand joint locations
	objCorners3D: A 8x3 matrix representing the 3D bounding box corners of the object
	objCorners3DRest: A 8x3 matrix representing the 3D bounding box corners of the object before applying the transormation
	objName: Name of the object as given in YCB dataset
	objLabel: Object label as given in YCB dataset
	camMat: Intrinsic camera parameters
	handVertContact: A 778D boolean vector whose each element represents whether the corresponding MANO vertex is in contact with the object. A MANO vertex is in contact if its distance to the object surface is <4mm
	handVertDist: A 778D float vector representing the distance of MANO vertices to the object surface.
	handVertIntersec: A 778D boolean vector specifying if the MANO vertices are inside the object surface.
	handVertObjSurfProj: A 778x3 matrix representing the projection of MANO vertices on the object surface.

It also comes with a visualization script (https://github.com/shreyashampali/ho3d) that can render the annotations as 3D meshes (using Open3D) or 2D projects of on object corners and hand points (using Matplotlib):

What I am trying to do is project the visualization created by Open3D back to the original image.

So far I have not been able to do this. What I have been able to do is get the point cloud from 3d mesh and apply the camera intrinsic on it to make it perspective correct, now the question is how to create a mask out of the point-cloud for both hands and objects like the one from Open3d rendering.

# code looks as follows
# "mesh" is an Open3D triangle mesh ie "open3d.geometry.TriangleMesh()" 
pcd = open3d.geometry.PointCloud()
pcd.points = mesh.vertices
pcd.colors = mesh.vertex_colors
pcd.normals = mesh.vertex_normals

pts3D = np.asarray(pcd.points)
# hand/object along negative z-axis so need to correct perspective when plotting using OpenCV
cord_change_mat = np.array([[1., 0., 0.], [0, -1., 0.], [0., 0., -1.]], dtype=np.float32)
pts3D = pts3D.dot(cord_change_mat.T)

# "anno['camMat']" is camera intrinsic matrix 
img_points, _ = cv2.projectPoints(pts3D, (0, 0, 0), (0, 0, 0), anno['camMat'], np.zeros(4, dtype='float32'))

# draw perspective correct point cloud back on the image
for point in img_points:
    p1, p2 = int(point[0][0]), int(point[0][1])
    img[p2, p1] = (255, 255, 255)

White dots are of the point cloud of the object, red dots are of the point cloud of the hand

Basically, I'm trying to get this segmentation mask out:

PS. Sorry if this doesn't make much sense, I'm very much new to 3D meshes, point clouds and their projections. I don't know all the correct technical words from them, yet. Leave a comment with a question and I can try to explain it as far as I can.

[RafayAK]

solved here: https://stackoverflow.com/questions/70273002/project-3d-mesh-on-2d-image-using-camera-intrinsic-matrix