A reimplementation of this project with new structure and features – thanks!

[Luo-ouL]

Hi there! 👋

First of all, thank you for your amazing work on this project – it's been incredibly helpful and inspiring!

I’ve been working on a reimplementation of the project with a significantly different structure and some added features.
It might be considered a more lightweight version, since the React frontend has been replaced with a PyQt-based UI, and it's built on top of the no-cv-sfm branch.

In case it’s helpful to others, or you're simply curious, here’s the repo:
🔗 https://github.com/Luo-ouL/Low-Cost-Mocap-Reimplement

This is not a request or an issue – just wanted to share it with the community and express my appreciation for your work 🙏

Thanks again and cheers!

[jyjblrd]

I saw your youtube video and was wondering who made it, great stuff!

[See71]

@jyjblrd Here1 and Here2
@Luo-ouL Here1 and Here2
I was hoping to get both your opinions on something。

• During the bundle adjustment, you both commented out the part where the intrinsic matrix gets updated – that’s at Here1.
• Then when computing the reprojection error, you’re still using the original intrinsics – that’s at Here2.
• What was the reasoning behind that?
• Why don’t we update the intrinsics and use the updated ones when computing the reprojection error, as theory would suggest?

[Luo-ouL]

Hi,

When I reimplemented this project, I didn’t pay attention to this comment and kept it as it was. As you pointed out, the intrinsic parameters used for calculating reprojection errors (and final positioning) are different from the ones optimized by the optimizer.

I checked the optimization results: taking one camera as an example, the optimized intrinsic parameters (focal length) became two 499s, while the values used to calculate reprojection errors (and final positioning) were the original 494 and 501. This confirms that the intrinsic parameters changed. However, the final positioning performance remained consistent with my previous experiments and looked fine.

I changed the focal lengths to two 499s, and the positioning results didn’t show much difference. The reason could be that the "change" in focal length is only about 1%. Additionally, performing bundle adjustment again after replacing 494 and 501 with 499s showed that the focal lengths remained almost unchanged after optimization, both being two 499.36s.

After testing, I think that when the intrinsic parameters are relatively accurate, the optimization is quite limited:
First, the change observed from the test is around 1%.
Second, the optimizer only takes the focal length as input without including the principal point coordinates and distortion coefficients, and it also assumes that fx and fy are equal.

If we don't fully trust the intrinsic calibration results and choose to optimize the intrinsic parameters, it might be better to optimize all of them (including focal length, principal point coordinates, and distortion coefficients). Why only optimize the focal length and assume that fx equals fy? So I think the change in focal length observed in the test may not necessarily be an "optimization."

On one hand, the test results indicate that the intrinsic calibration seems relatively accurate. On the other hand, combining focal length and principal point coordinates already gives four variables (not to mention distortion coefficients), while the extrinsic parameters only have six variables in total. The nonlinear optimizer might primarily focus on refining inaccurate initial extrinsic calibration results, and adding too many intrinsic variables might interfere with the optimizer's performance.

I’m not entirely sure about this.

Currently, I tend to fully trust the intrinsic calibration results. I removed the focal length variables from the optimizer’s input and re-tested, and as expected, the positioning results didn’t change significantly. I guess it might have been optimized a bit? Here is the optimized code:

def bundle_adjustment(image_points, camera_poses):
    cameras = Cameras.instance()

    def params_to_camera_poses(params):
        # camera1: as relative/world coordinate
        num_cameras = int(params.size/6)+1
        camera_poses = [{
            "R": np.eye(3),
            "t": np.array([0,0,0], dtype=np.float32)
        }]
        # camera2,3,4...
        for i in range(0, num_cameras-1):
            camera_poses.append({
                "R": Rotation.as_matrix(Rotation.from_rotvec(params[i*6 : i*6+3])),
                "t": params[i*6+3 : i*6+6]
            })

        return camera_poses

    def residual_function(params):
        camera_poses = params_to_camera_poses(params)
        object_points = triangulate_points(image_points, camera_poses)
        errors = calculate_reprojection_errors(image_points, object_points, camera_poses)
        errors = errors.astype(np.float32)

        return errors

    init_params = np.array([])
    for i, camera_pose in enumerate(camera_poses[1:]):
        rot_vec = Rotation.as_rotvec(Rotation.from_matrix(camera_pose["R"])).flatten()
        init_params = np.concatenate([init_params, rot_vec])
        init_params = np.concatenate([init_params, camera_pose["t"].flatten()])

    # modified parameters
    res = optimize.least_squares(
        residual_function, init_params, verbose=2, loss="cauchy", ftol=1E-4
    )

    return params_to_camera_poses(res.x)

Theoretically, using the optimized intrinsic parameters to calculate reprojection errors (and final positioning) should be better, but testing shows that the difference is minimal.
If you hadn’t raised this question, I probably wouldn’t have noticed this. My thoughts might be incorrect, so please take them as a reference.

[See71]

Thank you for answering my questions and providing such a detailed explanation. It really helped me deepen my understanding of the project. I truly appreciate it.

Perhaps the original intention was to account for the limited accuracy of intrinsic calibration at a resolution of 320×240, which led to the inclusion of intrinsic optimisation during bundle adjustment. However, the experiments confirmed that the calibration was quite adequate, leading to the removal of intrinsic optimisation—though perhaps not as thoroughly as it might have been.👍

[See71]

@jyjblrd @Luo-ouL
I would like to seek some more experience regarding coordinate system transformation。🤝
see the details belowCamera coordinate to Three.js coordinate

[See71]

@Luo-ouL @jyjblrd
Regarding the drone part, I would like to ask for your help. Please see the details here.discussions
For me, this is the final step in learning this project.🙏