Get transforms from ARkit data

[Vadkoz]

Hi,
I have extrinsic data (quaternions and translation) from ARkit. But if simply pass my quaternions instead COLMAP's, the result is clearly wrong. How do I need to preprocess ARkit data to make it similar to COLMAP before passing it to colmap2nerf script? Or what transformation i need to do instead of preprocessing and colmap2nerf?
I use ios_logger from this repo to get ARkit data.
Thanks!

[dodatko]

same question

[Tom94]

Hi there, the source code in nerf_loader.cu/.h contains the specifics of the coordinate system conversion instant-ngp does to load NeRF data. I do not know specifics about the coodinate system and camera intrinsics ARKit assumes/exposes, but hopefully the above code helps in translating between the two.

[zuixiaosanlang]

same question

[Ir1d]

Hi @Vadkoz are you able to get the axis correct? Can you please share your code?

[Vadkoz]

@Ir1d @zuixiaosanlang
Here are some tricky math terms. Intuitively, when someone talks about changing of basis, everyone thinks that if we want to change the basis from A to B via matrix R, we need to do something like B = R * A, but truth is, the right formula is A = R * B and B = R^(-1) * A
In the first case, we will get the coordinates of the transformed vector in the old basis, in the second case we get the coordinates of NOT transformed vector in a new basis. These two points of view on the R matrix make all difference between COLMAP and ARKit data formats.
Arkit's R matrix is a matrix of changing basis from world to camera coordinates. E.g R: world -> camera. COLMAP vice versa: R: camera -> world
In other words, ARKit uses R as the "basis changer" from world to camera, and i think this approach is quite clear. COLMAP - as a rotation matrix inside the camera basis

So:

X_w - the basis of the camera in world coordinates
X_c - the basis of the camera in camera coordinates 
T_w, T_c - translation vectors in world and camera coordinate system correspondingly
ARKit: X_w = R * X_c + T_w, and camera pose here is just T_w
COLMAP: X_c = R * X_w * A + T_c   =>   X_w = R^(-1) * X_c - R^(-1) * T_c - and camera pose here is -R^(-1) * T_c

Here A is a matrix which is probably unit, but i'm not sure. Will talk about it further.

Since R is rotation martix, R^(-1) = R^t, so with COLMAP we can obtain camera poses as -R.T * T_c

Hence, to map COLMAP data to ARKit format you need: R_arkit = R_colmap.T, T_arkit = -R_colmap.T * T_colmap

BUT!!!
I have done it a while ago and forgot about some problems with R_colmap matrix, specifically with the direction of Z axis. It's a sense of "A" matrix from formulas above. As I remember right, for correct mapping from COLMAP to ARKit data format, you need use this formula instead above one: R_arkit = R_colmap.T * Roll_180, where Roll is just an ordinary roll matrix. If it will not work correct, just try pitch or yaw matrix.

some code:

# colmap_images - its images.txt file from COLMAP's output
# general format - is ARKit data format, in which this formula works X_w = R.T @ X_c - R.T @ T
def colmap2general(colmap_images):
    # X_c = R @ X_w + T <=> X_w = R.T @ X_c - R.T @ T  ->  general format
    Ms = np.zeros((len(colmap_images), 3, 4))
    for n, data in enumerate(colmap_images):
            R = quat2mat(np.array(data[1:5], dtype=float))
            T = np.array(data[5:8], dtype=float)            
        Ms[n, :3, :3] = R.T @ ROLL_180
        Ms[n, :3, 3] = -R.T @ T
    return Ms   # [R | T] (n, 3, 4)

if you can point out my inaccuracies and errors to me, I will only be glad