[Feature-Request] {Frame sync for OAK-T}

[stephansturges]

Start with the why:

One of the most interesting and valuable aspects of the new OAK-T is the ability to collect data from the RGB and thermal camera overlapping in the same scene, allowing joint identification from one sensing modality to the other.... however this requires knowing that the two frames from each stream are in sync, at least to a certain degree of precision. For most applications this might mean <1/25s, or the max refresh rate of the thermal camera.

Move to the what:

I would like to have a way to use software sync an estimation of "how far apart in time" the frames from the thermal and RGB sensor are, so that I can save a frame every X seconds which is close to being perfectly in sync.

Move to the how:

There was a previous implementation using timestamp sync-ing here which appears close to what I want to achieve (since I believe the thermal camera is incapable of using hardware frame sync). I have made a minimal implementation of this (attached) getting the two streams from the OAK-T, however it fails to get any sync-frames after just a few frames presumably because the two sensors are so far out of sync due to their native frame rates that they never catch up again with each other.

OAKT_thermal_sync.zip

[moratom]

@stephansturges we have a Sync node you can try to use.

Here is an example https://github.com/luxonis/depthai-core/blob/main/examples/Sync/depth_video_synced.cpp

Would that work for you?