Currently, we run all ML on the Compute pool, but as slow models can run over multiple cycles, we ideally want them to be executed from the AsyncCompute pool. This in theory also makes sure that the Compute pool keeps low contention and we miss our control cycle deadline (82Hz/12ms) less. However, when we actually make this change we see our cycle time shoot up to almost 30ms in Rerun.
Current ideas:
- It could be that the
AsyncCompute pool is out of threads and these blocking tasks create a giant backlog. Would it make sense to instead spawn them inside the blocking crate's blocking::unblock?
Investigation tips:
sindri run --timings flag to look at profiler output- Task pool thread division can be configured in bevy
Currently, we run all ML on the
Computepool, but as slow models can run over multiple cycles, we ideally want them to be executed from theAsyncComputepool. This in theory also makes sure that theComputepool keeps low contention and we miss our control cycle deadline (82Hz/12ms) less. However, when we actually make this change we see our cycle time shoot up to almost 30ms in Rerun.Current ideas:
AsyncComputepool is out of threads and these blocking tasks create a giant backlog. Would it make sense to instead spawn them inside theblockingcrate'sblocking::unblock?Investigation tips:
sindri run --timingsflag to look at profiler output