slow for "python utils/merge_and_get_clean.py"

[fikry102]

python utils/merge_and_get_clean.py

the process is really slow even if I only process chunkwise part:

cw_src_all = [os.path.join(BASE, f"ODE6KCausal_chunkwise_{i}") for i in range(15)]
cw_dst = os.path.join(BASE, "clean_data")
cw_merged = merge_many(cw_src_all, cw_dst)
rm_dirs(cw_merged, desc="remove shards")

[zhuhz22]

This is quite normal, since we provide a relatively large amount of data. Anything within ten minutes is normal.

Or you can also let claude code optimize it by itself.

[fikry102]

This is quite normal, since we provide a relatively large amount of data. Anything within ten minutes is normal.

Or you can also let claude code optimize it by itself.

How long does it take for chunkwise part?

[zhuhz22]

In my tests, chunkwise and framewise together take about 8 minutes. If yours takes significantly longer, there may be other CPU processes competing for CPU resources.

[fikry102]

It seems my process is stucked. This is strange.

[zhuhz22]

That's indeed strange. I’ve attached a screenshot showing that it worked on my side:

Could it be that other processes are competing for CPU resources on your machine?

[fikry102]

That's indeed strange. I’ve attached a screenshot showing that it worked on my side:

Could it be that other processes are competing for CPU resources on your machine?

I think there is no other big process. The cpu is free.

(self_forcing) ➜ Self-Forcing git:(main) ✗ free -h
total used free shared buff/cache available
Mem: 640Gi 4.3Gi 275Gi 1.0Mi 360Gi 635Gi
Swap: 0B 0B 0B
(self_forcing) ➜ Self-Forcing git:(main) ✗

[fikry102]

Is this realted to lmdb library version?

[zhuhz22]

Maybe, which version do you install? Maybe switch to the version in our requirements.md?

[fikry102]

lmdb 2.2.0
The requirements.txt has no specific version.

[zhuhz22]

Oh sry, our lmdb is 1.7.5.

[fikry102]

Thanks, I installed lmdb==1.7.5. It is stucked at 4/15, lol

[zhuhz22]

IoIThat's really strange, I've never been reported such phenomenon. Since under both version it got stuck, I don't think the lmdb version matters. Maybe you can wait for an hour and see whether it's still stuck?

[fikry102]

Ok, thanks.
But last time it is stucked at 3/15. 4/15 this time.
It seems really strange, lol...

Should I adjust these parameters?

BATCH = 512
MAP_MULT = 2.2

[zhuhz22]

I think the point where it gets stuck is caused by randomness. You can try reducing the batch size.
Maybe this setting:

BATCH = 64
MAP_MULT = 1.3