Why does preparing step take 128G RAM?

[jocelyn2002]

Is there any solution to take less RAM?

[prenc]

Hi @jocelyn2002,

The issue is that Pandas is very inefficient in handling string columns, and the current implementation of the data split script is very naive. Let me work on that script; hopefully, I will reduce the memory requirement. Also, I am working on a new, more performant version of the ethos tokenization that will be public soon, and it comes with many improvements and bug fixes.

[jocelyn2002]

Thank you so much!

[prenc]

Hey,
It's not done yet :) I'm working on it; what are your memory requirements?

[jocelyn2002]

I've got 4 Nvidia V100 GPUs, and they each have 16GB RAM.

[prenc]

@jocelyn2002 The data preparation step does not require GPUs. I assume the bottleneck is on your node's RAM (not GPUs' memory).

[jocelyn2002]

Sorry, I made a mistake with those two. My RAM is 145GB. In this case, it seems the issue has been resolved..

[prenc]

Okay, you should be able to split it then. Good luck!

[jocelyn2002]

Thanks a lot!