add data rank split

[samsja]

What this pr does:

• allow to split locally by data rank for handling local experiments after the new datasets PR
• add a easy to use install script

this graph show the old and the new behavior where the data is not duplicated locally anymore

[Jackmin801]

lgtm. should we make the code download the data instead of the script in a future PR? I think the HF dataset way of using it is nice where you can just specify dataset repo / path

[samsja]

lgtm. should we make the code download the data instead of the script in a future PR? I think the HF dataset way of using it is nice where you can just specify dataset repo / path

yeah I am planning on refactoring the dataset part to do streaming as well from hf repo. So I might as well to the downloading option at the same time