Parallelize data loading tasks

[anjalijain22]

Hello,

I am using the recently released hail backend to load data. I currently have a pipeline already running for samples in Project A and I'd like to simultaneously run another pipeline for loading data for Project B. However, I get the below error message:

Is there a way to parallelize data loading tasks?

Thanks,
Anjali

[hanars]

Hi Anjali, it is not possible to run the pipeline simultaneously for multiple projects. Due to the underlying way data is snapshotted and updated, concurrent runs would only result in one project loading successfully while the other would not end up in the final data tables