Performance tests on CESNET infra

[guillaumeeb]

In #15, some first tests on using Xarray on CESNET with Swift are proposed. This shows that (thanks to @sebastian-luna-valero), it's quite easy (even with some manual steps) to write on the Swift object storage using Zarr and Dask.

One thing we might want to do know would be to perform some light but comprehensive benchmarks to identify what performances we could get on this Infrastructure.

A classical benchmark could be:

• Define some example datasets: small, medium, large (up to some TiB?).
  • 10GiB
  • 100GiB
  • 1TiB
• Write these datasets on Dask clusters varying in size:
  • 5 workers
  • 10 workers
  • 20 workers
• Read back the datasets on varying Dask clusters
• Compute things like troughput or other stats (maybe by analyzing the Dask task report)
• Analyze the results.

We might want to start from something like https://github.com/pangeo-data/benchmarking on wihch @tinaok contributed.

We need also to ask CESNET for potential limits or constraints they'd have @sebastian-luna-valero.

[sebastian-luna-valero]

Sounds like a good plan to me, thanks!

I am asking CESNET about the potential limits and constraints are will report back.

[sebastian-luna-valero]

According to CESNET benchmark tests can start right away. We have 10TB quota for object storage.

They shared this list of flavors: https://docs.cloud.muni.cz/cloud/flavors/

We are currently using the hpc.16core-64ram-ssd-ephem flavor with an average network throughput of 1250.0 Mbps. If that's not enough, we should switch flavors to one without limits.

[guillaumeeb]

If there is no limit, except the volume, let's try to scale :). We'll see later if we need bigger throughput.

[guillaumeeb]

So I've opened pangeo-data/benchmarking#45 if you want to look at it.

[sebastian-luna-valero]

Should we close this one, or redo it with MinIO?

[guillaumeeb]

I think we can leave it open, although I'm not sure I'll work on it soon... But this would give us some figures of the performance we can reach with our infrastructure!