There have been a couple failing tests for about a week now:
FAILED virtualizarr/tests/test_backend.py::TestReadFromURL::test_virtualizarr_vs_local_nisar[HDF5VirtualBackend] - fsspec.exceptions.FSTimeoutError
FAILED virtualizarr/tests/test_backend.py::TestReadFromURL::test_virtualizarr_vs_local_nisar[HDFVirtualBackend] - fsspec.exceptions.FSTimeoutError
I'm not yet sure what the cause of these failures are, but they're a bit frustratingly slow to debug because creating a local cache of the file takes a really long time (it's possible this is actually the cause of the failure, based on the timeout error). For example, my regular download speeds are >150 Mbps but the file used in test_virtualizarr_vs_local_nisar downloads at ~.3 MB/s (~2.5 Mbps) (e.g., screenshow below). Unless we implement a persistent local cache in GitHub actions, this will also slow down our CI. Possibly we could look into signed HTTPS or other mechanisms for faster downloads from ASF, but I'm wondering if we should host our own test files on Open Storage Network or a different cloud provider for more control.
I could apply for an OSN allocation for VirtualiZarr and set it up if it's something we're interested in. Or we'd probably fit in the free limits of cloudfare R2.
There have been a couple failing tests for about a week now:
I'm not yet sure what the cause of these failures are, but they're a bit frustratingly slow to debug because creating a local cache of the file takes a really long time (it's possible this is actually the cause of the failure, based on the timeout error). For example, my regular download speeds are >150 Mbps but the file used in test_virtualizarr_vs_local_nisar downloads at ~.3 MB/s (~2.5 Mbps) (e.g., screenshow below). Unless we implement a persistent local cache in GitHub actions, this will also slow down our CI. Possibly we could look into signed HTTPS or other mechanisms for faster downloads from ASF, but I'm wondering if we should host our own test files on Open Storage Network or a different cloud provider for more control.
I could apply for an OSN allocation for VirtualiZarr and set it up if it's something we're interested in. Or we'd probably fit in the free limits of cloudfare R2.
