|
1 | 1 | # Why Xee? |
2 | 2 |
|
3 | | -We noticed two clusters of users working with climate and weather data at |
4 | | -Google Research: Some were [Xarray](https://xarray.dev) (and |
5 | | -[Zarr](https://zarr.dev/)) centric and others, Google Earth Engine centric. Xee |
6 | | -came about as an effort to bring these two groups of developers closer together. |
| 3 | +We noticed two clusters of users working with climate and weather data at Google |
| 4 | +Research: Some were [Xarray](https://xarray.dev) (and [Zarr](https://zarr.dev/)) |
| 5 | +centric and others, Google Earth Engine centric. Xee came about as an effort to |
| 6 | +bring these two groups of developers closer together. |
7 | 7 |
|
8 | 8 | ## Goals |
9 | 9 |
|
10 | 10 | Primary Goals: |
11 | 11 |
|
12 | | -- Make [EE-curated data](https://developers.google.com/earth-engine/datasets) |
13 | | - accessible to users in the Xarray community and to the wider scientific Python |
14 | | - ecosystem. |
15 | | -- Make it trivial to avoid quota limits when computing pixels from Earth Engine. |
16 | | -- Provide an easy way for scientists and ML practitioners to coalesce Earth data |
17 | | - at different scales into a common resolution. |
| 12 | +- Make [EE-curated data](https://developers.google.com/earth-engine/datasets) |
| 13 | + accessible to users in the Xarray community and to the wider scientific |
| 14 | + Python ecosystem. |
| 15 | +- Make it trivial to avoid quota limits when computing pixels from Earth |
| 16 | + Engine. |
| 17 | +- Provide an easy way for scientists and ML practitioners to coalesce Earth |
| 18 | + data at different scales into a common resolution. |
18 | 19 |
|
19 | 20 | Secondary Goals: |
20 | 21 |
|
21 | | -- Provide a succinct interface for querying Earth Engine data at scale (i.e. via |
22 | | - [Xarray-Beam](https://xarray-beam.readthedocs.io/)). |
23 | | -- Make it trivial to quickly [export Earth Engine data to Zarr](https://github.com/google/xee/tree/main/examples#export-earth-engine-imagecollections-to-zarr-with-xarray-beam). |
24 | | -- Provide compelling alternative for the need to export Zarr in the first |
25 | | - place (e.g. during the ML training process). |
| 22 | +- Provide a succinct interface for querying Earth Engine data at scale (i.e. |
| 23 | + via [Xarray-Beam](https://xarray-beam.readthedocs.io/)). |
| 24 | +- Make it trivial to quickly |
| 25 | + [export Earth Engine data to Zarr](https://github.com/google/xee/tree/main/examples#export-earth-engine-imagecollections-to-zarr-with-xarray-beam). |
| 26 | +- Provide compelling alternative for the need to export Zarr in the first |
| 27 | + place (e.g. during the ML training process). |
26 | 28 |
|
27 | 29 | ## Approach |
28 | 30 |
|
29 | | -With the addition of Earth Engine's [Pixel API](https://medium.com/google-earth/pixels-to-the-people-2d3c14a46da6), |
| 31 | +With the addition of Earth Engine's |
| 32 | +[Pixel API](https://medium.com/google-earth/pixels-to-the-people-2d3c14a46da6), |
30 | 33 | it became possible to easily get NumPy array data from `ee.Image`s. In building |
31 | 34 | tools atop of this, we noticed that the best practices for managing data were |
32 | 35 | Xarray-shaped. For example: |
33 | 36 |
|
34 | | -- Our codebases involved many similar LOC to translate between Earth Engine and |
35 | | - arrays: Users typically thought in NumPy and molded EE's Python client to fit |
36 | | - those idioms. |
37 | | -- We often needed to page `computePixel()` requests in a way that's strikingly |
38 | | - similar to Dask/Xarray's concept of [`chunks`](https://docs.xarray.dev/en/stable/user-guide/dask.html#what-is-a-dask-array). |
39 | | -- Users were wrapping NumPy arrays within dataclasses to associate metadata and |
40 | | - labels with data. |
| 37 | +- Our codebases involved many similar LOC to translate between Earth Engine |
| 38 | + and arrays: Users typically thought in NumPy and molded EE's Python client |
| 39 | + to fit those idioms. |
| 40 | +- We often needed to page `computePixel()` requests in a way that's strikingly |
| 41 | + similar to Dask/Xarray's concept of |
| 42 | + [`chunks`](https://docs.xarray.dev/en/stable/user-guide/dask.html#what-is-a-dask-array). |
| 43 | +- Users were wrapping NumPy arrays within dataclasses to associate metadata |
| 44 | + and labels with data. |
41 | 45 |
|
42 | 46 | In an attempt to group these disparate solutions into a singular interface, we |
43 | 47 | experimented with wrapping `computePixels()` into |
44 | | -[Xarray's standard mechanism for defining backends](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html). The result of this effort is Xee. |
45 | | - |
| 48 | +[Xarray's standard mechanism for defining backends](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html). |
| 49 | +The result of this effort is Xee. |
46 | 50 |
|
47 | 51 | ## An array by any other name? (Xee vs Zarr) |
48 | 52 |
|
49 | | -[Zarr](https://zarr.dev/) has been growing in relevance to the world of [cloud-based scientific data](https://doi.org/10.1109/MCSE.2021.3059437). |
50 | | -Members of the open source community have [demonstrated](https://www.youtube.com/watch?v=0bqpxX3Nn_A) |
51 | | -that Zarr is more of a data protocol rather than a data format. In many ways, |
52 | | -Xee is inspired by this work. To this end, we'd like to point out some |
53 | | -similarities and differences between Zarr backed and Earth Engine backed data in |
54 | | -Xarray. |
| 53 | +[Zarr](https://zarr.dev/) has been growing in relevance to the world of |
| 54 | +[cloud-based scientific data](https://doi.org/10.1109/MCSE.2021.3059437). |
| 55 | +Members of the open source community have |
| 56 | +[demonstrated](https://www.youtube.com/watch?v=0bqpxX3Nn_A) that Zarr is more of |
| 57 | +a data protocol rather than a data format. In many ways, Xee is inspired by this |
| 58 | +work. To this end, we'd like to point out some similarities and differences |
| 59 | +between Zarr backed and Earth Engine backed data in Xarray. |
55 | 60 |
|
56 | 61 | Similarities: |
57 | | -- **Xarray-compatible**: Of course, this library proves that both types of data |
58 | | - stores can be compatible with Xarray. [Zarr](https://docs.xarray.dev/en/stable/user-guide/io.html#zarr) |
59 | | - reading and writing is deeply integrated into Xarray as well. |
60 | | -- **Optimal IO Chunks**: Ultimately, cloud-based data stores will inherently |
61 | | - involve networking overhead. There are similarities in the best way to page |
62 | | - data across a network into a local context: the optimal Zarr chunk |
63 | | - size is around [10-100 MBs](https://esipfed.github.io/cloud-computing-cluster/optimization-practices.html#chunk-size). With Earth Engine's backend, the maximum chunk size possible |
64 | | - is 48 MBs. |
| 62 | + |
| 63 | +- **Xarray-compatible**: Of course, this library proves that both types of |
| 64 | + data stores can be compatible with Xarray. |
| 65 | + [Zarr](https://docs.xarray.dev/en/stable/user-guide/io.html#zarr) reading |
| 66 | + and writing is deeply integrated into Xarray as well. |
| 67 | +- **Optimal IO Chunks**: Ultimately, cloud-based data stores will inherently |
| 68 | + involve networking overhead. There are similarities in the best way to page |
| 69 | + data across a network into a local context: the optimal Zarr chunk size is |
| 70 | + around |
| 71 | + [10-100 MBs](https://esipfed.github.io/cloud-computing-cluster/optimization-practices.html#chunk-size). |
| 72 | + With Earth Engine's backend, the maximum chunk size possible is 48 MBs. |
65 | 73 |
|
66 | 74 | Differences: |
67 | | -- **Quota vs No Quota**: Since Earth Engine is API based, there are quota |
68 | | - restrictions that limit IO, namely a 100 QPS limit on data requests. Readers |
69 | | - all need to be authenticated and tied to a GCP project quota. Zarr, on the |
70 | | - other hand, has a lower level access pattern. Reading is delegating to basic |
71 | | - permissions on cloud buckets. |
72 | | -- **On the fly vs up-front data shaping**: In Zarr, the representation of data |
73 | | - at rest fundamentally influences performance at query time. For this reason, |
74 | | - [rechunking](https://xarray-beam.readthedocs.io/en/latest/rechunking.html) and |
75 | | - projecting is a common routine performed up front on Zarr when data does not |
76 | | - quite fit the problem at hand. Earth Engine provides a more flexible interface |
77 | | - than this. Since datasets are pyramided (either at ingestion or server-side), |
78 | | - users are free to request the resolution and projection of the data |
79 | | - during dataset open. Similarly, while Earth Engine's internal dataset |
80 | | - does fit an internal chunking scheme, chunking schemes are a lot more |
81 | | - fungibile. |
82 | | - |
83 | | -We hope that this comparison provides the user of a set of useful precedents |
84 | | -for working with cloud-based datasets. |
85 | 75 |
|
| 76 | +- **Quota vs No Quota**: Since Earth Engine is API based, there are quota |
| 77 | + restrictions that limit IO, namely a 100 QPS limit on data requests. Readers |
| 78 | + all need to be authenticated and tied to a GCP project quota. Zarr, on the |
| 79 | + other hand, has a lower level access pattern. Reading is delegating to basic |
| 80 | + permissions on cloud buckets. |
| 81 | +- **On the fly vs up-front data shaping**: In Zarr, the representation of data |
| 82 | + at rest fundamentally influences performance at query time. For this reason, |
| 83 | + [rechunking](https://xarray-beam.readthedocs.io/en/latest/rechunking.html) |
| 84 | + and projecting is a common routine performed up front on Zarr when data does |
| 85 | + not quite fit the problem at hand. Earth Engine provides a more flexible |
| 86 | + interface than this. Since datasets are pyramided (either at ingestion or |
| 87 | + server-side), users are free to request the resolution and projection of the |
| 88 | + data during dataset open. Similarly, while Earth Engine's internal dataset |
| 89 | + does fit an internal chunking scheme, chunking schemes are a lot more |
| 90 | + fungibile. |
| 91 | + |
| 92 | +We hope that this comparison provides the user of a set of useful precedents for |
| 93 | +working with cloud-based datasets. |
0 commit comments