Pre-compute STAC Item Aggregations for Timeline UI

[A-Behairi]

Background & Context

See EOPF-Explorer/landing-page#61

To resolve this, we need to pre-compute item counts at the end of the ingestion process and output them as static JSON files for the application to fetch.

Key Tasks

The pipeline needs to ideally generate two JSON Objects per STAC Collection:

• Daily aggregation
• Monthly aggregation (optional)
  • This will help the user get a faster initial load in case of bigger collections

I suggest the structure of the file to mimic a datetime range Aggregation Collection response based on the Aggregation Extension to maintain STAC compliance, for e.g:

 {   
        "type": "AggregationCollection",
        "aggregations": [
            {
                "key": "datetime_daily",
                "buckets": [
                    { 
                        "key": "2026-01-01T00:00:00.000Z",
                        "value" : "8644819",
                    },
                    {
                        "key": "2026-01-02T00:00:00.000Z",
                        "value" : "5644819",
                    },
                    {
                        "key": "2026-01-03T00:00:00.000Z",
                        "value" : "7644819"
                    }
                ],
                "interval": "daily",
            }
        ]
    }

STAC Integration

I am unsure about what the best solution for this might be, considering STAC conformance and the effort required, here are a couple of suggestions that I can think of:

1. /aggregate Endpoint
   To have it under /aggregate with collection and data_type parameters, for example https://api.explorer.eopf.copernicus.com/stac/aggregations?collection="sentinel-2-l2a"&data_type="interval_daily"

2. Aggregation Links
   Have links with rel: "aggregate" and indicate the interval using an aggregation: prefixed property for e.g.:

{
      "rel": "aggregate",
      "href": "https://example.com/timeline-daily.json",
      "type": "application/json",
      "title": "Daily Item Aggregation",
      "aggregation:interval": "daily" 
}

or with the interval indicated in the rel property

[m-mohr]

Ideally make sure it doesn't conflict with https://github.com/stac-api-extensions/aggregation (e.g. using different relation types, endpoint names)

[lhoupert]

I think creating a standalone CLI script could be a good solution. It could run daily in its own cronworfklow and generate daily and monthly json?

I could create these new links:

 {
   "rel": "pre-aggregatation",
   "href": "https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/aggregations/sentinel-2-l2a/daily.json",
   "type": "application/json",
   "title": "Daily Item Aggregation",
   "aggregation:interval": "daily"
 },
 {
   "rel": "pre-aggregatation",
   "href": "https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/aggregations/sentinel-2-l2a/monthly.json",
   "type": "application/json",
   "title": "Monthly Item Aggregation",
   "aggregation:interval": "monthly"
 }

What do you think? @A-Behairi @emmanuelmathot

[A-Behairi]

Yes, that would be great 👍🏾

[lhoupert]

I thought maybe changing the relation "aggregate" to "pre-aggregatation" to make sure there is no ambiguity with the stac extension relation as suggested by @m-mohr

[lhoupert]

@A-Behairi I ran the tool created in #106 on dev-staging collection, you can access the aggregation with this command:

curl -s https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/aggregations/sentinel-2-l2a-staging/daily.json | python3 -m json.tool 
curl -s https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/aggregations/sentinel-2-l2a-staging/monthly.json | python3 -m json.tool 

Let me know if you are happy with this implementation and if it is solving your issue? I can then run it on the sentinel-2-l2a collection and can create an argo worfklow that run this tool daily to update the json

[A-Behairi]

It looks great! Thank you for the quick implementation!

[lhoupert]

I ran the tool on the sentinel-2-l2a collection, you can see the two aggregation links on the stac collection page (https://api.explorer.eopf.copernicus.eu/browser/external/api.explorer.eopf.copernicus.eu/stac/collections/sentinel-2-l2a) .

I will now start working on an Argo Worflow to run this daily.

[lhoupert]

The changes will be applied to prod once, this PR is merged https://github.com/EOPF-Explorer/platform-deploy/pull/123

[lhoupert]

I just realised the CronWorkflow running on dev-seed namespace (on the sentinel2 collection) kept failing because of a OutOfMemory error. I raised a fix here https://github.com/EOPF-Explorer/platform-deploy/pull/125. Will check that the aggregation is working as expected once the fix is merged.

[lhoupert]

@A-Behairi It seems to be working fine now! (see https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/aggregations/sentinel-2-l2a/daily.json)