Support 24h averaging with anemoi datasets

[MatKbauer]

Describe the task. It can be a feature, documentation, etc.

Currently, and also described in #1699, we currently cannot pass keyword arguments to the initialisation of anemoi datasets. (Except for frequency, which is the only supported and hard-coded argument).

To support 24h avarage calculation instead of taking instantaneous time stamps, we could pass rolling_average= [-4, 0, 'frequency’] to the anemoi dataset init.

We can introduce a new indentation section for anemoi dataset keywords, such as

ERA5:
  type: anemoi
  anemoi_arguments:
    frequency: 06:00:00
    rolling_average: [-4, 0, "frequency"]

Hedgedoc URL, if you are keeping notes, plots, logs in hedgedoc.

No response

Area

• datasets, data readers, data preparation and transfer
• model
• science
• infrastructure and engineering
• evaluation, export and visualization
• documentation
• performance

[clessig]

Sounds good. We should call it dataset_arguments or similar and then it can also be used for other data readers more naturally.

[vhertel]

@MatKbauer: @evenmn confirmed that this has been implemented in #2519. For consistency, would it make sense to put frequency in there and remove it as a top level key in the config?

[clessig]

@MatKbauer: @evenmn confirmed that this has been implemented in #2519. For consistency, would it make sense to put frequency in there and remove it as a top level key in the config?

Let's keep the frequency parameter as is. It's useful for most datasets.

[vhertel]

@clessig @MatKbauer sounds good, then I think we can close the issue :)

[MatKbauer]

Excellent, thanks for verifying, @vhertel. Closing this.

[clessig]

Excellent, thanks for verifying, @vhertel. Closing this.

@vhertel @MatKbauer : could we document the exact syntax for this functionality somewhere.

[vhertel]

@vhertel @MatKbauer : could we document the exact syntax for this functionality somewhere.

@clessig yes, I can add it to the streams section of Cristian's note on config files. Maybe we can add this to the repo similar to the FastEval documentation

[evenmn]

@clessig @vhertel I think it would be useful to collect documentation of the most relevant Anemoi dataset functionality for WeatherGenerator, including rolling_average, trim_edge, join, and area, in a common document

[clessig]

@clessig @vhertel I think it would be useful to collect documentation of the most relevant Anemoi dataset functionality for WeatherGenerator, including rolling_average, trim_edge, join, and area, in a common document

Agreed. Maybe we can use the issue to collect this. But it ultimately needs to go into a proper documentation of the config options ...