Repository navigation
Define and implement task for collecting MET stat info in a more usable form #52
Description
Currently, the stats task just produces .stat and .txt files from MET grid_stat and point_stat runs. The plots task works with these files directly, but users with advanced plotting needs will want to access the stats data in a more fluent way. Add a new top-level, public task to perform this.
Some notes:
- netCDF might be a good format; what follows will assume netCDF. Consult with would-be users about design.
- The task for this netCDF file should yield requirements for the stats task(s) that create the output that it will then package.
- It should be possible to augment the netCDF file. For example, if a user runs a range of cycles, then adds a new cycle to the YAML config, the new stats should be produced, then appended to the netCDF. This will require the ready predicate for the task to test file content, not just existence.
- Removing data from the netCDF is out of scope: If the user removes cycles, leadtimes, etc. from the config, the task need not remove data from the netCDF file.
- Consider simultaneously revamping the
plotstask to require this new task and then consume its output: The.stat/.txtparsing code currently used for plotting could then move to the new task, where it would otherwise probably have to be reimplemented anyway. This refactor would also provide a valuable internal use case for the netCDF, helping to ensure that it is properly maintained.
Activity
maddenp-cu commentedon May 7, 2026
We should look at Parquet as an output option, either alongside or in place of netCDF.
maddenp-cu commentedon May 8, 2026
SQLite could be another option: It has clients for every programming language, plus CLI tools to interact with it. It can store its entire database as a single file. It would not have the potential benefit of Parquet in terms of remote querying (you need the SQLite database file locally).
Actually, on the Parquet side, we should investigate to see what capabilities it really provides. The goal, though, would be to make it easy to obtain statistics matching specific criteria as easily as possible. For example, one could fairly easily write an SQL query (which would certainly work with SQLite and, reportedly, with Parquet -- TBD) to obtain the RMSE values for temperature at 1000mb and 500mb for 6-hour leadtimes of all 00Z cycles from March 2026, given an SQL database with the right schema. We have all the information to populate such a database.
A benefit of an SQL-based solution would be easy updates, as it's easy to insert new rows into an existing table at any point. I don't know how difficult that would be with a netCDF-based solution. From a wxvx-workflow perspective, that would be nice because new stat results could be added to the database at any point (maybe requiring no more than a mutex, if needed, for correctness under concurrency), without requiring, say, writing everything at once after all stats are available. The task-readiness criterion could be defined as "Is this row in the table?"
maddenp-cu commentedon May 16, 2026
Consider calculating validtime from inittime and leadtime for each stat and recording it with the stat and its other metadata, so that users who are interested in statistics by validtime can easily query for that.