Skip to content

[2810] Allow private config to override output directories and filename - #2811

Merged
clessig merged 13 commits into
ecmwf:developfrom
metno:feat/private-config-output-paths
Sep 13, 2026
Merged

[2810] Allow private config to override output directories and filename#2811
clessig merged 13 commits into
ecmwf:developfrom
metno:feat/private-config-output-paths

Conversation

@evenmn

@evenmn evenmn commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Adds optional private_config overrides for model, log, and results directories, plus the output filename. Explicit directories are used directly, without appending the run ID. Existing defaults remain when overrides are omitted.

path_shared_working_dir: /work/weathergen

path_model: /models/deployed
path_logs: /var/log/weathergen
path_results: /forecasts
output_name: forecast.zip

When multiple samples are generated, rank can be used to separate the different samples:

output_name: "output_rank{rank:04d}.zip"

Other variables that can be used in the filename are epoch and step.

Checkpoint loading and logging use the selected private configuration.

Issue Number

Closes #2810

Is this PR a draft? Mark it as draft.

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

FastEvaluation

  • I have updated the public documentation if necessary

@clessig
clessig requested a review from grassesi September 10, 2026 20:27
@clessig

clessig commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

@grassesi : could you also have a look?

@clessig

clessig commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

@evenmn : could you lint please :)

@github-actions github-actions Bot added infra Issues related to infrastructure model Related to model training or definition (not generic infra) labels Sep 10, 2026
clessig
clessig reviewed Sep 11, 2026
View reviewed changes

@clessig clessig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a useful feature. Some minor comments. We also should merge this into the preops branch.

Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/logger.py Outdated
Comment thread src/weathergen/run_train.py Outdated
Comment thread src/weathergen/run_train.py Outdated
Comment thread src/weathergen/run_train.py Outdated
@evenmn

evenmn commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@clessig Thank you for the comments. Please have a look at my answers and latest commits

clessig
clessig reviewed Sep 12, 2026
View reviewed changes

@clessig clessig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! Some minor comments left. For the functions we anyway touch, we should be clear if config is "model config" or "private config". It makes the code much more readable.

Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
def get_path_run(config: Config) -> Path:
"""Get the current runs results_path for storing run results and logs."""
return _get_shared_wg_path() / "results" / get_run_id_from_config(config)
return _get_path_output(config, "path_results", "results", get_run_id_from_config(config))

@clessig clessig Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function doesn't make sense--we don't the path for a run. Can you remove it. The function is used in trainer.py and in train_logger.py. It's calls should be replace by calls to get_path_logs()--but please check that you agree.

@evenmn evenmn Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, get_path_run was replaced by get_path_results

Comment thread packages/common/src/weathergen/common/config.py
Comment thread packages/common/src/weathergen/common/config.py Outdated
@github-actions github-actions Bot added the eval anything related to the model evaluation pipeline label Sep 12, 2026
evenmn
evenmn commented Sep 12, 2026
View reviewed changes

@evenmn evenmn left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have verified that the logs and models are written to the paths specified in private_config during training, and that the results are written to the expected file during inference

Comment thread packages/common/src/weathergen/common/config.py
Comment thread packages/common/src/weathergen/common/config.py
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
def get_path_run(config: Config) -> Path:
"""Get the current runs results_path for storing run results and logs."""
return _get_shared_wg_path() / "results" / get_run_id_from_config(config)
return _get_path_output(config, "path_results", "results", get_run_id_from_config(config))

@evenmn evenmn Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, get_path_run was replaced by get_path_results

clessig
clessig approved these changes Sep 13, 2026
View reviewed changes

@clessig clessig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. Looks great now--approving.

Can you open the PR also against the preops branch.

@clessig
clessig merged commit f730876 into ecmwf:develop Sep 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@clessig clessig clessig approved these changes
@grassesi grassesi Awaiting requested review from grassesi

Assignees

No one assigned

Labels

eval anything related to the model evaluation pipeline infra Issues related to infrastructure model Related to model training or definition (not generic infra)

Projects

WeatherGen-dev
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Custom output file handler

2 participants

@evenmn @clessig