Skip to content

[2810] Allow private config to override output directories and filename - #2811

Open
evenmn wants to merge 7 commits into
ecmwf:developfrom
metno:feat/private-config-output-paths
Open

[2810] Allow private config to override output directories and filename#2811
evenmn wants to merge 7 commits into
ecmwf:developfrom
metno:feat/private-config-output-paths

Conversation

@evenmn

@evenmn evenmn commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Description

Adds optional private_config overrides for model, log, and results directories, plus the output filename. Explicit directories are used directly, without appending the run ID. Existing defaults remain when overrides are omitted.

path_shared_working_dir: /work/weathergen

path_model: /models/deployed
path_logs: /var/log/weathergen
path_results: /forecasts
output_name: forecast.zip

Checkpoint loading and logging use the selected private configuration.

Issue Number

Closes #2810

Is this PR a draft? Mark it as draft.

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

FastEvaluation

  • I have updated the public documentation if necessary

@clessig

clessig commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

@grassesi : could you also have a look?

@clessig

clessig commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

@evenmn : could you lint please :)

@github-actions github-actions Bot added infra Issues related to infrastructure model Related to model training or definition (not generic infra) labels Sep 10, 2026

@clessig clessig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a useful feature. Some minor comments. We also should merge this into the preops branch.

Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/config.py Outdated
Comment thread packages/common/src/weathergen/common/logger.py Outdated
Comment thread src/weathergen/run_train.py Outdated
Comment thread src/weathergen/run_train.py Outdated
Comment thread src/weathergen/run_train.py Outdated
@evenmn

evenmn commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@clessig Thank you for the comments. Please have a look at my answers and latest commits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infra Issues related to infrastructure model Related to model training or definition (not generic infra)

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Custom output file handler

2 participants