Fix load_decoder_chkpt silently loading no decoder under FSDP - #2800
Merged
Conversation
load_model normalises the "module." state_dict prefix in both of its branches
(sharded and unsharded). load_decoder_from_checkpoint was written later and
copied only the unsharded half, so its sharded path -- taken whenever
with_ddp and with_fsdp, i.e. every decoder finetune here -- looked each
parameter up in model.state_dict() using the raw checkpoint key.
For a DDP-saved backbone (keys prefixed "module.") against an unprefixed
model state dict, every one of the 131 lookups missed, each was skipped
individually, and load_state_dict({}) succeeded. The run then trained a
randomly initialised decoder. Three things hid it: the per-parameter warning
fired once per rank (thousands of lines), the unexpected-keys report was
empty because nothing was loaded, and "Loaded 0 decoder tensors" was a single
INFO line.
Six completed decoder finetunes off cw6a4szu, plus one off nhv6tkln, trained
random decoders this way; the two that worked (new73gkw, oliqngmf backbones)
did so only because those checkpoints happen to be saved without the prefix.
- Extract _align_module_prefix / _strip_module_prefix and use them in all
four places the prefix logic was duplicated, including the decoder
overlay's sharded path, which is the actual fix.
- _strip_module_prefix removes only a leading prefix; the previous
key.replace("module.", "") stripped the substring anywhere in the path.
- Raise instead of loading nothing: matching N tensors but loading 0 is now
a RuntimeError, as is requesting an overlay from a checkpoint with no
decoder. The condition is identical on every rank, so all ranks raise
together rather than one hanging the others.
- Collapse the per-parameter skip warning into one rank-0 summary and report
loaded/matched counts.
Verified against all four backbone checkpoints: the old raw lookup gives
0/131 for cw6a4szu and nhv6tkln against an unprefixed model (reproducing the
bug) and 99/99 for new73gkw and oliqngmf (reproducing why those worked); with
the fix all four match 100% against both key conventions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsFj2pVh4pxs2UxthxewWy
moritzhauschulz
marked this pull request as ready for review
September 7, 2026 21:33
github-actions
Bot
added
the
model
Collaborator
|
Which branches are affected by this?
…________________________________
From: Moritz Hauschulz ***@***.***>
Sent: Monday, 07 September 2026 22:18:46
To: ecmwf/WeatherGenerator ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [ecmwf/WeatherGenerator] Fix load_decoder_chkpt silently loading no decoder under FSDP (PR #2800)
Description
Decoder modules were not loaded correctly with FSDP – this PR checks for the prefix and normalizes the loaded module names. It also raises an error if the decoder is not loaded – i.e. no weights are loaded for the decoder.
Issue Number
Is this PR a draft? Mark it as draft.
Checklist before asking for review
* I have performed a self-review of my code
* My changes comply with basic sanity checks:
* I have fixed formatting issues with ./scripts/actions.sh lint
* I have run unit tests with ./scripts/actions.sh unit-test
* I have documented my code and I have updated the docstrings.
* I have added unit tests, if relevant
* I have tried my changes with data and code:
* I have run the integration tests with ./scripts/actions.sh integration-test
* (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
* (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
* I have informed and aligned with people impacted by my change:
* for config changes: the MatterMost channels and/or a design doc
* for changes of dependencies: the MatterMost software development channel
FastEvaluation
* I have updated the public documentation if necessary
________________________________
You can view, comment on, or merge this pull request online at:
#2800
Commit Summary
* 32b6ab4<32b6ab4> Fix load_decoder_chkpt silently loading no decoder under FSDP
File Changes
(2 files<https://github.com/ecmwf/WeatherGenerator/pull/2800/files>)
* M src/weathergen/model/model_interface.py<https://github.com/ecmwf/WeatherGenerator/pull/2800/files#diff-2e85cd26b59453d8cbf3b54ed76e4cf092f911b8063dde53addf8ea81f0ca28d> (116)
* A src/weathergen/model/model_interface_test.py<https://github.com/ecmwf/WeatherGenerator/pull/2800/files#diff-18d25409d0e8eaa37287a9053878245d5bdb7f794aca45476602bd2cfd59e14e> (121)
Patch Links:
* https://github.com/ecmwf/WeatherGenerator/pull/2800.patch
* https://github.com/ecmwf/WeatherGenerator/pull/2800.diff
—
Reply to this email directly, view it on GitHub<#2800?email_source=notifications&email_token=AHCHOHTC5YFDKBHHSKXRJSD5N4Q3NA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DINRXHAYTGMZTG6THEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVRTG633UMVZF6Y3MNFRWW>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHCHOHT7DUSFPT7QZLH6B3D5N4Q3NAVCNFSNUABFKJSXA33TNF2G64TZHM4TCMRXGI3TSMZQHNEXG43VMU5TKMZXHEYTANJWHE3KC5QC>.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AHCHOHQPC3FLIJHRZPLLOSL5N4Q3NA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DINRXHAYTGMZTG6THEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVJTG633UMVZF62LPOM> and Android<https://github.com/notifications/mobile/android/AHCHOHUTCZI73L6WDKZX6635N4Q3NA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DINRXHAYTGMZTG6THEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE>. Download it today!
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
MatKbauer
approved these changes
Sep 8, 2026
MatKbauer
merged commit Sep 8, 2026
980f640
into
ecmwf:mk/jk/kt/dev-ssl-diff/omit-target-writing
3 checks passed
Contributor
Author
|
@clessig @MatKbauer I think it might affect both diffusion and ssl branches, and potentially others. For example, the relevant decoder loading function exists here but the prefix is not aligned in the fsdp case. I am not sure which ssl branch is currently the main one. In principle, any branch that has this commit will need the fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Decoder modules were not loaded correctly with FSDP – this PR checks for the prefix and normalizes the loaded module names. It also raises an error if the decoder is not loaded – i.e. no weights are loaded for the decoder.
Issue Number
Is this PR a draft? Mark it as draft.
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60FastEvaluation