Conversation
Signed-off-by: evenmn <evenmn@mn.uio.no>
Signed-off-by: evenmn <evenmn@mn.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
…nput. Updated ModelParams and ModelOutput classes accordingly Signed-off-by: evenmn <evenmn@fys.uio.no>
…n with targets.physical Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
…atherGenerator into evenmn/fix/incremental-rollout2
…hunk_size to number of steps, keeping the original behavior Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
… peak memory footprint Signed-off-by: evenmn <evenmn@fys.uio.no>
Conflict resolutions: - engines.py: keep branch's incremental index_copy_ embedding loop; add develop's assert message, get_scatter_idxs_vectorized, Local2GlobalSumEngine, IdentityEngine, and spherical RoPE updates. Fix: restore missing scatter_idxs computation. - model.py: keep branch's rollout_steps param and chunked forward pass; add develop's spherical RoPE, pushforward trick, IdentityEngine fallback for fe_num_blocks=0, and enabled-flag filtering for loss terms. Fix: restore if self.forecast_engine: guard. - target_and_aux_ssl_teacher.py: adopt develop's EncoderTeacher/EMATeacher/FrozenTeacher refactor; update both forward_teacher methods to pass rollout_steps=batch.get_output_len(). - trainer.py: auto-merged cleanly. Known pre-existing failures on ecmwf/develop (not introduced by this merge): - small1_test / small_multi_stream_test: ValueError in _add_lead_time_coord (dataarray_postprocessing.py, PR ecmwf#2140) - source_interval_start varies per ipoint - jepa1_test: omegaconf DictConfig/ListConfig mismatch - jepa1.yaml model_input uses old list format incompatible with develop's default_config.yml dict format
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: Even Marius Nordhagen <evnor2743@met.no>
…ith develop Signed-off-by: Even Marius Nordhagen <evnor2743@met.no>
…atherGenerator into evenmn/fix/incremental-rollout2
Signed-off-by: Even Nordhagen <evnor2743@met.no>
… not just the latent state coming from the forecast engine Signed-off-by: evenmn <evenmn@mn.uio.no>
github-actions
Bot
added
data
Signed-off-by: evenmn <evenmn@fys.uio.no>
Signed-off-by: evenmn <evenmn@fys.uio.no>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Functionality for writing the latent space to file is already implemented in develop. However, this writer can only be combined with the forecast engine, as it writes out the forecasted latent space from the forecast engine. For many applications this is sufficient, but sometimes it would be useful to write the encoded initial conditions as well. An example is when generating a dataset of latent states to train tail networks on.
Here, this feature is implemented in the most minimal way
Issue Number
Closes #2803
Is this PR a draft? Mark it as draft.
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60FastEvaluation