HEALPix Curriculum

[TillHae]

Description

This adds a HEALPix curriculum to our training pipeline. During my bachelor thesis, I found that starting at a lower HEALPix Level (HL) and stepping it up during training cuts early compute time by almost 95%, all without hurting the final model performance.

How it works

• Easy config: Uses the healpix_curriculum parameter to set how many isteps each HL should run for.
• Dynamic streams: Uses OmegaConf interpolation (e.g., streams_directory: "${curriculum_streams.${healpix_level}}") so the datasets swap automatically as the HL goes up.
• Auto-restarts: run_train.py now handles the transitions. When a stage ends, it stops the mini-epoch, saves a checkpoint and immediately starts the next stage within the same Slurm job.

FYI

• Configs: I threw in config_curriculum.yml just to make reviewing the OmegaConf logic easier. Before merging it can be deleted so we don't clutter the repo with new configs.

Issue Number

Closes #2794

@kctezcan

Is this PR a draft? Mark it as draft.

Checklist before asking for review

• I have performed a self-review of my code
• My changes comply with basic sanity checks:
  • I have fixed formatting issues with ./scripts/actions.sh lint
  • I have run unit tests with ./scripts/actions.sh unit-test
  • I have documented my code and I have updated the docstrings.
  • I have added unit tests, if relevant
• I have tried my changes with data and code:
  • I have run the integration tests with ./scripts/actions.sh integration-test
  • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
  • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
• I have informed and aligned with people impacted by my change:
  • for config changes: the MatterMost channels and/or a design doc
  • for changes of dependencies: the MatterMost software development channel

FastEvaluation

• I have updated the public documentation if necessary

[TillHae]

Here are the results I got from my test run. It works pretty good! I had some problems with the configs and auto-restarting after a transition, but was able to make everything work with some help from AI.

I will do a self-review of my code on Friday.

[TillHae]

I will do a self-review of my code on Friday.

Took me some time to get back to it, but it looks good.

@kctezcan do you have time to review it?