Hydra Config File Support (Post LightningCLI integration)

[bonham79]

LightningCLI supports YAML configs. However we're going to need to add some additional OmegaConf and Hydra support to our existing scripts to best utilize this. This issue will work on full integration of LightningCLI post PTL 2. Also creates YAML files to document base files.

[kylebgorman]

Migrating to PTL 2 and its LightningCLI interface (#60) also gives us configuration file support.

[bonham79]

The two go together. LightningCLI builds off Hydra and OmegaConf:

https://github.com/Lightning-AI/pytorch-lightning/blob/cf24a190ce52d65ae7316fceec584145f1e1f006/requirements/pytorch/extra.txt#L7-L8

[kylebgorman]

Are the lightingcli YAML files instances of Hydra or OmegaConf files?Basically what I want to know is whether, if we migrate to LightningCLI, this issue is complete or there is subsequent work needed to add support for these other tools.On Aug 6, 2024, at 12:32 PM, Travis Bartley ***@***.***> wrote: The two go together. LightningCLI builds off Hydra and OmegaConf: https://github.com/Lightning-AI/pytorch-lightning/blob/cf24a190ce52d65ae7316fceec584145f1e1f006/requirements/pytorch/extra.txt#L7-L8 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[bonham79]

Hydra is used for command line processing of YAML configs and OmegaConf is used for Python processing of YAML configs. So those would be the backend. However, we're still going to need to instantiate our own idiosyncratic way of using them.

At base, we can keep this issue up as a reminder to create YAML config files for every base model we have. I'll rename accordingly.

[kylebgorman]

Thanks for clarification; I think a successful completion of the migration to PTL 2 will have this as a side effect but let’s keep this open to make sure we’re happy with config file support. I much prefer config files over Bash for this kind of thing, sounds like you too.On Aug 6, 2024, at 12:55 PM, Travis Bartley ***@***.***> wrote: Hydra is used for command line processing of YAML configs and OmegaConf is used for Python processing of YAML configs. So those would be the backend. However, we're still going to need to instantiate our own idiosyncratic way of using them. At base, we can keep this issue up as a reminder to create YAML config files for every base model we have. I'll rename accordingly. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[bonham79]

It has pros and cons. But our argument parser is getting a bit unwieldy so the alternative would be easier on our eyes.