-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Description
Problem description
Currently, the Experiment class in Experiment.py has several issues leading to errors when running experiments:
- Order of operations —
check_minio_credentials()was called beforeenv_setup(), causing environment variables for MinIO to not be set in time. - Faulty condition — the code tried to log
self.auth.methodeven whenself.authwasNone, raising:AttributeError: 'NoneType' object has no attribute 'method' - Missing credentials — if MinIO credentials were only present in the
.cfgfile but not exported in the environment, the process failed.
This behavior causes experiments to crash unexpectedly.
Steps to Reproduce
- Run:
mlops run scripts/train.py -c config/local_config.cfg --ignore_git_check - Observe the crash in
check_minio_credentials().
Proposed Solution
-
Change the initialization order:
- Call
self.config_setup(). - Call
self.env_setup()to set environment variables. - Call
self.check_minio_credentials().
- Call
-
Add setting of MinIO environment variables directly in
env_setup()to ensure they’re available before credentials check.
Related Context
- Part of fixing an issue raised in MLOpsTutorial.
- The
.cfgupdates to include MinIO credentials are tracked in the MLOpsTutorial repo, while the code fix is in MLOps.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels