Skip to content

Conversation

guarin
Copy link
Contributor

@guarin guarin commented Sep 17, 2025

What has changed and why?

  • Relax checks if run is resumed or overwritten

This should "fix" most occurrences where we raise an error because a mmap file already exists and we don't know whether it is the correct one. It makes the assumption that resume_interrupted and overwrite are used without changing the underlying data. For resume_interrupted this makes always sense because resume doesn't work properly if you change the data anyways. For overwrite this makes sense if we assume that overwrite is mostly used during debugging. Proper runs should always use a new directory.

How has it been tested?

  • Unit tests

Did you update CHANGELOG.md?

  • Yes
  • Not needed (internal change)

Did you update the documentation?

  • Yes
  • Not needed (internal change without effects for user)

@Copilot Copilot AI review requested due to automatic review settings September 17, 2025 11:03
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR improves existing memory-mapped (mmap) file handling by relaxing validation checks when runs are resumed or overwritten. The changes allow reuse of existing mmap files in these scenarios, assuming the underlying data hasn't changed.

  • Adds resume_interrupted and overwrite parameters to mmap-related functions to enable conditional file reuse
  • Updates error message formatting with proper f-string usage
  • Modifies all test calls to include the new boolean parameters

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
src/lightly_train/_commands/common_helpers.py Core logic changes to add resume/overwrite parameters and relaxed validation
src/lightly_train/_commands/train.py Updated function calls to pass config flags for resume and overwrite
src/lightly_train/_commands/embed.py Updated function calls with new parameters, hardcoded resume to False
tests/_commands/test_common_helpers.py Updated all test function calls to include the new boolean parameters

Copy link
Contributor

@yutong-xiang-97 yutong-xiang-97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved!

@guarin guarin merged commit c5579df into main Sep 19, 2025
16 checks passed
@guarin guarin deleted the update-existing-mmap-path-handling branch September 19, 2025 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants