Handle automatic chunks duration for SC2 #3721
Open
+157
−21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yger
added
sortingcomponents
yger
changed the title
for more information, see https://pre-commit.ci
zm711
reviewed
Mar 2, 2025
| n_jobs = int(min(n_jobs, memory_usage // ram_requested)) | ||
| job_kwargs.update(dict(n_jobs=n_jobs)) | ||
| else: | ||
| print("psutil is required to use only a fraction of available memory") |
There was a problem hiding this comment.
are these prints or warnings? in general.
There was a problem hiding this comment.
I'll use the warning package instead
| if recording.get_total_memory_size() < memory_usage: | ||
| recording = recording.save_to_memory(format="memory", shared=True, **job_kwargs) | ||
| else: | ||
| print("Recording too large to be preloaded in RAM...") |
| @@ -73,6 +75,8 @@ class Spykingcircus2Sorter(ComponentsBasedSorter): | |||
| "matched_filtering": "Boolean to specify whether circus 2 should detect peaks via matched filtering (slightly slower)", | |||
| "cache_preprocessing": "How to cache the preprocessed recording. Mode can be memory, file, zarr, with extra arguments. In case of memory (default), \ | |||
| memory_limit will control how much RAM can be used. In case of folder or zarr, delete_cache controls if cache is cleaned after sorting", | |||
| "chunk_preprocessing": "How much RAM (approximately) should be devoted to load data chunks. memory_limit will control how much RAM can be used\ | |||
There was a problem hiding this comment.
I think you need to say what units of RAM this will work with. I'm confused by this with the way it is currently written. I think users will need a bit more info to really understand how to use this.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… into total_memory
… into total_memory
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using large number of cores, we might want to automatically set the chunk_size as function of the available RAM and/or number of cores. While I know that currently, in SI, the preprocessing nodes can consumes more RAM than chunk_size, at least this will avoid swapping and automatically adjust the sizes