Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

don't use rayon on files where it will be slower #39

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rot13maxi
Copy link

when a user asks for turbo mode, first stat the file and make sure its over 128k, if its not just hash normally. According to the blake3 docs, rayon is slower for files smaller than this.

@casey
Copy link
Owner

casey commented Sep 14, 2024

I'm actually wondering if we should do this! update_mmap_rayon calls update_mmap, which says:

In those cases, this method will fall back to standard file IO. The heuristic for whether to use memory mapping is currently very simple (file size >= 16 KiB), and it might change at any time.

So we'll already fall back to not using rayon if the filesize is below 16 KiB, so it seems like it might not be a huge win to hard-code the case between 16 KiB and 128 KiB, and makes it slightly less predictable for the user, since without implementing our own heuristics, the different modes map directly to the different functions in the blake3 crate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants