Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix spaceleak in ByteString hashing #15

Merged
merged 2 commits into from
Nov 12, 2024

Conversation

phagenlocher
Copy link
Contributor

The current implementation for ByteString hashing requires many times more memory than the hashed data and causes a stack overflow if the ByteString is too large. This is PR fixes the issue by strict evaluation of the hash.

For lazy bytestrings this allows hashing in constant space, when treating them as streams. The following example runs with arbitrarily large inputs.

main :: IO ()
main = do
  let gib = 1024 * 1024 * 1024
      contents = BL.take (50 * gib) $ BL.repeat 0
  print $ hash64 contents

I have verified that the hashes stay the same for the following strings as lazy (BL) and strict (BS) bytestrings.

(BL) "AAA"      Hash64 0xa35eab6e4dcc6458       Hash32 0x1f2c96d
(BL) "aaa"      Hash64 0x99a0cdf698b33a11       Hash32 0x5ec168b8
(BL) "FOOBAR"   Hash64 0x67de7c0fad236411       Hash32 0xbb60b0d3
(BS) "AAA"      Hash64 0x7fc0dac8b37d0287       Hash32 0xcf7ed713
(BS) "aaa"      Hash64 0x2b32d37fe2c25614       Hash32 0xbc1171ce
(BS) "FOOBAR"   Hash64 0x307a25cc9112e346       Hash32 0x543fa0dc

@phagenlocher
Copy link
Contributor Author

@nominolo Thoughts?

@andreasabel andreasabel added this to the 0.1.0.11 milestone Nov 12, 2024
@andreasabel andreasabel force-pushed the fix-bytestring-spaceleak branch from e180362 to 17b85ba Compare November 12, 2024 20:10
@andreasabel
Copy link
Collaborator

@andreasabel andreasabel linked an issue Nov 12, 2024 that may be closed by this pull request
@andreasabel andreasabel self-assigned this Nov 12, 2024
@andreasabel andreasabel merged commit 8ca835e into nominolo:master Nov 12, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Space leak in Hashable64 ByteString
2 participants