Move MinHashLSH logic to Rust, we're back

[yonromai]

Rebase of #2191

Refs:

• [Epic] Reliable Dedup Pipeline #2091
• [dedup] Milestone 3: Make Exact Dedup Better #2147

[ravwojdyla]

lgtm, thank you!! We may change some inputs/outputs, but we can do that in a follow up PR 🙇

[rjpower]

Looks great! Just one comment about the bench test location.

(Alternatively could change the benchmark code to copy-paste some of the logic, so long as we don't have the Marin -> Dupekit dependency)

[rjpower]

This feels a little weird, maybe this should be in marin instead? I don't like the inverted dependency

[yonromai]

Looks great! Just one comment about the bench test location.

(Alternatively could change the benchmark code to copy-paste some of the logic, so long as we don't have the Marin -> Dupekit dependency)

This makes sense, I agree! I looked into the two ways you suggested:

1. Copying code from Marin to Dupekit: The python code is a little deep as it depends on marin.processing.classification.deduplication.vendor.datasketch.minhash.
2. Add the bench code to Marin: This could make sense, but that's not an existing pattern and it would require bringing in some extra packages to Marin (which already has a lot):

benchmark = [
    "pytest-benchmark>=5.2.3",
    "pytest-memray>=1.8.0"
]

I think my favorite option for now is just to delete this benchmark. If we happen to actually need it in the future, I volunteer to bring it back. cc: @ravwojdyla

Note: I'm going to eager merge this PR, to unblock this. @rjpower If you think of something else I should do please LMK and I'll open another PR.