Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use hash join when writing sparkey #5402

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

aslotnick
Copy link

When writing to sparkey, allShards represents every expected shard even if there is no corresponding data in shards for that shard number.

shards.rightOuterJoin(allShards) (added in #5208) fails when a shard contains large amounts of data, leading to the error described in #5300: java.lang.OutOfMemoryError: Required array length 2147483639 + 15534 is too large.

This PR replaces rightOuterJoin with hashFullOuterJoin (note that there is no hashRightOuterJoin implementation). A hash join is a good fit because the right-hand side contains very little data (only the keys of the shards) and it doesn't need to use an array to represent the large left-hand side's values. As a result, some failing workflows that succeeded in Scio 0.13.* will run successfully again.

Copy link
Contributor

@shnapz shnapz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allShards should fit into memory given numShards is Short

Copy link
Contributor

@kellen kellen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey thanks for looking into this, was about to start investigating myself. This seems like it should work and still sidestep the issues fixed in #5208

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants