Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nwparker/conv v2 #3

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Nwparker/conv v2 #3

wants to merge 15 commits into from

Conversation

nwparker
Copy link
Collaborator

@nwparker nwparker commented Sep 6, 2018

This PR introduces some major changes. It fixes many issues with the previous versions and changes some core functionality.

Changes in the PR:

  • Re-written encoder (to deal with additional cases and data types)
  • A decoder to convert from SparkSQL Rows back into the Java Thrift generated objects
  • Supporting Additional Data Types
    • Map w/ complex (non-primitive) keys
    • Struct/Unions w/ self-recursive definitions (these will be converted to binary as there's no way to represent arbitrary recursive data types in SQL. A benefit of keeping in binary format is that the data can be decoded back into the original data structure with the decoder)
  • A unit test to preserve the encode/decode invariant through (uses generated data via ScalaCheck)
  • Thrift Java objects are no longer check into repo, but generated from Thrift sources
  • Moves project from Maven to SBT (allows cross-building for multiple Scala versions)
  • Updates Readme w/ some new info

Fixes #2

@liyintang
Copy link
Collaborator

@nwparker Thanks for making this change ! Let me take a look !

@@ -0,0 +1 @@
gen-java
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I can remove this now that gen-java is written directly into /target

@nwparker
Copy link
Collaborator Author

nwparker commented Oct 2, 2018

@nwparker Thanks for making this change ! Let me take a look !

Hey @liyintang -- just wondering if you may have any update. Thanks!

@nwparker
Copy link
Collaborator Author

Another friendly bump ! 😄

@nwparker
Copy link
Collaborator Author

Bumping @liyintang -- & Happy Thanksgiving !

@nwparker
Copy link
Collaborator Author

bumping for the new year @liyintang

@nwparker
Copy link
Collaborator Author

Any update ?

@nwparker
Copy link
Collaborator Author

2020 bump

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Supporting More Types / Fixing Bugs/ Etc
2 participants