Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to serialize large R*Tree using FlatBuffer #54

Open
ksaurab opened this issue Jun 24, 2016 · 1 comment
Open

Unable to serialize large R*Tree using FlatBuffer #54

ksaurab opened this issue Jun 24, 2016 · 1 comment

Comments

@ksaurab
Copy link

ksaurab commented Jun 24, 2016

I was trying to Serialize a large R*Tree using the APIs in this Library, but it failed with error :

Exception in thread "main" java.lang.AssertionError: FlatBuffers: cannot grow buffer beyond 2 gigabytes.
at com.google.flatbuffers.FlatBufferBuilder.growByteBuffer(FlatBufferBuilder.java:127)
at com.google.flatbuffers.FlatBufferBuilder.prep(FlatBufferBuilder.java:173)
at com.google.flatbuffers.FlatBufferBuilder.addOffset(FlatBufferBuilder.java:291)
at com.google.flatbuffers.FlatBufferBuilder.addOffset(FlatBufferBuilder.java:562)
at com.github.davidmoten.rtree.fbs.generated.Entry_.addObject(Entry_.java:33)
at com.github.davidmoten.rtree.fbs.generated.Entry_.createEntry_(Entry_.java:26)
at com.github.davidmoten.rtree.fbs.FlatBuffersHelper.addEntries(FlatBuffersHelper.java:77)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:96)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.addNode(SerializerFlatBuffers.java:102)
at com.github.davidmoten.rtree.fbs.SerializerFlatBuffers.write(SerializerFlatBuffers.java:73)

Seems like FlatBuffers can't serialize large structures with size more than 2GB? Am I mistaken here?
Does this implementation support any other serialization apart from FlatBuffer?
I saw few test cases for Kryo Serialization but don't think if the current structure supports this.
Please guide.

@davidmoten
Copy link
Owner

Just FlatBuffers serialization is supported at the moment but I might need to add a Kryo option given that 2GB limit. I'll have a think about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants