[#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec #171

tolbertam · 2016-08-09T04:21:11Z

For #76, Adds protocol v3 and v4 support. This is a work in progress, but it's functional and currently all tests pass.

Summary of the major changes:

Use scodec for serialization/deserialization. All serde has been moved into a standalone codec module with scodec being the only dependency.
Change Prime store implementations to keep the request intact instead of deconstructing on input and reconstructing when receiving via the route APIs. This should keep things simpler. Added a memoized prime member on the Then implementations to convert the inputs to a Prime.

There should be no functional changes for users with one slight exception. Previously if no Prime is found for a query, query parameters would not be recorded in the activity log for batch and prepared statements. Now it records the query parameters as if they were varchars. There may also be some small nuances in conversions from json strings/numbers/booleans/etc. into the native type for a given data type, but i haven't identified any yet.

What is remaining:

Update cql-antlr to include the new types added in v3/v4 (tuples, udt, time, date, smallint, tinyint, etc.). The codecs exist for these, but need to be tested. I think there might be some barriers to supporting UDTs (covered in User defined type support #159), so depending on the difficulty level I may defer this. ([#3] Add support types added in protocol v3 and v4 cql-antlr#4)
Clean up, document and add more tests for the codec module. scodec is really nice, but its API is intimidating from my view. It is also not a complete implementation of the protocol, but the existing implementation isn't either (can maybe do more as follow ons).
Replace ServerStubrRunnerIntegrationTest / PrimingJsonHelperTest. I removed these earlier, need to replace them with new alternatives.
Add/update headers (pending Update license headers and add validation to 'check' #170 merge)
Clean up commit history

chbatey · 2016-09-05T14:37:09Z

How's this going?

Are you at the Cassandra summit? Might be nice to review together

tolbertam · 2016-09-05T14:41:17Z

Hi @chbatey, it's going pretty well! Existing tests are passing, what's left is adding additional tests in the codec module, adding additional tests for the new types and maybe some clean up. Hopefully I can complete that really soon.

I'll be at the summit, I'd definitely be up for a review, that'd be fun!

tolbertam · 2016-09-05T18:33:46Z

Rebased against master and added license headers. Working on finishing up tests in the codec module and updating rest of the code to support the v3/v4 types. I'll probably get that all going and clean up the commit history today.

chbatey · 2016-10-11T11:06:27Z

Busy man!

tolbertam · 2016-10-11T14:14:23Z

rebased on master to get the cql-antlr changes. I'll proceed to wire in the new types, and from there that should leave just some testing/doc updates/cleanup!

tolbertam · 2016-10-13T06:18:28Z

Getting closer, completed adding the new types and failures. Going to spend some time tomorrow evening updating the docs. I'll then do some cleanup / optimization of the codec code and clean up the commit history. If all goes well should complete sometime this weekend.

Use 'AnyMatch' as catch all Use terminate instead of shutdown.

Inline own implicit conversions. One small difference between this implementation and scodec-akka is this uses reflection instead of a java class.

Have collection-based data types consider protocol version for length. Batch considers protocol version for whether or not to include flags. Prepare considers protocol version for whether or not to include pks. native PF for encoding from Any -> Native type for data type. Use CL from statement only if prime does not include one.

There are enough changes here, some of which may be (unintentionally) breaking, bumping up the minor version seems like the right way to go here.

Also add example for priming a query with a pattern and some other cleanup.

Need to be able to differentiate when there are no columns and no column metadata was requested.

Also add documentation

Rename BadCredentials to AuthenticationError. Switch from scalaz for Memo to guava for Cache as would like to use a LRU cache for Row codecs since this could grow large if you have a lot of unique metadata.

tolbertam · 2016-10-15T20:59:24Z

I've spent some time the last few days adding integration tests for the new types and errors, and some more tests and documentation for the codec module.

I think i've got everything worked out and this is ready aside from the commit history. @chbatey would you prefer I squash the commits now or would you have me leave it as is for now for review?

mgobec · 2016-10-16T18:14:56Z

Great work @tolbertam 👍

chbatey · 2016-11-18T16:35:15Z

Taking a look this weekend

spray-routing-shapeless2 depends on shapeless 2.1, where scodec depends on shapeless 2.3. Depending on what is being used for dependency resolution, if shapeless 2.1 is chosen over 2.3 this causes incompatibility issues with scodec.

chbatey · 2017-01-05T10:38:15Z

server/src/main/scala/org/scassandra/server/actors/ConnectionHandler.scala

+  private[this] def inspectProtocolVersion(input: ByteVector): Try[(ProtocolFlags, ByteVector)] = {
+    next[ProtocolFlags](input).flatMap {
+      case ((ProtocolFlags(_, UnsupportedProtocolVersion(v)), _)) =>
+        Failure(new UnsupportedProtocolException(v))


No need for new

chbatey · 2017-01-05T11:29:03Z

server/src/main/scala/org/scassandra/server/actors/NativeProtocolMessageHandler.scala

+          val registerHandler = registerHandlerFactory(context)
+          write(Ready, frame.header)
+          context become initialized(queryHandler, batchHandler, registerHandler)
+        case _ => {


remove braces to make it consistent with the other cases

chbatey · 2017-01-05T11:31:22Z

server/src/main/scala/org/scassandra/server/actors/ExecuteHandler.scala

-      case result: FatalResult => result.produceFatalError(connection)
+        writePrime(execute, prime, header, Some(connection), alternative=Some(Reply(VoidResult)), consistency = Some(execute.parameters.consistency))
+      case None =>
+        var errMsg = s"Could not find prepared statement with id: 0x${execute.id.toHex}"


should be a val

d'oh! I've been doing too much javascript lately 😄

chbatey · 2017-01-05T11:33:16Z

server/src/main/scala/org/scassandra/server/actors/ExecuteHandler.scala

+        writePrime(execute, prime, header, Some(connection), alternative=Some(Reply(VoidResult)), consistency = Some(execute.parameters.consistency))
+      case None =>
+        var errMsg = s"Could not find prepared statement with id: 0x${execute.id.toHex}"
+        activityLog.recordPreparedStatementExecution(errMsg, execute.parameters.consistency, Nil, Nil)


i take it you found this useful for testing? just a little off we use an error msg as the statement text

Hrmm, I may be misunderstanding, isn't the behavior the same as before? I think the result is in this case that we record a prepared statement execution with the error message as the query text, and then an Unprepared message gets sent back to the client which will cause most drivers to repreare the statement.

chbatey · 2017-01-05T11:42:29Z

Happy to merge this whenever. I'll be adding some features shortly

tolbertam · 2017-01-05T15:08:23Z

Happy to merge this whenever. I'll be adding some features shortly

Excellent, thanks for the review! I'll address your feedback this morning.

tolbertam · 2017-01-05T16:48:47Z

added commit for review feedback, thanks! Would you like me to do anything with the commit history, or would you just prefer to squash the whole thing on merge?

chbatey · 2017-01-05T17:04:39Z

Squashing into one if find!

It has been a long time since I looked at Scala lol

tolbertam · 2017-01-05T17:16:16Z

Just remembered, when it comes time to do a release with these changes, there are a few activites:

Merge Add deprecation notice cql-antlr#7 to indicate that repository is deprecated in favor of cql-antlr being in this repository now.
Update gh-pages branch / push to read the docs since i made some doc updates.

chbatey · 2017-01-06T16:58:43Z

Thanks @tolbertam

tolbertam · 2017-01-10T21:02:11Z

@chbatey would you like me to merge and go through the process to release this? Can do it tomorrow if you like.

chbatey · 2017-01-10T21:34:39Z

Go for it!

…

On 10 Jan 2017, at 21:02, Andrew Tolbert ***@***.***> wrote: @chbatey would you like me to merge and go through the process to release this? Can do it tomorrow if you like. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

tolbertam · 2017-01-10T22:16:54Z

Awesome, thanks! Will take care of that this evening :)

This was referenced Sep 9, 2016

Support for named parameters in queries #172

Open

[#173] Migrate cql-antlr from its own repo to a submodule #174

Merged

tolbertam added 22 commits October 15, 2016 15:13

Initial commit of scodec-based protocol parser.

044c6b1

Initial commit of scodec changes.

94231e8

Delete some tests (to be replaced).

719aed8

Fixes

373acbc

Remove JDK7 specific annotations

e686b60

Use 'AnyMatch' as catch all Use terminate instead of shutdown.

Update driver dependencies.

feebe73

Remove scodec-akka dependency since it requires JDK8

2197d00

Inline own implicit conversions. One small difference between this implementation and scodec-akka is this uses reflection instead of a java class.

Fix java-it-tests source compatibility

b3c5f42

Fallback on toString for Any in String codecs.

af7edbf

Fix batch id parsing.

7aaf41b

Replace duplicate batch primes.

70769d7

handle null values to json.

77a2e4d

Expect variables even if no prime is matched (enhancement).

4f09b0b

Grab variable types from preparedMetadata.

0ca3ad6

Default consistency / column types like before.

62e76b0

Expect query parmeters as they are now recorded (weren't before).

a1ec36a

Default variable types on prime incoming.

f078c0c

Update tests to consider implicit protocol version.

e5883f1

Remove printlns

9c64e09

Address some TODOs, remove obsolete ones.

859f98c

Bump to 1.1.0

c6418e2

There are enough changes here, some of which may be (unintentionally) breaking, bumping up the minor version seems like the right way to go here.

tolbertam added 12 commits October 15, 2016 15:13

Add documentation for protocol v3 and v4 changes.

318ccc2

Also add example for priming a query with a pattern and some other cleanup.

Avoid repeated creation of codecs by binding them to ProtocolVersion

ed2cd8e

Cache dataType codecs.

dffcfdb

Fix and simplify Batch codec, add Message specs

69791ce

Fix PreparedMetadata, tests for Result

79f1b33

Fix compilation issues introduced by previous commit.

4f7534a

Revert logic to treat RowMetadata as an Option.

0882ad9

Need to be able to differentiate when there are no columns and no column metadata was requested.

Optimize FrameHeader codec by tying it to ProtocolVersion.

b919f79

Also add documentation

Cache Row codecs and add more documentation.

c65a11d

Rename BadCredentials to AuthenticationError. Switch from scalaz for Memo to guava for Cache as would like to use a LRU cache for Row codecs since this could grow large if you have a lot of unique metadata.

More documentation of codec module.

239005f

Update readme to include new modules and note protocol version support.

ea0b084

Force java 1.6 for codec module. Update some URLs in gradle.

462eff8

Make more clear that standalone jar is the one that should be used.

023eaa8

tolbertam changed the title ~~[#76] [wip] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec~~ [#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec Oct 16, 2016

chbatey approved these changes Jan 5, 2017

View reviewed changes

Changes based on review feedback.

83b90f0

tolbertam merged commit 41432b2 into scassandra:master Jan 11, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec #171

[#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec #171

tolbertam commented Aug 9, 2016 •

edited

Loading

chbatey commented Sep 5, 2016

tolbertam commented Sep 5, 2016 •

edited

Loading

tolbertam commented Sep 5, 2016

chbatey commented Oct 11, 2016

tolbertam commented Oct 11, 2016

tolbertam commented Oct 13, 2016

tolbertam commented Oct 15, 2016

mgobec commented Oct 16, 2016

chbatey commented Nov 18, 2016

chbatey Jan 5, 2017

chbatey Jan 5, 2017

chbatey Jan 5, 2017

tolbertam Jan 5, 2017

chbatey Jan 5, 2017

tolbertam Jan 5, 2017

chbatey commented Jan 5, 2017

tolbertam commented Jan 5, 2017

tolbertam commented Jan 5, 2017

chbatey commented Jan 5, 2017

tolbertam commented Jan 5, 2017 •

edited

Loading

chbatey commented Jan 6, 2017

tolbertam commented Jan 10, 2017

chbatey commented Jan 10, 2017 via email

tolbertam commented Jan 10, 2017

[#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec #171

[#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec #171

Conversation

tolbertam commented Aug 9, 2016 • edited Loading

chbatey commented Sep 5, 2016

tolbertam commented Sep 5, 2016 • edited Loading

tolbertam commented Sep 5, 2016

chbatey commented Oct 11, 2016

tolbertam commented Oct 11, 2016

tolbertam commented Oct 13, 2016

tolbertam commented Oct 15, 2016

mgobec commented Oct 16, 2016

chbatey commented Nov 18, 2016

chbatey Jan 5, 2017

Choose a reason for hiding this comment

chbatey Jan 5, 2017

Choose a reason for hiding this comment

chbatey Jan 5, 2017

Choose a reason for hiding this comment

tolbertam Jan 5, 2017

Choose a reason for hiding this comment

chbatey Jan 5, 2017

Choose a reason for hiding this comment

tolbertam Jan 5, 2017

Choose a reason for hiding this comment

chbatey commented Jan 5, 2017

tolbertam commented Jan 5, 2017

tolbertam commented Jan 5, 2017

chbatey commented Jan 5, 2017

tolbertam commented Jan 5, 2017 • edited Loading

chbatey commented Jan 6, 2017

tolbertam commented Jan 10, 2017

chbatey commented Jan 10, 2017 via email

tolbertam commented Jan 10, 2017

tolbertam commented Aug 9, 2016 •

edited

Loading

tolbertam commented Sep 5, 2016 •

edited

Loading

tolbertam commented Jan 5, 2017 •

edited

Loading