Add js compiler protodef implementation, that is 10x faster #102

Karang · 2020-05-11T19:56:49Z

My attempt at #86

The compiler can generate reading and writing code. It follows the protodef standard and can be extended with custom native types. It can compile a code that is able to read the whole nbt spec.

I also included an experimental function that can optimize the generated code using the closure compiler. It is not always faster and, when it is, not by much.

Running examples/compiled.js:

Running 1000000 tests
write / read compiled: 195.35 ms (5119.09k packet/s)
serializer / parser: 18095.48 ms (55.26k packet/s)
write / read compiled (+closure): 123.37 ms (8105.85k packet/s)

Running examples/compiled_nbt.js:

Running 10000 tests
read compiled: 282.89 ms (35.35k packet/s)
parser: 2237.11 ms (4.47k packet/s)
read compiled (+closure): 288.36 ms (34.68k packet/s)

rom1504 · 2020-05-11T20:04:42Z

That's cool. Did you try yet to read minecraft protocol.json with this or are there missing features that prevent it ?

…

On Mon, May 11, 2020, 21:57 Karang ***@***.***> wrote: My attempt at #86 <#86> For now the compiler can only generate reading code, writing will be done later. It follows the protodef standard and can be extended with custom native types. It can compile a code that is able to read the whole nbt spec. I also included an experimental function that can optimize the generated code using the closure compiler. It is not always faster and, when it is, not by much. Running examples/compiled.js: Running 1000000 tests read compiled: 98.29 ms (10173.47k packet/s) parser: 5061.42 ms (197.57k packet/s) read compiled (+closure): 79.97 ms (12504.31k packet/s) Running examples/compiled_nbt.js: Running 10000 tests read compiled: 414.75 ms (24.11k packet/s) parser: 2324.88 ms (4.30k packet/s) read compiled (+closure): 408.18 ms (24.50k packet/s) ------------------------------ You can view, comment on, or merge this pull request online at: #102 Commit Summary - First compiler version - Refactor compiler - Add nbt example File Changes - *A* examples/bigtest.nbt <https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-708b8e0cd839473a86e479bc881a4589> (0) - *A* examples/compiled.js <https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-5ed6b27e089def914e547cb305c8782d> (95) - *A* examples/compiled_nbt.js <https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-9e714e61d25d5e6f12b4fcd39795423f> (140) - *A* examples/nbt.json <https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-cec856c9b6a6b27ffc0b3be07fc46ee4> (113) - *M* package.json <https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-b9cfc7f2cdf78a7f4b91a753d10865a2> (1) - *A* src/compiler.js <https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-b0b85d67c2201828828422795fb44581> (243) - *M* src/index.js <https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-1fdf421c05c1140f6d71444ea2b27638> (1) Patch Links: - https://github.com/ProtoDef-io/node-protodef/pull/102.patch - https://github.com/ProtoDef-io/node-protodef/pull/102.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#102>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437U4WNDPLTRG6SEEOJDRRBKBBANCNFSM4M6GOAPQ> .

Karang · 2020-05-11T20:15:41Z

That's cool. Did you try yet to read minecraft protocol.json with this or are there missing features that prevent it ?

As far as node-protodef is concerned, nothing is missing. But to implement the minecraft protocol, the specific primitives needs to be implemented : https://github.com/PrismarineJS/node-minecraft-protocol/blob/master/src/datatypes/minecraft.js (nbt is already done in my example). But following the example of https://github.com/ProtoDef-io/node-protodef/pull/102/files#diff-9e714e61d25d5e6f12b4fcd39795423fR64 it shouldn't be hard to do.

I included the nbt example here (and copy pasted some code from prismarine-nbt) for testing purpose, but in reality, the non- node-protodef code should be in its respective repository.

So, to support minecraft protocol, we need to first update prismarine-nbt then node-minecraft-protocol. (I can already implement the missing bits and include them here for testing, then we can move it later)

rom1504 · 2020-05-11T22:26:34Z

ok I get it, would be good to add a .md file similar to https://github.com/ProtoDef-io/node-protodef/blob/master/doc/newDatatypes.md that would explain how to add these types

Saiv46 · 2020-05-13T10:54:50Z

src/compiler.js

 }

 function optimize (code, cb) {
  const closureCompiler = new ClosureCompiler({
-    language_in: 'ES6',
    compilation_level: 'SIMPLE'


Umm, did 'ADVANCED' mode breaks code?

Yes, it is too aggressive with variable renaming, even when specifying the correct externs. There might be a way to make it work but a lot of configuration would be required.

rom1504

overall it looks good

it would be nice to run the tests using it

doc/compiler.md

rom1504 · 2020-05-15T21:58:18Z

doc/compiler.md

+
+## New datatypes
+
+Like the ProtoDef interpreter, the ProtoDef compiler can be extended with custom datatypes. To register a custom type, use the `addTypes(types)` method of the ProtoDef compiler. The `types` parameter is an object with the following structure:


what do you think of grouping the types (each with its read, write, sizeof) instead of doing groups of read/write/sizeof containing each all the types ?

yes, I can do that

Actually, this was done to simplify the implementation: internally there is 3 separate compilers for each functionality, it's just easier to give them each their own types. We can go around that with a conversion step but that's a bit of work I prefer to do later.

doc/compiler.md

rom1504 · 2020-05-15T22:06:19Z

Overall API looks ok, so I think it makes sense to start integrating directly in prismarine-nbt and node-minecraft-protocol (you can git depend on node-protodef for now in PRs)

rom1504 · 2020-05-16T21:04:33Z

would be nice to adapt the unit tests (https://github.com/ProtoDef-io/node-protodef/tree/master/test/dataTypes) and benchmarks (https://github.com/ProtoDef-io/node-protodef/tree/master/benchmark) to run on both the interpreter and the compiler (separately)
it'll make it possible to catch any future regression + to do more precise benchmarking whenever needed

Saiv46 · 2020-05-18T07:43:13Z

src/compiler.js

    if (!isNaN(fn)) {
-      this.primitiveTypes[type] = fn
+      this.native[type] = (value) => { return fn }


Can we inline native functions?

In this case fn is a number (!isNaN(fn)) It cannot be inline because it needs to have the same function interface, but anyway the JIT compiler will optimise it for us.

Karang · 2020-05-19T23:37:30Z

would be nice to adapt the unit tests (https://github.com/ProtoDef-io/node-protodef/tree/master/test/dataTypes) and benchmarks (https://github.com/ProtoDef-io/node-protodef/tree/master/benchmark) to run on both the interpreter and the compiler (separately)
it'll make it possible to catch any future regression + to do more precise benchmarking whenever needed

done

rom1504 · 2020-05-20T10:21:08Z

LGTM, thanks for this huge improvement !

Karang added 3 commits May 11, 2020 15:13

First compiler version

e1570af

Refactor compiler

93ec4f7

Add nbt example

14167fa

Improve perf by copying natives function into context

7897c47

Karang force-pushed the karang_compiler branch from f539566 to 7897c47 Compare May 11, 2020 20:51

Karang added 4 commits May 12, 2020 03:02

Improve compiler, add anon inlining

57feabd

Add compiler.getField to handle scope properly

e234fe1

remove debug print

a0ef224

Add WriteCompiler

5ffe0d5

Saiv46 reviewed May 13, 2020

View reviewed changes

Karang added 8 commits May 13, 2020 17:17

Improve writer, add nbt test

7f289d0

Correctly handle partial read errors

b987e58

Add SizeOfCompiler

fccf3b3

Improve code generation

0009efb

Use sizeof in nbt example

048fd73

Extract default datatypes from compilers

10cf510

Improve compiler api

60af77d

Add some doc

20d36ae

Karang force-pushed the karang_compiler branch from 1a8e728 to 20d36ae Compare May 15, 2020 19:55

Add mc protocol example

729189d

rom1504 requested changes May 15, 2020

View reviewed changes

Karang added 4 commits May 16, 2020 00:37

Add sync compile function

d8214b5

Fix small bugs

4267759

Fix more bugs

263f41d

Add count support

8a2a19b

Karang mentioned this pull request May 16, 2020

Use protodef compiler PrismarineJS/node-minecraft-protocol#714

Merged

Make sizeof native types also a function

c4fbe57

Karang mentioned this pull request May 16, 2020

Use protodef compiler PrismarineJS/prismarine-nbt#23

Merged

Saiv46 reviewed May 18, 2020

View reviewed changes

Karang added 7 commits May 18, 2020 18:07

CompiledProtodef implements all Protodef api

190bfe5

Remove mc related tests

2e889df

Remove nbt tests

26cd664

Remove nbt.json

749a178

Add api

2b365d2

Add compiled tests, fix bitfields

d390ba0

Add compiled benchmarks

ac52f1a

rom1504 changed the title ~~WIP: investigate js code generation from protodef json~~ Add js compiler protodef implementation, that is 10x faster May 20, 2020

rom1504 approved these changes May 20, 2020

View reviewed changes

rom1504 merged commit 70641a8 into ProtoDef-io:master May 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add js compiler protodef implementation, that is 10x faster #102

Add js compiler protodef implementation, that is 10x faster #102

Karang commented May 11, 2020 •

edited

Loading

rom1504 commented May 11, 2020 via email

Karang commented May 11, 2020 •

edited

Loading

rom1504 commented May 11, 2020

Saiv46 May 13, 2020

Karang May 14, 2020

rom1504 left a comment

rom1504 May 15, 2020

Karang May 15, 2020

Karang May 15, 2020

rom1504 commented May 15, 2020

rom1504 commented May 16, 2020

Saiv46 May 18, 2020

Karang May 18, 2020

Karang commented May 19, 2020

rom1504 commented May 20, 2020


		## New datatypes

		Like the ProtoDef interpreter, the ProtoDef compiler can be extended with custom datatypes. To register a custom type, use the `addTypes(types)` method of the ProtoDef compiler. The `types` parameter is an object with the following structure:

Add js compiler protodef implementation, that is 10x faster #102

Add js compiler protodef implementation, that is 10x faster #102

Conversation

Karang commented May 11, 2020 • edited Loading

rom1504 commented May 11, 2020 via email

Karang commented May 11, 2020 • edited Loading

rom1504 commented May 11, 2020

Saiv46 May 13, 2020

Choose a reason for hiding this comment

Karang May 14, 2020

Choose a reason for hiding this comment

rom1504 left a comment

Choose a reason for hiding this comment

rom1504 May 15, 2020

Choose a reason for hiding this comment

Karang May 15, 2020

Choose a reason for hiding this comment

Karang May 15, 2020

Choose a reason for hiding this comment

rom1504 commented May 15, 2020

rom1504 commented May 16, 2020

Saiv46 May 18, 2020

Choose a reason for hiding this comment

Karang May 18, 2020

Choose a reason for hiding this comment

Karang commented May 19, 2020

rom1504 commented May 20, 2020

Karang commented May 11, 2020 •

edited

Loading

Karang commented May 11, 2020 •

edited

Loading