[pug-code-gen] Rewrite using babel

[jeromew]

From @ForbesLindesay on November 6, 2015 10:1

babel does a really good job of generating source maps. Generating ASTs will also help us only generate proper, syntactically valid javascript. As a bonus we can rapidly run any babel plugins we want on the output for optimisation or implementation of advanced features.

One of the restrictions this will impose is that unbuffered code blocks (e.g. - var x = 10;) will now need to be valid statements. This forces people to use our built in constructs for if, else, each etc. I don't think that's a bad thing though, and we can make up for it with much better error messages for syntax errors.

I think this is too big a feature to block the 2.0.0 release, which I really want to roll out this weekend. I do think we should hold off on plugins for code-gen until this is complete though.

Copied from original issue: pugjs/pug-code-gen#21

[jeromew]

for info I am keen on trying to see how then-jade and react-jade can be built on the new 2.0.0 toolkit ; these 2 libraries modify the code-gen.

by 'Rewrite using babel' you mean that jade-code-gen should feed AST directly into babel ?

[jeromew]

From @ForbesLindesay on November 6, 2015 10:39

Yes, that's what I mean by re-write using babel. In the short term, then-jade should fully fork the code-gen module, but will benefit from a more stable and defined AST structure. In the long term I hope we will have a plugin architecture that allows overriding just the mixin parts of code-gen. react-jade is fairly irrelevant here because it's such a total fork of code-gen anyway (it doesn't share any code-gen code with jade core).

[jeromew]

I'll look into forking this for then-jade once 2.0 is publish ; I think the buf += path was chosen over the buf.push so the diff will be a bit bigger than before.

regading react-jade I agree that it has a lot of specific React tricks. The jade transformed into a call tree as something generic to it though ; a bit like http://mochi.github.io/mochikit/doc/html/MochiKit/DOM.html was doing

[jeromew]

From @ForbesLindesay on November 6, 2015 11:56

I suppose all you would need from a plugin API is the ability to override buffer, bufferExpression, visitMixin and visitMixinBlock (you no longer need visitEach because we no longer generate an immediately invoked function for each blocks). We could create a simple plugin system that let you do that, but I think it would end up being brittle and would need to be replaced when we moved to babel anyway. A better plan (I think) would be to continue with your current inheritance model, which is actually pretty simple and effective.

[jeromew]

I'll look into that ; do you have a sample repository that shows how to derive a new compiler from jade 2.0 ? currently, the inheritance in then-jade is straightforward but I am not sure how to fork only jade-code-gen.

would you copy/paste https://github.com/jadejs/jade as a new project and start from there replacing jade-code-gen with the forked version ?

as a side question, I guess i would be interested in trying to rewrite jade-code-gen using babel as you suggested. Regarding coding conventions, have you seen a sane example project where AST is direcly built and fed into babel ?

[jeromew]

From @ForbesLindesay on November 6, 2015 13:32

You can create a new CodeGenerator class by inheriting from require('jade-code-gen').CodeGenerator. For now, I would copy paste https://github.com/jadejs/jade as you suggest, although we may come up with a better solution down the road.

No, unfortunately I'm not aware of any projects that do this already, but I've asked Sebastian McKenzie what he thinks and he felt it was a pretty sensible approach. I'll ask if he knows of any projects currently doing this/has any advice on how to get started.

[jeromew]

From @TimothyGu on November 13, 2015 2:36

I'll ask if he knows of any projects currently doing this/has any advice on how to get started.

Have you done so yet? Frankly, the notion of generating a fully qualified JavaScript AST from {} sounds pretty difficult, so an example would really help.

I also wonder how one can generate source maps with a different language, but I guess adding location information to the Babel AST should suffice.

[jeromew]

I looked a bit at the babel source code. Even if the code of jade-code-gen is not very big, I suspect that generating from {} will need some helper functions.

The basic node construction feature of the AST seem to be given by https://github.com/babel/babel/blob/master/packages/babylon/src/parser/node.js

It might be feasible with https://github.com/benjamn/ast-types but I did not check how compatible the ASTs are.

[jeromew]

From @ForbesLindesay on November 17, 2015 16:24

He wasn't aware of anything, but gave me a few pointers on where to get started. We should use:

• babel-types for building the AST (I have a pull request to generate some basic documentation at Generate documentation for babel-types babel/babel#2933, for now, you can read the docs in ForbesLindesay/babel) (we will also need to extend/wrap this to add location info to the nodes, but that shouldn't be very difficult).
• babel-traverse for running any optimisations/plugins on the generated AST. We can use this to run generic babel-plugins (for ES2015+ features an optimisations) as well as run custom plugins for doing things like inlining mixin calls and combining multiple separate calls to add strings onto the buffer. This will allow our own code-gen phase to be much more naive and then leave the work of building optimised code to babel.
• babel-generator to generate the JavaScript and source-map (We will need to fork/extend this to support source maps that originate from multiple files, but that's just a case of replacing babel-generator/src/source-map.js with something that takes the filename from each individual node)
• babylon for parsing user supplied JS expressions and statements.

[jeromew]

I have vaguely started to look at this (very low priority on my side).

While trying to see what I could do with the buffer() method, I wonder whether every new buffer addition should lead to a pug_html = pug_html + ".." or if we should keep the "smart concatenation" code that is inside buffer()

clearly, a babel plugin could take care of such concatenation afterwards but I wondered what you think because every babel plugin that we will add will have its own running cost.

[jeromew]

From @ForbesLindesay on September 12, 2016 15:3

I think we need to at least make sure that consecutive strings result in pug_html = pug_html + 'SomeString'; not pug_html = pug_html + 'Some'; pug_html = pug_html + 'String';. I'm not sure if we should bother reducing expressions, since it's definitely detrimental to performance after a certain point (I don't know what the optimum number of concatenations per statement is).

I don't have a strong opinion on whether this should be done as part of generating the initial code or in a babel plugin. I think it depends a lot on how difficult the plugin would be to write.

[jeromew]

I worked a bit on this again. I convinced myself that at this stage, it is better (if possible) to try and generate the closest javascript to the current generated javascript (it seems easier to me to compare the 2 generated javascript in parallel). It should be easy to refactor once we have test-parity.

Now I hit a problem because of how js strings are generated. In pug we use js-stringify to work out some escaping before inlining the strings in the output javascript.

With babel, declaring literals with t.stringLiteral("<p>"), we get strings with

"<p>"

instead of strings like those pug currently generates
"\u003Cp\u003E"

so we lose the escaping, js-stringify seems to not help with babel because t.stringLiteral("\u003Cp\u003E") produces code with double escaping of slashes.

Can you tell me more about why we do this escaping and if when generating the whole code from AST we would still need it ?

I currently don't see a way to have babel generator generate u-escaped brackets from literals the way js-stringify does; I would appreciate some feedback on this.

The closest issue I found is babel/babel#4478

note: maybe this has to do with incompatibilities when the generated js is written inside an html script tags

found on the web: HTMLEscape appends to dst the JSON-encoded src with <, >, &, U+2028 and U+2029 characters inside string literals changed to \u003c, \u003e, \u0026, \u2028, \u2029 so that the JSON will be safe to embed inside HTML script tags. For historical reasons, web browsers don't honor standard HTML escaping within script tags, so an alternative JSON encoding must be used

Extract found here

by the way it is weird that in this extract that choose \u003c, \u003e, \u0026, \u2028, \u2029 while js-stringify chooses \u003c, \u003e, \u002F, \u2028, \u2029.

it could maybe be a babel plugin used when generating literals (?)

[jeromew]

From @ForbesLindesay on October 27, 2016 9:23

This is an interesting point, that I had not considered. The reason we generate those characters, instead of < etc. is to ensure that it is always safe to embed a compiled pug template directly inside a <script> tag. In reality, the only sequence that needs to be escaped is </ (according to the spec). I think in actual browsers, (someone should check the latest spec) you only need to escape the sequence </script> so maybe we could check for that and either:

1. fork babel-code-gen to escape the characters in </script> when it outputs a string literal
2. or generate a concatenation so we output '<' + '/script>'

Option 1 should produce much more efficient code. This may be something babel is keen to offer, but I don't know.

I think long term we should avoid escaping everything like js-stringify does. I took the approach of escaping everything in js-stringify to be as safe as possible since I (and many other people) was using it to embed user generated info in inline scripts in JavaScript, i.e.

script.
  var CSRF_TOKEN = !{stringify(csrfToken)};

The overhead of this extra escaping probably does slow pug down and certainly makes for bigger/less readable output. We should try and minimise this overhead.

[jeromew]

ok. At this stage I think that I will not take this escaping into account. Since t.stringLiteral evaluates the string, there is no way I think to avoid the internal un-escaping. So it will have to be handled at code-generation (either by a fork or a PR on babel-generator)

babel generator uses https://github.com/mathiasbynens/jsesc for literal escaping and there is a isScriptContext option :

The isScriptContext option takes a boolean value (true or false), and defaults to false (disabled). When enabled, occurrences of </script and </style in the output are escaped as </script and </style, and <!-- is escaped as \x3C!-- (or \u003C!-- when the json option is enabled). This setting is useful when jsesc’s output ends up as part of a script or style element in an HTML document.

currently babel-generator does not allow to specify jsesc options - https://github.com/babel/babel/blob/master/packages/babel-generator/src/generators/types.js#L132

[jeromew]

for information i could force the escaping using an undocumented feature of stringLiterals via

 var lit = t.stringLiteral(this.lastBuffered);
 lit.extra = { rawValue: lit.value, raw: stringify(this.lastBuffered) };

not sure if we want to keep this is what babylon does when it parses escaped strings.