Compatibility with odoc-parser.2.3.0

[panglesd]

This PR makes ocaml-lsp compatible with the soon release odoc-parser.2.3.0

odoc-parser adds two new constructs:

• "Outputs" of code blocks. This is meant to be generated by the code blocks and promoted by mdx.
• Tables

Tables are tricky, as odoc tables are allowed to have some block content inside ("nestable" block contents) while markdown is not.
This means that when translating odoc tables into cmarkit tables, we need to turn blocks into inlines. This issue is responsible for most of the code of the PR!

[panglesd]

I'm converting to draft as this should only be released when odoc-parser 2.3.0 is released.

[jonludlam]

odoc-parser.2.3.0 is now released.

I'd like to suggest that the parser is vendored in order to increase compatibility. odoc-parser is also vendored by ocamlformat and mdx for similar reasons. Right now we can't co-install odoc and ocaml-lsp-server because of this.

[Julow]

This PR is not a draft and is ready to be reviewed.

[Julow]

This function might be a lot of maintenance and hurts the content.
Would it be reasonable to use raw HTML tags for tables ?

[panglesd]

I agree that this function is not ideal...

An advantage of not using proper table compared to raw html is that the output might be more readable even without translating it to html. Also, we get cmarkit type guaranty.

Raw html tags could be an idea, but their start-end semantics not that simple (https://spec.commonmark.org/0.30/#html-blocks).

In particular, it seems to break out of the containing block. For instance,

- first item
  + first subitem
  + second subitem

renders by github as:

• first item
  • first subitem
  • second subitem

while with html inside (the blank line is the end condition fot the uninterpreted html)

- first item
  + first subitem
<table><tr><td>

useless table
</td></tr></table>

  + second subitem

renders as

• first item
  • first subitem

useless table

• second subitem

@voodoos would you like me to investigate the "raw html tag" solution?

[voodoos]

In particular, it seems to break out of the containing block

So in the case of tables the content would break out of the table ?
It looks like raw html blocks are only intended to be used at the top level...

Would using only raw html (for the entire table, not only the cell content) be an option ?

[panglesd]

So in the case of tables the content would break out of the table ?

What I meant is that the whole table would break out of its container block (recall that all of this is because, in odoc, tables are allowed to be nested in blocks, while it is not allowed in markdown).

For instance, in the example above, I had in mind a table inside a list (which is possible in odoc).

Julow suggested (IIUC) to render in this case the table, not by using markdown tables, but by embedding raw <table>/<tr>/<td> before the table/line/cell, and the corresponding closing tag after.
The markdown example above was thus the translation of a (nested) list containing a table, using this method.

However, we can see, from how github renders it, that in fact the table is not interpreted as inside the list, but "toplevel", and thus also breaks the rest of the list.
(I included a nested list to make it more visible that the included html breaks the flow of the list interpretation, but it might have created noise!)

It seems that cmarkit type system won't enforce constraint on raw html embedding, which is frightening!

[voodoos]

So that's not suitable since it would also break more visibly things like numbered lists (see below).
So far I think the more convincing workaround is your current proposition.

- first item
  1. first subitem
<table><tr><td>

useless table
</td></tr></table>

  2. second subitem

• first item
  1. first subitem

useless table

2. second subitem

[panglesd]

Update: The example was wrong, it is in fact possible to have html inside list items in commonmark.

(With the help of cmarkit) I could generate the right markdown for the example above:

So:

1. first item
   1. first subitem
      <table><tr><td>
      
      useless table
      </td></tr></table>
      
   2. second subitem

renders as

1. first item

   1. first subitem

      useless table

   2. second subitem

So I think the suggestion to use html is still valid, and the discussion open again. Speaking of tables, let me use one to summarize the differences:

	Current approach	Html embedding
Source	🟠 Source is not very readable due to tables not properly formatted, and blocks turned into inlines	🟠 Source is not very readable, due to html tags and blank lines added all over the place
Faithful rendering	🔴 The content is modified	🟢 The content is preserved
Compatibility	🟠 The renderer needs to know about tables, which is not in the commonmark standard	🟠 The parser needs to follow the commonmark standard, which is different from the original one if I understand correctly
Rendering target	🟢 No constraint	🔴 Has to be HTML/markdown, otherwise (eg in latex) embedded html will certainly be ignored
Scariness	🟢 Not scary, a problem won't leak outside of the table	🟠 Html embedding is a litlle bit scary as making a mistake can break the whole output...
Maintainability	🔴 Adds maintenance	🟢 Does not add maintenance

If the "rendering target" can handle embedded html, maybe the html embedding approach is better? The "content is preserved" is quite a strong benefit, I guess...

[voodoos]

According to the specification, LSP clients that have the capability to parse markdown should follow the Github Flavored Markdown Specification.

That plays in favor of Html but they also warn that:

Please Note that clients might sanitize the return markdown. A client could
decide to remove HTML from the markdown to avoid script execution.

[voodoos]

And there is more: the client can advertise which tags will actually be allowed in markdown with the "MarkdownClientCapabilities.allowedTags" capability.

If we want to respect the protocol we need to have at least two fallbacks:

• A text-only version when markdown is not allowed. (Right now ocaml-lsp simply prints the raw comment which is okay)
• A markdown without html when required tags are not allowed

And we could do better by using raw html when the client accepts it.

In any case we need to keep the current version, and I'm not sure having the additional html one is really worth it ?

[panglesd]

In any case we need to keep the current version, and I'm not sure having the additional html one is really worth it ?

I agree with this analysis, and I do not think it is worth it: in the current approach, the content is only hurt in the case of table/lists inside tables, which I think will happen very seldomly.

If it turns out a problem, we can add the html approach later.

[panglesd]

The test is failing because the test depend on ocamlformat 0.24.1, which depends on odoc-parser >= "2.0.0" & < "2.3.0". So the dependencies and test dependencies are not co-installable!

The solutions are:

• Switch to a higher ocamlformat version where odoc-parser has been vendored, or
• vendor odoc-parser.

In any case, I guess it would make more sense in another PR?

[voodoos]

I'd like to suggest that the parser is vendored in order to increase compatibility

I am not fond of vendoring, but this looks like a legitimate use case. As we see with this PR, reducing co-instability with ocamlformat which is a commonly used tool along ocaml-lsp might impact a lot of users.

In any case, I guess it would make more sense in another PR?

I think we should do the vendoring in that PR, rename it Upgrade to odoc-parser 2.3.0 and allow the CI to run correctly.

[panglesd]

I think we should do the vendoring in that PR

Then I'll do that!

Just to be sure, would a vendoring in the style of dune or mdx, with a vendor directory and an update-odoc-parser.sh script downloading the sources from a given hash, be good for this project? Or (as I'm seeing a submodule directory) would you rather have the vendoring made with git submodules?

[voodoos]

Just to be sure, would a vendoring in the style of dune or mdx, with a vendor directory and an update-odoc-parser.sh script downloading the sources from a given hash, be good for this project? Or (as I'm seeing a submodule directory) would you rather have the vendoring made with git submodules?

@rgrinberg do you have a preference ?

[rgrinberg]

Just copying the part that you need is good enough for me. Submodules are more useful when you're also developing the vendored component in conjunction with this repo. I don't have the intention to do that, and I assume you two don't either.

[rgrinberg]

By the way, isn't this a breaking change in odoc-parser? Shouldn't there be a major version bump.

I also don't see anything in the change log regarding the breaking change.

[panglesd]

The odoc parser was in a separate repo (https://github.com/ocaml-doc/odoc-parser, the README needs to be updated to reflect this) but it was merged back in the odoc repository in ocaml/odoc#973.

Currently, the version number from odoc and odoc-parser are tied together (so, 2.3.0), but I cannot find a reason for this in the past discussions. I can only find:

We're in the happy position that the current releases of odoc-parser and odoc are such that we can coalesce the version numbers so we'll have 2.3.0 as the next release of both, which is one less thing to worry about.

but I agree that it's not exactly true, since odoc-parser needed a major bump. There is also ocaml/odoc#973 (comment) which opens up to untie the odoc and odoc-parser versions later, if I understand correctly.

[panglesd]

In dune's vendoring, all the dune files I took inspiration from (such as this one) have (wrapped false).

I'm not sure if it's needed, but it still compiles without it..

[rgrinberg]

It's not needed. It's needed under the following circumstance: the code is being vendored in a library so we need to avoid name collisions with the original library if we link it to the same binary.

In this case, we're using this odoc parser only in ocamllsp, so we can drop it.

[panglesd]

The vendoring is ready for review. The compatiblity patch needs a new conclusion to the thread started by #1184 (comment) !

[rgrinberg]

we don't need this shim. we don't support versions of OCaml that don't have the result module.

[panglesd]

I opened ocaml-opam/opam-repository-mingw#12 for the windows CI failure.