Converting HTML to markdown doesn't appear to preserve HTML entities

[sebpowell]

Consider the following HTML example:

<p>I think &amp;</p>

When I try converting this to Markdown using Turndown, I get the following output:

I think &

I guess I would expect Turndown to preserve HTML entities and to output something like this instead:

I think &

I couldn't see an option to turn this on, so unless I'm missing something, I assume I need to use something like https://www.npmjs.com/package/html-entities. But I just wanted to check I'm not missing anything obvious?

Here's the config I'm using:

const INITIAL_TURNDOWN_OPTIONS: Turndown.Options = {
  headingStyle: "atx",
  hr: "---",
  bulletListMarker: "-",
  codeBlockStyle: "fenced",
  fence: "```",
  emDelimiter: "_",
  strongDelimiter: "**",
  linkStyle: "inlined",
};

Any help much appreciated!

[bjones1]

Using the CommonMark dingus, entering I think & or I think & renders to <p>I think &amp;</p>. So, the HTML entity in the HTML source doesn't need to be preserved in the resulting Markdown to still render properly. Are you asking for a way to preserve HTML entities, even if they don't need to be preserved to render correctly?

[Aloso]

@bjones1 it does need to be preserved in this case:

<br>

which is converted to

<br>

and in this case:

& is an ampersand

and in this case:

A big   space

and in this case:

    Not a code block

[jeacott1]

not just html entities, but an option to ignore already escaped html would be very helpful.

[bwat47]

Using the CommonMark dingus, entering I think & or I think & renders to <p>I think &amp;</p>. So, the HTML entity in the HTML source doesn't need to be preserved in the resulting Markdown to still render properly. Are you asking for a way to preserve HTML entities, even if they don't need to be preserved to render correctly?

It's not much of an issue in the & example, but already escaped html (that's not inside code) can cause some annoying issues (depending on the HTML tag) after conversion to markdown, e.g. this is a real world example I came across:

<p>this is a &lt;table&gt;</p>

Which gets converted to:

this is a <table>

So if your markdown renderer supports HTML, it will attempt to render a table

I currently handle this with manual DOM pre-processing to wrap already escaped HTML like tags in code before turndown conversion

[bjones1]

I've applied several fixes/improvements to the code; see my repo for the latest. Using that, converting

<p>&lt;table&gt; works. &amp; works. Big &nbsp; works.</p>

to Markdown yields

\<table> works. & works. Big   works.

where the extra space in the middle of the sentence is actually a U+00A0 : NO-BREAK SPACE [NBSP].

If you have Visual Studio Code, you can install the CodeChat Editor extension I wrote, open a Markdown file, hit ctrl+shift+p, type "CodeChat Editor" to select "Enable the CodeChat Editor", then edit it live to see this working.

In the screenshot below, I'm editing a Rust source file (the CodeChat editor works on code) with the test above.