Extra HTML to Markdown Work to fix bugs

[mellenberger]

For the issue of escaping the '<>' symbols for the HTML to render correctly, we have come up with two separate possible solutions.

1. The first solution is on the client side, in the CodeChatEditor.mts file. A new rule was added to the turndown service to attempt to escape only the necessary symbols. This solution leaves the escaped versions of the symbols in the source code & also escapes the '<>' symbols within code sections. The escaped HTML does not render correctly within code sections, so this solution is not fully working as of now.

2. The second finished solution is on the server side, in the webserver.rs file. A regex was used to escape some of the '<>' symbols during the markdown-to-HTML process. All of the '<>' symbols not within code sections (between backticks for markdown) are escaped so they are not read as HTML tags. The '<>' symbols within code sections are not escaped due to them not rendering correctly if escaped.

Other minor fixes from the previous PR:

• For HTML-to-markdown conversion, the anchor tag is now kept as HTML.
• The smart punctuation option has been turned off within the markdown-to-html pulldown-cmark parser. This should stop future conversions to smart punctuation within the source code.

[mellenberger]

Also, not sure what happened with the commits for this PR, but the only new ones are the last four. The changed files look to be correct though and show only the changes from the last four commits.

[bjones1]

Thanks for working on this! I'll take a look, but probably not until next week.

[bjones1]

I'm continuing to work on this, looking to integrate it into the main branch. Thoughts/comments:

• One big problem is turndown's bug; it doesn't escape HTML tags. I sent a PR to fix this: Fix: correctly escape text that would otherwise be interpreted as raw HTML and HTML blocks. mixmark-io/turndown#438.
• Another concern: it looks like there are a number of Turndown bugs, some with good fixes, but the maintainer isn't addressing/merging these. See e.g. Lists over 9 items long break with indented content mixmark-io/turndown#410, heading inside anchor renders to invalid markdown mixmark-io/turndown#409. I think the maintainer is tired of the project.

Another approach: abandon this and use Pandoc instead. Pro: well-maintained and active project. Cons: requires users to install Pandoc; requires work to interface with the Pandoc server.