How to stringify a parsed Markdown AST as-is (preserving the original source) while changing only some nodes

[ole]

Intro

I'd like to use remark (or the underlying mdast-util-from-markdown and mdast-util-to-markdown) to selectively edit thousands of Markdown files while keeping the bulk of the original Markdown sources unmodified. In other words, my code should only modify the parts of the Markdown that I actually want to change (e.g. the contents of link nodes) while keeping the rest of the text as-is.

In my experiments (see below for an example), I found that mdast-util-to-markdown "normalizes" the Markdown it outputs, e.g. it will use a consistent style for bullets, emphasis, strong, thematic breaks, etc. I know I can configure which of these styles to use via options, but that still wouldn't do what I want. I'd like to keep the bullet/emphasis/strong etc. styles in the original Markdown sources, even if that means I have to live with inconsistencies etc. I understand that normalizing the output is often a good idea, but in my case I feel that it actually makes my life harder, if only because it creates noisy diffs that make it hard to distinguish cosmetic changes from the material edits I do want to make (and which I want to check by looking through the diffs) Again, we're talking thousands of files, so no noise really matters.

Example

In the following example (written in TypeScript for Deno), my input is a Markdown source text with several incongruities and inconsistencies (multiple strong and emphasis styles, a somewhat unusual format for thematic breaks etc.):

Heading
=======

[Link 1](http://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [link 2](http://example.com)

I want my code to selectively edit the "link" nodes (by changing "http" to "https") and then write the modified tree back as Markdown. (My actual problem uses different AST transformations, e.g. adding a field to frontmatter. But the argument is the same.)

// rewrite-markdown.ts
//
// This is a script written for Deno. You should be able to run it with:
//
//     deno run rewrite-markdown.ts

import type * as Mdast from "npm:@types/mdast"
import * as From from "npm:mdast-util-from-markdown"
import * as To from "npm:mdast-util-to-markdown"
import * as V from "npm:unist-util-visit"

const input = `Heading
=======

[Link 1](http://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [link 2](http://example.com)
`

// Parse Markdown to AST
const tree: Mdast.Root = From.fromMarkdown(input)

// Walk AST and edit link nodes
V.visit(tree, "link", (node: Mdast.Link, _index, _parent): V.VisitorResult => {
  if (node.url.startsWith("http://")) {
    node.url = node.url.replace("http://", "https://")
  }
  return V.SKIP
})

// Write modified AST back into Markdown text
const output = To.toMarkdown(tree)
console.log(output)

Actual vs. desired result

deno run --check rewrite-markdown.ts

This prints (actual result):

# Heading

[Link 1](https://example.com)

***

* **Lorem** **ipsum**
* *dolor* *sit* [link 2](https://example.com)

I would like it to print (desired result):

Heading
=======

[Link 1](https://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [link 2](https://example.com)

I.e. preserving all idiosyncracies in the original source except for the link nodes I edited; the only change compared to the input is "http" to "https". As you can see, the actual output is normalized Markdown, e.g. it uses a consistent style for bullets/strong/emphasis etc. Which is not what I want.

Possible solutions

One idea I had was to install custom toMarkdown handlers for all nodes that I don't want to change. These handlers would then take the node's position information to copy the text directly from the input text into the output. For example, if we want to preserve the original text of "thematicBreak" nodes in the output, we can change the code:

  // rewrite-markdown.ts
  …
  // Write modified AST back into Markdown text
- const output = To.toMarkdown(tree)
+ const toMarkdownOptions: To.Options = {
+   handlers: {
+     "thematicBreak": (node: Mdast.ThematicBreak, parent, state, info): string => {
+       // Slice the relevant segment directly out of the input text
+       return input.slice(node.position!.start.offset, node.position!.end.offset)
+     }
+   }
+ }
+ const output = To.toMarkdown(tree, toMarkdownOptions)
  console.log(output)

Now, the original format of the thematic break is preserved in the rendered output. Great!

This approach seems relatively trivial (if a little tedious to write) for leaf nodes of the tree. But I don't know how to achieve the same for nodes that can have children. E.g. in my example, the "listItem" nodes should generally render all their children (and themselves) as-is, unless one of their children is a "link" node.

I feel this approach could potentially work, but I don't understand the mdast APIs well enough to make it work. Any ideas?

Or are there other solutions you can think of?

Thanks for reading, and thank you to everybody who works on these packages. I love the unified/remark/micromark ecosystem!

[ChristianMurphy]

Duplicate of https://github.com/orgs/remarkjs/discussions/1067, https://github.com/orgs/remarkjs/discussions/1194, and micromark/micromark#59

remark is an AST library

AST is short for Abstract Syntax Tree, the Abstract part focuses on simplifying working on structure, intentionally glossing over or normalizing stylistic parts of the language, like spaces or which list marker is used which don't change how the document will be structured/displayed (https://en.wikipedia.org/wiki/Abstract_syntax_tree)

What you are describing is a Concrete Syntax Tree (https://en.wikipedia.org/wiki/Parse_tree), which could be built on top of micromark (syntax-tree/mdast#36 (comment)), but would have a completely different structure and way of working from remark or rehype.

[wooorm]

You could do things with the positional info. It contains positional info about where in the original file things happened. You can then change only things at particular positions, yourself.