upgrade remark and parse wiki links (#265)

* add tests for reference links

- in preparation for parsing (but not resolving) wikilinks, add test cases for link references. 

* update remark and co; clean-up tests; add ofm fork

* pass basic slate tests; unwrapImages; re-name legacy functions

- update the markdown functions to use the new routines; re-name all  the old ones *Legacy
- make basic Slate tests pass, requires re-writing unwrapImages (trivial - this was a package before?)
- update consuming code to use new interfaces; it does not quite work yet


* more tests, configuration, clean-up

- add full markdown test; use it to slowly isolate edge cases
- clean-up and organize tests (continued)
- discover mdast-util-to-string configuration options, use them, update tests
- figure out some markdown escape rules, document and accept the behavior

* remove several remark packages, refactor types

- kick out all the legacy remark packages
- refactor types: drop ts-mdast, prefer types from mdast, refactor base Slate node import to customized Slate node

* fix image / paragraph wrapping + test

* slate-transformer cleanup

* strip wikilinks and fix a few tests

- add wikilink handling at the mdast->slate level, converting them to plain text. This results in them being ignored in slate and escaped when serialized back to markdown
- fix a few tests and remove some redundant (wikilink related) ones

Plan here is to support wikilinks on initial parsing so importer pathway can convert to markdown links / image embeddings, but then to ignore them thereafter. Future work can consider supporting them directly

* remove rehype

* also remove remark

Author: cloverich

.prettierignore

@@ -2,4 +2,8 @@
 build
 dist
 node_modules
-packaged
+packaged
+
+# Some of the changes prettier makes to md files differs from what
+# mdast-util-to-string does, making it unusable as a snapshot. 
+src/markdown/test-docs/*

package.json

@@ -11,7 +11,7 @@
     "postinstall": "yarn run electron-rebuild",
     "prestart": "tailwindcss -i ./src/index.css -o ./src/index-compiled.css",
     "prebuild": "yarn run lint && tailwindcss -i ./src/index.css -o ./src/index-compiled.css",
-    "start": "node ./scripts/dev.js",
+    "start": "node ./scripts/dev.mjs",
     "pretest": "node ./scripts/test.mjs",
     "test": "mocha 'src/**/*.test.bundle.js'"
   },
@@ -47,7 +47,6 @@
     "@types/mocha": "^7.0.2",
     "@types/react": "^18.2.48",
     "@types/react-dom": "^18.2.18",
-    "@types/rehype-react": "^4.0.0",
     "@udecode/cn": "^29.0.1",
     "@udecode/plate": "^34.1.0",
     "chai": "^5.0.3",
@@ -61,6 +60,10 @@
     "lodash": "^4.17.21",
     "lucide-react": "^0.314.0",
     "luxon": "^2.4.0",
+    "mdast-util-from-markdown": "^2.0.2",
+    "mdast-util-gfm": "^3.0.0",
+    "mdast-util-to-markdown": "^2.1.2",
+    "micromark-extension-gfm": "^3.0.0",
     "mobx": "^5.15.4",
     "mobx-react-lite": "^2.0.7",
     "mocha": "^10.2.0",
@@ -71,13 +74,6 @@
     "react-day-picker": "^8.10.0",
     "react-dom": "^18.2.0",
     "react-router-dom": "^6.3.0",
-    "rehype-react": "^7.0.3",
-    "remark": "^14.0.1",
-    "remark-gfm": "^4.0.0",
-    "remark-parse": "^8.0.3",
-    "remark-rehype": "^10.0.0",
-    "remark-stringify": "^8.1.1",
-    "remark-unwrap-images": "^3.0.0",
     "slate": "^0.101.5",
     "slate-history": "^0.100.0",
     "slate-hyperscript": "^0.100.0",
@@ -86,9 +82,7 @@
     "tailwind-merge": "^2.2.1",
     "tailwindcss": "^3.4.1",
     "tailwindcss-animate": "^1.0.7",
-    "ts-mdast": "^1.0.0",
     "typescript": "^5.3.3",
-    "unified": "^10.1.0",
     "yaml": "^2.5.1"
   }
 }

scripts/dev.js scripts/dev.mjs

@@ -1,7 +1,7 @@
-const esbuild = require("esbuild");
-const cp = require("child_process");
-const electron = require("electron");
-const lodash = require("lodash");
+import cp from "child_process";
+import electron from "electron";
+import esbuild from "esbuild";
+import lodash from "lodash";
 
 // Builds and watches the application for development
 // Bundles the renderer, preload, and main processes, and

src/markdown/remark-slate-transformer/README.md src/markdown/README.md

@@ -1,3 +1,5 @@
+# remark-slate-transformer
+
 This is a partial fork of https://github.com/inokawa/remark-slate-transformer
 
 I did this to quickly modify some of the output node types when converting mdast to slate.
@@ -7,3 +9,7 @@ If it works, consider formally forking, or opening a PR to the original repo and
 https://github.com/inokawa/remark-slate-transformer/issues/37
 https://github.com/inokawa/remark-slate-transformer/issues/31
 https://github.com/inokawa/remark-slate-transformer/issues/67
+
+# mdast-util-ofm
+
+Partial fork of https://github.com/MoritzRS/obsidian-ext initially because of ESM import issues. I need no immediate modifications of this project and if / once this project completes esm updates may be preferable to move this back to a dependency.