v1 - early feedback

[pngwn]

this discussion will form two parts. this one is about general goals and the API; another will be about markdown specifically. in some ways, the two are inextricably linked, and the markdown discussion should arguably come first, but i need to do more research until i can discuss that intelligently

i've put off publishing a v1 for far too long, but it has allowed me to change things pretty aggressively, and since the initial PoC a few years ago, i have completely rewritten mdsvex (twice, one never got published), learned a great deal, and got some excellent feedback.

taking all of the things i have learned along the way, I am now starting to think about a 1.0 and what that might look like (spoiler: there will be breaking changes).

---

story time...

when i originally wrote mdsvex, i didn't really know how people would use it, but i did have a use-case in mind. i mainly imagined it would be used to create interactive blog posts and articles. in reality, there are a wide variety of uses.

the addition of layouts really unlocked some doors for creative uses without writing lots of custom logic in the mdsvex documents themselves. layouts allow users to abstract that stuff away and focus on the authoring experience. but in addition to this @sw-yx raised the idea of mdsvex as an alternative component authoring experience, the addition of custom components made this a viable approach and something i've done myself for simple page-level components.

so in that way mdsvex is svelte-in-markdown but also markdown-in-svelte, it is up to you.

regardless the main goal remains the same: make the authoring experience as good as it can possibly be and let the user focus on content, not implementation.

about that...

the authoring experience with mdsvex can be really good, but my god, do you have to do a lot of setup. this isn't a bad thing, the tool is flexible and can cover lots of use-cases, but i have also found it frustratingly involved to setup some pretty basic features.

stuff like slug-ids for headings, clickable links for those titles, getting a ToC (either printing into the document or as data to use in layout files etc.). These are pretty common requirements, @sw-yx alludes to them in some of his writings on markdown. I could just include those things by default.

but there is a flip side to that; not everyone writes blog posts or docs with mdsvex. forcing something like slugs or links or ToCs would be incredibly annoying. making them available via a flag is a great way for the interface to explode in complexity (there are already too many options, in my opinion).

and the flip-flip side is that a few features probably aren't enough, like really good defaults. there is more a tool like mdsvex can do for specific cases. i really get to a place where most people can just do this:

preprocess: mdsvex()

i think we are quite a way off that right now.

---

so version 1...

there are some things i'm almost definitely going to change. the markdown syntax that is supported is probably going to be one of these. various markdown standards pose a number of challenges, but that is part 2 that i mentioned earlier. my thought is too unrefined to discuss that at present. this is an important factor though, some of my decision rest on the assumption that the supported markdown syntax will be a touch stricter.

with this context in mind, i'll be moving away from the unified ecosystem. this means no more remark or rehype plugins. there are a few reasons for this: performance, API instability, poor documentation, possible changes in what kinds of syntax extensions are allowed. this means i need to write some parsers; in reality a bunch of this work has already been done, at least on the svelte side. the markdown parts are pending more investigation.

no unified means no plugins, though, so there will be a new plugin system. as dangerous as plugins are, tools like mdsvex are used in such a variety of ways and often form a core part of the infrastructure of app frameworks and app developments that there needs to be some level of configuration. i am thinking that most of the current configuration can be removed entirely, with the core api mostly concerning itself with initialisation. this would require a powerful plugin system but would simplify the interface significantly. there are other benefits too.

you may be wondering how removing all of the config (which supports good defaults) is going to mark setup easier, which i said was a shortfall earlier. presets.

---

presets?

yes.

almost every mdsvex config option and almost every feature request can be implemented as a remark plugin. in the future, custom syntax will probably not be supported, but AST transforms will. The AST spec will be well documented and form part of the official API, something like SVAST which borrows heavily from unified's MDAST/ HAST. there will also be a series of utilities to make transforming these trees more straightforward.

a preset can be considered a series of pre-composed plugins that offer everything you need for a specific case.

there could be a blog preset that comes with syntax highlighting, slugs ids for titles, anchors for heading links, title/ navigation generation for us in ToCs/ navigation, frontmatter.

there could be a mathy preset that comes with stuff like latex support.

import { mdsvex } from 'mdsvex';
import { blog } from 'mdsvex/presets';

export default {
  ...
  preprocess: mdsvex(blog) 
  // or mdsvex({ presets: [ blog ] })
  ...
}

or

import { mdsvex_blog } from './mdsvex';

export default {
  ...
  preprocess: mdsvex_blog()
  ...
}

you could even combine multiple presets.

import { mdsvex } from 'mdsvex';
import { blog, stuff } from 'mdsvex/presets';

export default {
  ...
  preprocess: mdsvex({ ...blog, ...stuff ) 
  // or mdsvex({ presets: [ blog, stuff ] })
  ...
}

or you could use the core library with none of those things and build your own.

presets are interesting because they are shareable too; you could essentially publish your own config to be used by others. we could even create a registry to share pre-built configurations. a website could be created to share both official plugins (of which there will be many), user-created plugins, official presets, and user-created presets.

---

in many ways, my aim in v1 is to strip everything right back down and then open it back up again in ways that are currently very difficult. but more importantly, it is about taking what we have now and allowing even more community involvement and interaction. i've learned a huge amount from mdsvex users and seen some pretty incredible things; it is a shame that it is very hard to share these amazing setups that people have created in their applications. presets provide a path to that, but they aren't without their pitfalls, it could be a disaster and, as such, will require some thought.

---

There will be other changes too, things like switching to shiki for default highlighting. writing some more tooling (like prettier plugins and ide extensions), and improving the documentation.

---

i'd love everyone's thoughts on this. don't hold back. this isn't a new idea, and there are plenty of challenges, so if you have had good or bad experiences with plugin systems and presets in the past, i'd love to learn more.

[vedam]

Hey pngwn,
Just want to give some feedback on this from my use of mdsvex. It's my No.1-Tool. I'm currently going all-in on mdsvex-pages with my personal portfolio-site while I'm updating from sapper to svelte-kit.

On configuration
In my experience it was not difficult to setup mdsvex and get it up running. Things got complicated for me as I wanted i.e. to make a file-list for an index-page and extract the frontmatter-data for that. But these are all more problems with my lack of knowledge than the mdsvex-side of things. OK, I didn't try to throw in more remark- or rehype-plugins, I imagine that things can get messy quickly then. You rarely check what you bring in with these plugins.
So, the easier the better. Less parts involved = less errors.

On presets
I'm currently have only one remark-plugin remark-attr in use which helps me to easily add styles to the markup. Nothing else.
Other things like custom animated titles, custom links or an image-wrapper I was able to easily append with my own svelte-components.
So even if I would plan to make a blog, I'm not sure if I need the mentioned blog-preset containing x, y and z-features.
Wouldn't it make more sense to even break these down to the feature-level (or addons, plugins)?

import { toc, attrs, x, y, z } from 'mdsvex/features';

export default {
  ...
  preprocess: mdsvex({ ...toc, ...attrs, ...x, ...y, ...z  ) // use them
  ...
}

This doesn't exclude to provide recommended presets.
A possible Blog-Preset:

import { toc, slug_ids, attrs, title_links } from 'mdsvex/features';

export default {
  ...
  preprocess: mdsvex({ ...toc, ...slug_ids, ...attrs, ...title_links  ) // use them
  ...
}

[pngwn]

Yeah, my thinking is that presets would just be a collection of plugins. All of those plugins would be available individually as well.

[vedam]

heh, two sentences to summarize all my lyrics. Next time I'll do better. 😄

[swyxio]

appreciate the amount of deep thinking going into this!

i honestly dont see a lot of detail to comment on - sure thing on the presets and stuff, and the rehype ecosystem was annoying to figure out at first but now has enough usage that it actually does have a lot of decent stuff out of the box.

i trust your judgment though, as long as you agree with the usecases i discussed i'm sure i'll love whatever you land on

[wlach]

This is a pattern that I've seen before: using general-purpose libraries and tools to build new things can make sense up to a point, but eventually to enable more control and/or new features you really need to build something new more adapted to your problem domain. I'm sure there's a name for this.

My use case: I've been using MDSveX as a way to create a library for creating interactive scientific notebooks (a bit like RMarkdown). In the interests of not creating yet-another-file-format I've been trying to make it process the MyST markdown format -- starting with the subset around executing code chunks. The existing remark/rehype plugin system exposed by MDSvex (once I learned about it, earlier prototypes didn't use it) was plenty powerful enough to build what I needed for this (prototype) phase of the project (code). I'm only currently parsing a small subset of MyST, but so far it's been working well, albeit a little inelegantly in places.

The architecture you're describing sounds like it'll continue working for me (maybe even a little more cleanly than what I have now), although it's hard to say for sure until I've seen the API / code.

[pngwn]

Yeah, this is mainly about the approach/ philosophy. In terms of implementation, this approach can definitely be made to work for a variety of different usecases and will be no less flexible than the current API and hopefully a good deal simpler. The newest version of the remark/micromark/unified/whatever-it-actually-is API is incredible complex and mostly undocumented. mdsvex does not use that and never will.

I will be collecting usecases soon in order to come up with a proposed API for this.

[pngwn]

On that note: #270

[pngwn]

If you are building stuff with mdsvex please go take a look at #270 and comment if possible.

[michaeloliverx]

I would love to have hooks into various stages of the preprocess pipeline, an example I can think of is modifying the metadata exports.

[pngwn]

What kind of modifications might you want to perform? Do you have an example of this?

[michaeloliverx]

This comment highlights a usecase I would love to have for declaring cover images. I would also like to be able to use regular markdown image syntax and have that transformed into a <picture> element with multiple srcsets. The latter would require an additional external library to perform the image transformations but having the necessary hooks would be nice.

[michaeloliverx]

Shortcode support would be amazing, being able to transform a shortcode into the necessary script imports and template syntax would allow for easier integrations with a CMS and allow the content to be technology (Hugo, MDsveX etc) agnostic.

[TheOnlyTails]

What if we could use retext plugins (along with remark and rehype)?
mdsvex could find all the plain text (not codeblocks, etc) and run retext on them.
It may be a bit unconventional, but it could be a cool idea.

[pngwn]

1.0 will not be using the unified ecosystem, so this won't be possible but writing a plug-in to process text nodes will be trivial. Porting retext plug-ins will be simple in list cases.

[TheOnlyTails]

Hmm, why not? It's been very nice to use remark/rehype plugins directly with mdsvex.

[jamesb93]

FWIW a few years later, relying on those plugins is kind of brittle. I just got burned where a maintainer won't update a plugin and I would prefer to write my own, given there was a concise API around it intended for mdsvex.

[lenianiva]

I concur with this. unified is quite brittle

[akauppi]

My use case is an interactive playground for a JS component I’m working on, but now that I know about MDSveX (heh, what’s the right want to type it?) I’ll likely be using it in any SvelteKit project that is text intensive.

Your suggestions suit great for me - and thanks for pouring out the thoughts in such a readable fashion. Enjoyed it! 🍿

[dev-msp]

I just wanted to call out @pngwn's comment in Discord here:

"I'm hoping this way I can apply Svelte components to only the dynamic of the parts tree."

mdsvex will do this in the next version [...]

I think/hope this is exactly what I'm looking to achieve with MDsveX - ultimately my goal is to be able to pre-render a static blog such that the document loaded by the browser doesn't hydrate the full text of the blog post, and thus load it twice: once in markup and another in the "fetched" data for hydration. Distinguishing between the static and dynamic bits of the tree would seem to solve this problem, or else I'm way off base.

Only mentioning it here to add my support for this particular feature!

[aejenk]

Hey! On my end I wanted to vocalize my thoughts regarding moving away from the remark/rehype ecosystem.

Specifically because my site is currently using Nuxt 3, with plugins such as remarkMath, remarkReadingTime, etc.

My worry here is that a lot of functionality will end up being lost by moving away from that ecosystem. There could be any number of functionality - such as readingTime for example - which takes time to get covered. For example, currently due to the outdated version of remark used by mdsvex - I'm left unable to use the reading time functionality, although math functionality still works by using v3.

In essence, I'm wondering how this problem of sorts would be tackled. Is the thought that a new system will be created that's better documented/easier to use - hopefully making it so that plugin replacements can be written quickly?

To add - and I want to make sure on this point since you've been explicit - upgrading remark/rehype is not going to happen correct? I might need to abandon certain functionality if that's the case, but it's not critical - purely cosmetic.

Other than this, I really like the idea for presets, and sticking with an existing AST model is nice to hear since it'll include less of reinventing the wheel. Part of me does wish for a workaround until v1.0 eventually comes, but I'm excited nonetheless.

[artemkovalyov]

@pngwn, is this still alive? Are you talking to Svelte/Svelte Kit folks to join efforts?

[pngwn]

Very alive. I don't like the svelte ppl, they smell.

[clearfram3]

I want to use this for a rewrite of the typescript website into a SvelteKit project. When do you expect to drop next version? Any places you think people new to the code could help or get a foothold in?

FYI, I am using svelte 5 syntax

[riziles]

What about LaTeX? Would there be a way to do that without Unified?

[pngwn]

Latex/ katex support isn't tied to unified. Math support will be first class, no plugins required.

[david-merryweather]

Will there be a way to add mhchem functionality to the mix? David Merryweather | VP of CloudOps and SRE | T 512.646.4971 Cloud Technology - Technology Education Pronouns | (He|Him|His) Who We Are <https://www.macmillanlearning.com/college/us/our-story> | Our Community <https://community.macmillanlearning.com/t5/macmillan-community/ct-p/college> | Diversity, Equity & Inclusion <https://www.macmillanlearning.com/college/us/our-story/diversity-and-inclusion> | Our Student Store <https://store.macmillanlearning.com/us?_ga=2.170182880.1146378192.1615214800-546528760.1614987034> <https://www.macmillanlearning.com/college/us/our-story>

…

On Sat, Mar 2, 2024 at 6:03 PM pngwn ***@***.***> wrote: Latex/ latex support isn't tied to unified. Math support will be first class, no plugins required. — Reply to this email directly, view it on GitHub <#259 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF3KYJPBKXRGXDNVJ6ABQR3YWJSEZAVCNFSM45DG2AU2U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQNRVGQ3DSNQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[pngwn]

Yeah, that should be possible.

[clearfram3]

Do you have a release timeline?

[pngwn]

Nope. Will be done when it's done.

[clearfram3]

Any particular issues/pull-requests I could work on to get it done faster? I wanted to showcase your library in the TS website redo, so have a general weeks/months ballpark helps

[pngwn]

Not at the minute sorry.

[codevogel]

Have you seen my post here?
#593