feat(markdown): add support for math rendering with typst and katex #2791
Conversation
cestef
force-pushed
the
feature/math-rendering
branch
2 times, most recently
from
bad4e13 to
5d8d725
Compare
|
I don't really know how to get rid of these |
|
This PR adds math rendering on the "server-side" with typst, while keeping the katex option open for latex. A The output SVGs generated by typst can also be minified via the svgo CLI. This is kind of a hacky implementation because it relies on the CLI but it does the job. Newly added configuration parameters: math = "typst" # Defaults to "none", one of "typst", "katex"
math_svgo = true # Optimize the SVGs generated by the math renderer
math_svgo_config = "svgo.config.mjs" # Optional, Path to the svgo config
math_css = "styles/typst-embed.css" # path to the CSS file to inject inside the SVGs generated CI is failing because The size of the binary also drastically increases because of typst's ecosystem ( |
|
This is great work! One possible place to improve: the current implementation outputs SVG for equations. It might be better to use MathML to make sites smaller and more accessible. |
|
Thanks a lot :D Typst doesn't yet support HTML or MathML output (see typst/typst#5512) The basic katex implementation however already outputs HTML I think. I don't use LaTeX that much so if anyone is up to help with this, I'd be glad. |
|
I think it might be possible to write a backend visitor to output MathML (cf https://github.com/wcshds/typst-math-to-mathml-converter). This would be a lot easier than what the Typst team is building, since we don't have to handle everything else, but the downside is that we're then implementing a possibly buggy different spec from what Typst does. As a stopgap for accessibility, we might also just include the math source as alt text for the generated SVGs. |
|
This indeed sounds a bit risky, including the original source in the alt should be enough for now. I will tinker a bit with the library you linked to try out mathml rendering, I also stumbled accross this one: |
cestef
changed the title
cestef
force-pushed
the
feature/math-rendering
branch
2 times, most recently
from
d276f92 to
4d37093
Compare
|
I have tried out a bit the library you linked, here's the result I get:
I am using For reference, here's the result with SVG rendering:
|
cestef
force-pushed
the
feature/math-rendering
branch
from
4d37093 to
df62c23
Compare
There was a problem hiding this comment.
this looks great! i've left a few comments but i've been toying locally for a few days and i'm very happy.
+1 for svg+mathml too. it's always what i gravitate to and it always works well. one day soon mathml will be widespread and complete enough that it will become a viable primary target, but for now, svg rendering is the only consistent option.
on that note, did you ever consider mathjax? the latex support is a little better than katex these days.
| V: Serialize + for<'de> Deserialize<'de>, | ||
| { | ||
| cache_file: PathBuf, | ||
| cache: DashMap<K, V>, |
There was a problem hiding this comment.
i'm not totally sure whether it matters at the kind of scale/concurrency zola operates on, but you might find papaya a better fit for a concurrent hash map. this (biased) blogpost by the author totally convinced me.
components/markdown/src/markdown.rs
Outdated
| _ => FenceSettings::new(""), | ||
| }; | ||
| let (block, begin) = match CodeBlock::new(fence, context.config, path) { | ||
| let (block, begin) = match CodeBlock::new(&fence, context.config, path) { |
There was a problem hiding this comment.
performance wise, though it's somewhat (but not totally) moot, you might want to clone fence.language beforehand so you don't have to pay the larger cost of cloning inside CodeBlock::new.
components/markdown/src/markdown.rs
Outdated
| accumulated_block.clear(); | ||
| let inner = &accumulated_block; | ||
| match code_block_language.as_deref() { | ||
| Some("typ") => { |
There was a problem hiding this comment.
is there a reason katex isn't supported here too?
There was a problem hiding this comment.
I don't use latex/katex that much, but I think there's no concept of "raw" rendering in katex. From what I understood, it's only made for math rendering, whereas typst can render whole documents. the "raw" mode was initially intended for rendering graphs/tables. Let me know if katex supports such things!
|
Update:
[markdown.math]
engine = "typst" # typst, katex, or none
svgo = true # true, false, or "path"
css = "styles/typst-embed.css" # optional
addon = "helpers.typ" # optionalWe can also simplfy specify: [markdown]
math = "typst"Which is equivalent to [markdown]
math = {
engine = "typst"
}
[markdown]
cache = true # true, false or "path"By default the caches are now stored in
## Wonderful test
This is markdown content.
```typ
= Hello, World!
This is a raw typst document: $y^2 = x^3 + a x + b$
```
PS: Thank you so much for your feedback soqdb 😄 |
|
I wouldn't bother with the |
|
@soqb after benchmarking papaya and dashmap, it looks like the difference for our scale is very very small... Hot run (read heavy): |
|
I tried this PR, and used |
|
css attribute was for css injected inside the svg (typst). For KaTeX, there is nothing injected. Maybe I should update the documentation for this. |
|
Hey folks, great work! I'm quite interested in this PR. I'll gladly check out this branch and use it for now, but I was wondering if there are any plans from the maintainer's side to upstream this, and if yes what the blockers to this PR are (inb4 maintainer capacity). Cheers! |
|
This was intially just a PoC to fit my own needs for a blog, and I posted a PR here just in case anyone was interested. I'd gladly help you set up a clean PR to integrate it into zola! |
|
I can try to recreate the changes without merge conflict if that would help? Is someone actively working on getting this done? |
|
Merge conflicts were resolved |
|
Hey @Keats! Any thoughts on this math rendering PR? Would love to get your feedback to move this forward 🙂 |
|
Hey, just a random new user passing through. This PR looks great! For me getting the ability to render typst would be really helpful for some of my use cases. Mainly because typst has so much more functionality for rendering complex mathematics compared to regular katex: # Markdown
Here *is* some **example** markdown code:
$#diagram(
spacing: 8pt,
cell-size: (8mm, 10mm),
edge-stroke: 1pt,
edge-corner-radius: 5pt,
mark-scale: 70%,
blob((0,1), [Add & Norm], tint: yellow, shape: hexagon),
edge(),
blob((0,2), [Multi-Head\ Attention], tint: orange),
blob((0,4), [Input], shape: house.with(angle: 30deg),
width: auto, tint: red),
for x in (-.3, -.1, +.1, +.3) {
edge((0,2.8), (x,2.8), (x,2), "-|>")
},
edge((0,2.8), (0,4)),
edge((0,3), "l,uu,r", "--|>"),
edge((0,1), (0, 0.35), "r", (1,3), "r,u", "-|>"),
edge((1,2), "d,rr,uu,l", "--|>"),
blob((2,0), [Softmax], tint: green),
edge("<|-"),
blob((2,1), [Add & Norm], tint: yellow, shape: hexagon),
edge(),
blob((2,2), [Feed\ Forward], tint: blue),
)$Rendering these would require some modifications since you would have to include some packages from typst universe, but that is something I can totally volunteer to work on as soon as this gets merged. |
|
Hey, thank you so much for taking interest in this PR ! It is already possible to include packages by default via the [markdown.math]
engine = "typst"
svgo = "svgo.config.mjs"
css = "styles/typst-embed.css"
addon = "helpers.typ"This will include the What you're trying to achieve can also be done via a ```typ
= Hello, World!
This is a normal typst file
$ a + b = 42 $
``` |
|
I think that looks amazing. It's pretty big, I'll set up some time to properly review it |
cestef
force-pushed
the
feature/math-rendering
branch
from
601f1e7 to
fe7d683
Compare
|
This should be it, had to fight a bit with git rebase, sorry for the force-pushes |
| #[default] | ||
| None, | ||
| Typst, | ||
| Katex, |
There was a problem hiding this comment.
Do we need both typst and katex?
There was a problem hiding this comment.
The idea behind the generic MathCompiler trait was to add support for multiple backends, many people still use LaTeX in my opinion.
There was a problem hiding this comment.
But for the Zola crowd, do we need multiple renderer?
There was a problem hiding this comment.
Chiming in as a Zola user who's been using cestef's fork: I personally am using the Typst math rendering right now, but I had to manually port all the math on my blog from LaTeX to do so. I think a lot of people using this feature might be moving over from a previous math implementation, so they will likely use LaTeX, but my experience is that writing new posts with Typst is way easier. In the ideal world, it would be possible to specify math on a per-page or per-block basis, but that might be a little too complex.
There was a problem hiding this comment.
my experience is that writing new posts with Typst is way easier
That's what I've seen online as well. I don't use any math/LaTeX whatsoever myself so I don't have an opinion myself on that other than keeping things simple
There was a problem hiding this comment.
Imo, its about supporting legacy use cases, 95% of all math/physics papers are still written in latex, and its also the standard format for math in markdown: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/writing-mathematical-expressions. I think the main reason everyone on this thread is using typst is just the attraction to rust. But if no one on this thread actually wants support for Katex, it might be worth it to remove it for now and only add it if someone requests over typst.
There was a problem hiding this comment.
I think the main reason everyone on this thread is using typst is just the attraction to rust
This is not the case for everyone probably, example is me, I am rather a Typst user & enthusiast (rust as well but not as much experience using it) and actually would like to use it for a documentation website involving lots of mathematics code. Typst is gaining functionality and convenience very quickly and left LaTeX already behind in most aspects (my opinion).
There was a problem hiding this comment.
I personaly still use LaTeX because it is what I used untill now, and also for tikz figures and graphs (not with zola, just in my other documents). If Typst has the same math rendering quality as Katex and maintaining multiple engines is a pain, it would bother me to switch to Typst but I would do it anyways. I won't complain on this decision, I'm already grateful people made math rendering possible.
I can only see one downside of having only Typst: if people write things both in their blog and in research papers, they would need to write things in two languages each time. But I guess it's rare.
There was a problem hiding this comment.
I think the main reason everyone on this thread is using typst is just the attraction to rust
Every rust rewrite has a portion of supporters purely based on the fact that it is new, fancy, and written in rust. Existence of such supporters is a terrible indicator to judge a project's actual quality, soundness of architectural decisions and overall adoption trajectory.
When typst first got out, I was very critical because it was quite young and it had a few rough edges, but I quickly flipped because it made me way faster. I wrote my 140p thesis in typst and will never go back. After only one year of learning I was able to get my hands dirty with custom commands and document introspection in a way that after 7 years of latex I was never able to come close to.
Typst does numerous things a ton better in a way that latex will never be able to: To name one, TeX is a macro processing language, and this will never change. That makes it impossible to give good syntactical error messages or editor completion (heuristics are always possible, but as soon as you have custom commands you are out the window), and you will not be able to change that. To name another, the packaging ecosystem is easy to use and growing rapidly. And we don't have to talk about compilation speed (although to be fair the static web-rendering use case is different from getting a live preview while typing, where one sees the most benefit).
All this is to say two things.
First, I guarantee you there will be a growing portion of users who will want to use typst and feel very strongly about its adoption in up-and-coming projects such as zola.
Second, LaTeX is still a requirement for most people, and will continue to be. A portion of such people are likely to cast out zola if they want to typeset math but don't want the burden of learning a „fancy new typesetting language that nobody uses and will probably die in a few years, anyway“. Especially in academia things are slowly moving (rightfully so), and the change in adoption will probably come with a generational change; some communities will never reach it. Something similar is happening with automated theorem proving, and something similar happened in math with the introduction of category theoretical methods. We're talking on the scale of decades here.
Sure, much of this is anecdotal, and I might be wrong about my projections, but I like to think spending 6+ months doing nothing but writing typst and having had extensive LaTeX experience beforehand counts for something other than „irrelevant because opinion-based and not measurable“.
I am however not sure what you should do with this information, because I can't seriously tell someone to “just support both” without being the one who has to carry the maintenance burden.
| } | ||
|
|
||
| #[derive(Clone, Debug, PartialEq, Eq, Default, Hash)] | ||
| pub enum BoolWithPath { |
There was a problem hiding this comment.
What's that? Can you add a comment?
| #[derive(Clone, Debug, Serialize, Default)] | ||
| pub struct MathRenderer { | ||
| pub engine: MathRenderingEngine, | ||
| pub svgo: BoolWithPath, |
There was a problem hiding this comment.
is that a path to the binary?
| pub engine: MathRenderingEngine, | ||
| pub svgo: BoolWithPath, | ||
| pub css: Option<String>, | ||
| pub addon: Option<String>, |
There was a problem hiding this comment.
Only one addon supported? What are addons?
| enum MathRendererConfig { | ||
| // String case: math = "engine" | ||
| Engine(MathRenderingEngine), | ||
| // Table case with full configuration |
There was a problem hiding this comment.
let's just only have the full table so we don't have to support that kind of things
|
|
||
| // Generate cache key | ||
| let key = { | ||
| let mut hasher = XxHash64::with_seed(42); |
There was a problem hiding this comment.
Is the speed difference worth adding that crate? Image proc still uses the std DefaultHasher because in practice it's not really noticeable
| ImageFormat::Webp => { | ||
| // let _pixmap = typst_render::render(page, 300.0); | ||
| // TODO: svg2webp | ||
| unimplemented!("WebP is not supported yet") |
There was a problem hiding this comment.
Is it not supported in typst?
There was a problem hiding this comment.
typst supports png output, but not webp yet. https://typst.app/docs/reference/png/
There was a problem hiding this comment.
Would it be easier to go the PNG=>WebP route?
| {code} | ||
| "#, | ||
| ) | ||
| } |
There was a problem hiding this comment.
Do we want people to be able to use their own templates?
There was a problem hiding this comment.
I don't see much use, people can already configure templates further with the addon option, which includes additional code in the templates. inline and block templates only consist of base page settings and a math block. For further customization people can use the raw rendering via code blocks.
| let output_path = path.join(config.output_dir.clone()); | ||
|
|
||
| let caches = match config.markdown.cache { | ||
| config::BoolWithPath::True(ref maybe_path) => Some(Arc::new(match maybe_path { |
There was a problem hiding this comment.
When would the user want to specify their own cache directory?
There was a problem hiding this comment.
CI/CD pipelines or network shared cache, I can remove this if you feel it's overkill
There was a problem hiding this comment.
Yeah i think it's better to keep things simple
|
|
||
| ## SVG Optimization | ||
|
|
||
| Enable `svgo = true` to optimize the generated SVG files using [SVGO](https://svgo.dev). This can significantly reduce the file size of complex mathematical expressions. |
There was a problem hiding this comment.
so the reason we do that is because it's inlined as an image right? So there's no SVG to optimize in a build pipeline. I'm surprised there's not a library like svgo in Rust but I would leave that part out for now
| /// Whether to cache the rendered math | ||
| pub cache: BoolWithPath, | ||
| /// Whether to enable GitHub-style alerts | ||
| pub github_alerts: bool, |
There was a problem hiding this comment.
Github alerts seems to be duplicated in this enum, if you try to compile the program you get an error around this
| github_alerts: false, | ||
| math: MathRenderer::default(), | ||
| cache: BoolWithPath::True(None), | ||
| github_alerts: false, |
There was a problem hiding this comment.
Also errors out on this declaration as well
11 participants
IMPORTANT: Please do not create a Pull Request adding a new feature without discussing it first.
The place to discuss new features is the forum: https://zola.discourse.group/
If you want to add a new feature, please open a thread there first in the feature requests section.
Sanity check:
Code changes
(Delete or ignore this section for documentation changes)
nextbranch?If the change is a new feature or adding to/changing an existing one: