Handling Line Breaks in Equations for HTML to Word Conversion

[vishalmudgalpw]

I am working on converting HTML content that includes mathematical equations(MathML/ Latex) into Word documents using Pandoc. The equations in the HTML are rendered using MathJax, and we need to ensure that line breaks within these equations are preserved in the resulting Word document. Specifically, when the equations are broken into multiple lines in the HTML, these breaks should be reflected in the Word output.

suppose our sample.html containing math equation which have linebreak:

we have used below source to render math equations in html:
source: 'https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js';

sample.txt

Converted into word using command:
pandoc --standalone --lua-filter='${pandocFilterFilePath}' --reference-doc='${styleReferenceFilePath}' --wrap=preserve '${path}' -o '${uniqueFilePath}';

reference file not important for particular case.

word output:
e9r8gkj1z5ivkxtfuyn76za5b.docx

We need a solution that ensures line breaks within equations in the HTML are correctly rendered as line breaks in the Word document.

Any guidance or examples from the community would be greatly appreciated. Thank you!

[jgm]

How are you representing the line breaks in your HTML? Please just give a small sample, not a huge file with tons of JavaScript.

[vishalmudgalpw]

Sure, I've created a sample HTML file(tried to make it short) to illustrate the issue.

Sample HTML file:
sample.txt

Output DOCX file:
j3t1klfn633skexzbrhhyftuw.docx

Explanation:
Content:

`<p><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>FE</mi><mtext>cell&#160;</mtext><mo>&#176;</mo></msubsup><mspace linebreak="newline"></mspace><msubsup><mi mathvariant="normal">E</mi><mtext>cell&#160;</mtext><mo>&#176;</mo></msubsup><mo>=</mo><mo>-</mo><mi>ve</mi></math></p>
<p>&nbsp;</p>`

Steps:-
1: Suppose we have to render this content in our html.

The source to render the MathML equations using MathJax:
source: 'https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js';

2: MathJax Configuration:

window.MathJax = { loader: { load: [ 'input/tex-base', 'output/svg', 'ui/menu', '[tex]/require', '[tex]/ams', ], }, tex: { packages: { packages: { '[+]': ['ams'] } }, }, };

3: Global Style for Line Breaks:

Since MathJax v3 doesn't support automatic line breaks (reference: MathJax Documentation), we added the following style in index.html:

<style type="text/css">
  mjx-mspace {
    display: block !important;
  }
</style>

4: By all above steps we rendered HTML with Line Breaks:

5: Convert HTML to Word using Pandoc:

We send the HTML body to an API for conversion to Word using Pandoc.

6: Real-Time Logging:

Integrated Pandoc with real-time logging to monitor the process:

process.stdout.on('data', (data) => {
  this.logger.log(`THIS IS STDOUT stdout: ${data}`);
});
process.stderr.on('data', (data) => {
  this.logger.error(`stderr: ${data}`);
});

7: Pandoc Filter:

Used a print statement in the Pandoc filter to log the process. Here are the logs showing how Pandoc interpreted the HTML:

function Span(elem)
  print("----------------------------------span element: running", elem)
end

LOGS:

[Nest] 46933 - 06/05/2024, 12:35:29 PM LOG [FileConversionService] THIS IS STDOUT stdout: ----------------------------------span element: running Span ("",[],[]) [SoftBreak,Math InlineMath "{FE}_{\\text{cell~}}^{{^\\circ}} E_{\\text{cell~}}^{{^\\circ}} = - {ve}",SoftBreak]

8: The Pandoc command generates the resulting DOCX file.

I hope this explanation clarifies the issue we're facing. Any guidance or examples would be greatly appreciated. Thank you! 🙏

[jgm]

Since MathJax v3 doesn't support automatic line breaks (reference: MathJax Documentation), we added the following style in index.html:

Pandoc won't pay any attention at all to styles. It just handles structure. If the line break isn't explicitly represented in the MathML, then no dice.

[jgm]

Removing all the cruft, your MathML is just this:

                     <math xmlns="http://www.w3.org/1998/Math/MathML">
                        <msubsup>
                           <mi>FE</mi>
                           <mtext>cell&nbsp;</mtext>
                           <mo>°</mo>
                        </msubsup>
                        <mspace linebreak="newline"></mspace>
                        <msubsup>
                           <mi mathvariant="normal">E</mi>
                           <mtext>cell&nbsp;</mtext>
                           <mo>°</mo>
                        </msubsup>
                        <mo>=</mo>
                        <mo>-</mo>
                        <mi>ve</mi>
                     </math>

pandoc -f html -t native converts this to

[ Plain
    [ Math
        InlineMath
        "{FE}_{\\text{cell~}}^{{^\\circ}} E_{\\text{cell~}}^{{^\\circ}} = - {ve}"
    ]
]

You are correct, there is nothing corresponding to the MathML element <mspace linebreak="newline"></mspace>.

Essentially this is jgm/texmath#91
Also relevant: jgm/texmath#99

[jgm]

A workaround could be to use an mtable element with two rows.
Pandoc will convert this correctly.

[vishalmudgalpw]

Okay, if we need to adapt a workaround solution here:

For MathML: Convert MathML data into mtable format if there is a line break.
For LaTeX: Convert Latex data into an appropriate environment, such as align or multiline, if there is a line break.

I have some questions:

Q1)Are there any specific ways to convert large volumes of MathML/LaTeX equations with line breaks to DOCX which Pandoc could interpret and show the desired result?

Q2)Are there any Pandoc-specific configurations or Lua filters that can help better handle inline and display math with line breaks during conversion?

I think we can't use Lua filters here because, as seen in our logs, there isn't a way to detect line breaks as they don't appear in the logs. So, how can we identify when a line break occurs and convert it to the required format. Am I correct in this understanding?

Q3) Do we need to modify our HTML to the required format before sending it for conversion to adapt this solution?

Thanks 🙏

[jgm]

Q1: No, nothing I can think of.
Q2: Lua filters won't help with MathML, because pandoc converts this to TeX and stores it that way, so by the time a filter sees it the linebreak is gone. It could help with TeX input. (But \\ does not cause a line break in TeX except in certain special environments...)
Q3: Yes probably so

[vishalmudgalpw]

Since, we could potentially adapt a solution where we modify our HTML content before parsing it for conversion via Pandoc.

I have some findings and a blocker to discuss on this approach.

To handle equations with line breaks, I preprocess my HTML to transform <math> elements containing <mspace linebreak="newline"/> into an <mtable>, which effectively introduces the desired line breaks in the DOCX output.

Here is the preprocessing function I use:

function transformMathML(htmlContent) {
  const cheerioInstance = cheerio.load(htmlContent, {
    xmlMode: true,
    decodeEntities: false,
  });

  cheerioInstance("math").each((index, element) => {
    const mathElement = cheerioInstance(element);
    const mathHtml = mathElement.html().replace(/<mspace\s+linebreak="newline"\s*\/>/g, "SPLIT_HERE");
    const parts = mathHtml.split("SPLIT_HERE");

    if (parts.length > 1) {
      const tableRows = parts.map(part => `<mtr><mtd>${part.trim()}</mtd></mtr>`).join("");
      const mtable = `<mtable>${tableRows}</mtable>`;
      mathElement.html(mtable);

      if (mathElement.parents("mjx-assistive-mml").length) {
        mathElement.parents("mjx-container").find("mjx-math").remove();
      }
    }
  });

  return cheerioInstance.html();
}

This transformation works well for introducing breaks; however, I face an issue with the alignment of text within these tables in the resultant DOCX file. By default, the text within the mtable cells is centered. I can manually set the alignment to left via Word's interface, but I need this setting to be automatic for all such cases.

Word output file sample:
aoheuxfjbx5dfui9lxwly0m8a.docx

Word interface screenshot where we are manually setting alignment:

Question:
Is there a way to set a default left alignment for mtable content in the DOCX output using Pandoc?
Can I define a style in the custom style reference DOCX file or use a Lua filter that specifies left alignment for these mtable contents?

I am looking for a solution to ensure that all mtable cell contents are left-aligned by default when converted to DOCX.

Thanks 🙏

[jgm]

Have you tried adding the attribute columnalign="left" on your mtable?

[vishalmudgalpw]

I've successfully used the columnalign="left" attribute in an mtable to align the contents to the left, which works as expected.

Also looking for a solution which could align mtable with regular text flow on top instead of center.

Word interface screenshot where we are manually setting alignment:

What attribute we could use for this alignment? or Any other suggestion?

I've attempted to use align="top" and rowalign="top" within the mtable tag, but these have not had any impact on the vertical positioning in the DOCX output.

mtable element's attributes as defined in the MathML specification:

mtable = element mtable {
  mtable.attributes, TableRowExpression*
}
mtable.attributes = {
  CommonAtt, CommonPresAtt,
  attribute align {xsd:string {
    pattern = '\s*(top|bottom|center|baseline|axis)(\s+-?[0-9]+)?\s*'
  }}?,
  attribute rowalign {list {verticalalign+} }?,
  attribute columnalign {list {columnalignstyle+} }?,
  attribute groupalign {group-alignment-list-list}?,
  attribute alignmentscope {list {("true" | "false") +}}?,
  attribute columnwidth {list {("auto" | length | "fit") +}}?,
  attribute width {"auto" | length}?,
  attribute rowspacing {list {(length) +}}?,
  attribute columnspacing {list {(length) +}}?,
  attribute rowlines {list {linestyle +}}?,
  attribute columnlines {list {linestyle +}}?,
  attribute frame {linestyle}?,
  attribute framespacing {list {length, length}}?,
  attribute equalrows {"true" | "false"}?,
  attribute equalcolumns {"true" | "false"}?,
  attribute displaystyle {"true" | "false"}?,
  attribute side {"left" | "right" | "leftoverlap" | "rightoverlap"}?,
  attribute minlabelspacing {length}?
}

verticalalign values:
  "top" | "bottom" | "center" | "baseline" | "axis"

Thanks 🙏

[vishalmudgalpw]

Hi everyone,

Just checking in on my previous post regarding matrix alignment of mtable elements to the top in DOCX output.
Any insights or suggestions on how to achieve this?

Thanks!