[Suggestion]: Remove free-form attached modifiers

[vhyrro]

This issue serves as a place for discussion with regards to free-form attached modifiers.

The existing syntax allows for extending the capabilities of regular attached modifiers by allowing whitespace and chars of the same type within. Examples include:

• $|a dollar sign ($) here|$
• $| space inbetween the modifier and the text |$

This syntax is problematic for various reasons.

1. The syntax is ambiguous with the ranged tag syntax:

   *| this is
   some ranged content
   |* <- this creates a new ranged tag
   

   Creating ranged tags with special character names like this is required for various parts of the attribute system to work.
2. The syntax creates dozens of new edge cases that need to be handled by the parser
3. Non-verbatim free-form and verbatim free-form behave the same, but verbatim free-form disallows the backslash.
   The reason is LaTeX: $| \LaTeX |$ would otherwise require two backslashes to type (to escape the escape sequence).
4. Free-form modifiers for non-verbatim attached modifiers

The following solution comes from a several hour chat with mrossinek, trying to find the most optimal set of tradeoffs:

Remove The Syntax

The best option is to entirely remove this syntax. This gives rise to a new attached modifier type: the super verbatim (name to be reconsidered :p)

The list looks as follows:

• Regular markup (bold, italic, etc.)
• Verbatim markup (`this`)
• Super verbatim markup (&this& and $this$)

Regular verbatim permits backslashes, thus providing all of the convenience features that come with being able to do that. Super verbatim markup treats backslashes verbatim, therefore being the purest form of verbatim content. Super verbatim exists in a standalone fashion (&this&, useful in certain contexts), as well as a convenience version for inline maths ($this$, functionally equivalent to &this&(math)).

This alleviates the issue of having to escape all LaTeX sequences. The super verbatim modifier also freely permits content that may require several backticks in its content.

Note: It is not possible to contain the opening/closing character of a super verbatim in its content.

Make Verbatim Free-Form-By-Default

The second part of this change is to make all verbatim modifiers (including the super verbatims) free-form by default.

This is possible when used in conjunction with a different proposal which disallows unclosed markup (#34). Thus, all attached modifiers have to be closed in some fashion. This makes a parser's job incredibly easy: parse an opening modifier, keep parsing till you find the closing modifier (potentially respecting escape sequences).

There is no incentive to allow regular attached modifiers like bold to be free-form, as the way they are rendered does not change when there is surrounding whitespace (one could really try to argue that this is the case for strikethrough, although we cannot find a valid use case in any scenario).

[benlubas]

Super verbatim markup disallows backslashes within its content

\ is extremely important for math represented as latex. How am I to represent a math expression with latex if I cannot put a \ inside of a math block?

[boltlessengineer]

@benlubas I think he mean super verbatim disallows backslash as an escape modifier. So $\$$ is valid syntax while `\`` is not. So `\`` is valid syntax while $\$$ is not.

Edit: correct

[vhyrro]

Super verbatim markup disallows backslashes within its content

\ is extremely important for math represented as latex. How am I to represent a math expression with latex if I cannot put a \ inside of a math block?

By "disallow" I meant "on Norg's side", aka it'd be treated as a verbatim backslash. Let me amend that now.

Edit: done.

[mrossinek]

4. Free-form modifiers for non-verbatim attached modifiers

Looks like you forgot to finish this point?

---

Besides the above, I agree with everything 👍

---

Replying to boltless:

$\$$

This is in fact NOT valid!
See the original description:

Note: It is not possible to contain the opening/closing character of a super verbatim in its content.

While this is not ideal, we have discussed the benefits and drawbacks of this at quite some length offline and we feel like that this is a compromise that can be worked around.

To summarize:

• verbatim markup (the backtick) has lower precedence than the escape character (the backslash). Thus, a single backtick can indeed be placed inside of verbatim markup:

  Here is `some verbatim text \` with an escaped backtick` inside of it.
  

• super verbatim markup (& and $) both have higher precedence than the backtick. Theefore, the parser will open on the first & or $ encountered and then greedily consumes everything within the paragraph until the next such character is obtained (if the paragraph ends, unclosed markup results in an error, see [Suggestion]: Disallow unclosed markup #34).
• this also implies that one can easily write LaTeX inside $ because the backslash has no special meaning to Norg and can be treated purely by LaTeX.
• Finally, if one really needs a $ sign inside LaTeX, one can write $10\textdollar$ or use &10$&(math)
• Placing a & inside of &-superverbatim becomes virtually impossible. One could use $whatever \& something else$ and live with the fact that this is "technically" inline math or we find a way to have an attached modifier attribute which disables the math-mode: $whatever & something else$(none) (for example).

[Yehoslav]

There's one more use for regular free form modifiers, it's better user experience. I believe it's just an advantage to be able to visually parse when a modifier starts anh when it ends, especially when there are nested modifies. Especially considering that still a limited number of editors support the sintax.

I know that ease of parsing is a great goal, but stould it come before user experience? At the end of the day once the parser is implemented this feature will not bother the development anymore, but at the same time the sintax feature influences the daily usage.

If it creates ambiguity, wouldn't it be better to replace the | with a different character that has no special mening at the beginning of the line?

[Yehoslav]

furthermore, izhaving explicit stacsrt and ending for modifiers makes them easier to work with external tools, like grep, sed. And even in vim replacing one modifier for another becomes more trivial, maybe even configuring nvim-surround to easier switch between them 😅

Sadly I couldn't yet integrate efficiently neorg into my workflow, but these are the pain points I have with other markup languages.