Unexpected switch to drawer content

[kristijanhusak]

Hey,

I've got an issue where user reported freezing. It is caused by my manual markup highlighter, but the problem is also a big file.

I tried to narrow down what's the issue, and I noticed one thing.

Having this file:

* Headline 1
  Text 1
:LOGBOOK:
:LAST_REPEAT: [2023-10-11 Wed 13:46]
:END:
* Headline 2
  Text 2
:LOGBOOK:
:LAST_REPEAT: [2023-10-09 Mon 16:36]
:END:

Parses it correctly:

(document [0, 0] - [10, 0]
  subsection: (section [0, 0] - [5, 0]
    headline: (headline [0, 0] - [1, 0]
      stars: (stars [0, 0] - [0, 1])
      item: (item [0, 2] - [0, 12]
        (expr [0, 2] - [0, 10])
        (expr [0, 11] - [0, 12])))
    body: (body [1, 2] - [5, 0]
      (paragraph [1, 2] - [2, 0]
        (expr [1, 2] - [1, 6])
        (expr [1, 7] - [1, 8]))
      (drawer [2, 0] - [5, 0]
        name: (expr [2, 1] - [2, 8])
        contents: (contents [3, 0] - [4, 0]
          (expr [3, 0] - [3, 13])
          (expr [3, 14] - [3, 25])
          (expr [3, 26] - [3, 29])
          (expr [3, 30] - [3, 36])))))
  subsection: (section [5, 0] - [10, 0]
    headline: (headline [5, 0] - [6, 0]
      stars: (stars [5, 0] - [5, 1])
      item: (item [5, 2] - [5, 12]
        (expr [5, 2] - [5, 10])
        (expr [5, 11] - [5, 12])))
    body: (body [6, 2] - [10, 0]
      (paragraph [6, 2] - [7, 0]
        (expr [6, 2] - [6, 6])
        (expr [6, 7] - [6, 8]))
      (drawer [7, 0] - [10, 0]
        name: (expr [7, 1] - [7, 8])
        contents: (contents [8, 0] - [9, 0]
          (expr [8, 0] - [8, 13])
          (expr [8, 14] - [8, 25])
          (expr [8, 26] - [8, 29])
          (expr [8, 30] - [8, 36]))))))

But when I start adding a headline in between, it converts first :END: and last :END into a drawer, and treats everything in between as a drawer content:

Updated file:

* Headline 1
  Text 1
:LOGBOOK:
:LAST_REPEAT: [2023-10-11 Wed 13:46]
:END:
*
* Headline 2
  Text 2
:LOGBOOK:
:LAST_REPEAT: [2023-10-09 Mon 16:36]
:END:

Parsed as:

(document [0, 0] - [11, 0]
  subsection: (section [0, 0] - [11, 0]
    headline: (headline [0, 0] - [1, 0]
      stars: (stars [0, 0] - [0, 1])
      item: (item [0, 2] - [0, 12]
        (expr [0, 2] - [0, 10])
        (expr [0, 11] - [0, 12])))
    body: (body [1, 2] - [11, 0]
      (paragraph [1, 2] - [4, 0]
        (expr [1, 2] - [1, 6])
        (expr [1, 7] - [1, 8])
        (expr [2, 0] - [2, 9])
        (expr [3, 0] - [3, 13])
        (expr [3, 14] - [3, 25])
        (expr [3, 26] - [3, 29])
        (expr [3, 30] - [3, 36]))
      (drawer [4, 0] - [11, 0]
        name: (expr [4, 1] - [4, 4])
        contents: (contents [5, 0] - [10, 0]
          (expr [5, 0] - [5, 1])
          (expr [6, 0] - [6, 1])
          (expr [6, 2] - [6, 10])
          (expr [6, 11] - [6, 12])
          (expr [7, 2] - [7, 6])
          (expr [7, 7] - [7, 8])
          (expr [8, 0] - [8, 9])
          (expr [9, 0] - [9, 13])
          (expr [9, 14] - [9, 25])
          (expr [9, 26] - [9, 29])
          (expr [9, 30] - [9, 36]))))))

Once I add the space after the * it parses it correctly.
This is more-less expected, but for that 1 change a lot of nodes are updated. In the reported issue this happens between two headlines where it ends up putting ~500 lines into the drawer content for a split second and freezes the editor.

I can think of a few solutions here, but I was able to test only one myself:

1. Never allow :END: to be a start of a drawer
2. Consider asterisk(s) at the start of a line a valid node (expr) even if it does not have a space after it

For the 2. point, what I mean is this:

Content:

* TODO Test
  Content
*
* TODO Test
  Content

Parsed:

(document [0, 0] - [5, 0]
  (ERROR [0, 0] - [2, 1]
    subsection: (section [0, 0] - [2, 0]
      headline: (headline [0, 0] - [1, 0]
        stars: (stars [0, 0] - [0, 1])
        item: (item [0, 2] - [0, 11]
          (expr [0, 2] - [0, 6])
          (expr [0, 7] - [0, 11])))
      body: (body [1, 2] - [2, 0]
        (paragraph [1, 2] - [2, 0]
          (expr [1, 2] - [1, 9]))))
    (stars [2, 0] - [2, 1]))
  body: (body [2, 1] - [3, 0])
  subsection: (section [3, 0] - [5, 0]
    headline: (headline [3, 0] - [4, 0]
      stars: (stars [3, 0] - [3, 1])
      item: (item [3, 2] - [3, 11]
        (expr [3, 2] - [3, 6])
        (expr [3, 7] - [3, 11])))
    body: (body [4, 2] - [5, 0]
      (paragraph [4, 2] - [5, 0]
        (expr [4, 2] - [4, 9])))))

When something is added to that line that's not a space:

* TODO Test
  Content
*t
* TODO Test
  Content

(document [0, 0] - [5, 0]
  subsection: (section [0, 0] - [3, 0]
    headline: (headline [0, 0] - [1, 0]
      stars: (stars [0, 0] - [0, 1])
      item: (item [0, 2] - [0, 11]
        (expr [0, 2] - [0, 6])
        (expr [0, 7] - [0, 11])))
    body: (body [1, 2] - [3, 0]
      (paragraph [1, 2] - [3, 0]
        (expr [1, 2] - [1, 9])
        (expr [2, 0] - [2, 2]))))
  subsection: (section [3, 0] - [5, 0]
    headline: (headline [3, 0] - [4, 0]
      stars: (stars [3, 0] - [3, 1])
      item: (item [3, 2] - [3, 11]
        (expr [3, 2] - [3, 6])
        (expr [3, 7] - [3, 11])))
    body: (body [4, 2] - [5, 0]
      (paragraph [4, 2] - [5, 0]
        (expr [4, 2] - [4, 9])))))

This change in the parser does the trick for me:

diff --git a/src/scanner.c b/src/scanner.c
index f305612..37be99e 100644
--- a/src/scanner.c
+++ b/src/scanner.c
@@ -267,6 +267,10 @@ bool scan(Scanner *scanner, TSLexer *lexer, const bool *valid_symbols) {
             skip(lexer);
         }
 
+        if (lexer->lookahead == '\n') {
+          return false;
+        }
+
         if (valid_symbols[SECTIONEND] && iswspace(lexer->lookahead) &&
             stars > 0 && stars <= VEC_BACK(scanner->section_stack)) {
             VEC_POP(scanner->section_stack);

I think the 2nd option is simpler and probably more correct. Unless there is a space after *, there's no need to treat it as headline.

Let me know what you think.