Ignore fenced code blocks

[dharmab]

Thanks for this useful tool! Here's a first pass at ignoring fenced code blocks as defined in the GFM spec. Fixes #4

I'd like to contribute tests for this, but the build_toc() function is currently somewhat difficult to test because it can only take input from files. I propose a change to the api to provide a build_toc_from_string() function which takes the markdown content as a string. Alternatively, I can add a bunch of test case markdown files to the repo. Let me know what approach you prefer.

For now, a brief manual test:

[dharmab@n7 md-toc]$ cat test.md
# Foo

[](TOC)

```python
# Bar
#
## Baz
```

````
# Bar
#
## Baz
`````

~~~
# Bar
#
## Baz
~~~
[dharmab@n7 md-toc]$ md_toc github test.md 
- [Foo](#foo)

When I ran make pep on master it changed a bunch of files, so I decided not to run make pep to keep the diff small. It would be helpful to include a requirements.txt file with pinned versions of tools like yapf and flake8. I know that flake8 changes its behavior quite a bit with each release.

[dharmab]

More testing:

[dharmab@n7 md-toc]$ cat test.md 
# Foo1

[](TOC)

```python
# Bar
#
## Baz
```

# Foo2

````
# Bar
#
## Baz
`````

# Foo3

~~~
# Bar
#
## Baz
~~~

# Foo4
[dharmab@n7 md-toc]$ md_toc github test.md 
- [Foo1](#foo1)
- [Foo2](#foo2)
- [Foo3](#foo3)
- [Foo4](#foo4)

[frnmst]

Thanks for this useful tool! Here's a first pass at ignoring fenced code blocks as defined in the GFM spec. Fixes #4

Hello,

thanks!
Could you rebase the commit to the newly created frnmst:ignore-code-fences branch? I can't accept it in the master branch because of the PyPI and AUR packages. Then, once the dev branch is ready, we'll merge ignore-code-fences in dev.

At the moment I am working on the TOC list indentation problem because it was something I left behind for a long time. There is also the fact that GitLab changed markdown parser in the past year:

• md-toc/md_toc/api.py

  Line 224 in a56ec2d

  # FIXME: the following works only for some cases.

• md-toc/md_toc/api.py

  Line 277 in 1232ba1

  def compute_toc_line_indentation_spaces(header_type_curr=1,

• GitLab migrated to commonmarker #5

I'd like to contribute tests for this, but the build_toc() function is currently somewhat difficult to test because it can only take input from files. I propose a change to the api to provide a build_toc_from_string() function which takes the markdown content as a string. Alternatively, I can add a bunch of test case markdown files to the repo. Let me know what approach you prefer.

Yes, tests are mandatory for md_toc. The API changed (and it is still changing) in the dev branch. See:

• https://github.com/frnmst/md-toc/compare/dev#diff-3f21202575ecc1082240d6bc0de75a77R103
• https://github.com/frnmst/md-toc/compare/dev#diff-3f21202575ecc1082240d6bc0de75a77R73
• etc...

Maybe it's possible to move the code fence detection part of your code to a new function, something like is_code_fence(parser, line, is_in_code_fence), so you can run tests easily. The only thing build_toc_line would do is to keep track of the state, just like it does for the headers. What do you think?

For now, a brief manual test:

[dharmab@n7 md-toc]$ cat test.md
# Foo

[](TOC)

```python
# Bar
#
## Baz

# Bar
#
## Baz

# Bar
#
## Baz

[dharmab@n7 md-toc]$ md_toc github test.md

• Foo

When I ran make pep on master it changed a bunch of files, so I decided not to run make pep to keep the diff small. It would be helpful to include a requirements.txt file with pinned versions of tools like yapf and flake8. I know that flake8 changes its behavior quite a bit with each release.

I didn't know about this but I'll certainly look into it.
Thanks

[dharmab]

👍 Splitting the code fence detection to two functions makes sense- one to detect the opening, and another to detect the closing. There is still state tracking that remains untested, but that's an easy way to both increase the test coverage and reduce the complexity of the main parser.

[frnmst]

Great, thanks!

[frnmst]

See https://github.com/frnmst/md-toc/projects/1

[frnmst]

Merged in the dev branch.