Allow showing only a range of lines for code blocks

[llucax]

Description

I'm using mkdocstrings and Sybil to extract code examples in the documentation and test by running pylint on them via pytest.

This forces me to sometimes add too much boilerplate to examples, which is distracting from the main idea I want to show in an example, but it is also really important that examples stay up to date and (mostly) working.

An alternative would be to move the examples to separate files and use the snippet extension line selection feature, but I want to keep the examples in the docstrings, so they are also accessible in code editors, as python include all the docs in the code itself.

Related links

• Highlight specific lines feature
• Markdown Snippet line selection
• Original issue reported to mkdocs-material

Extra:

• mkdocstrings
• sybil
• pylint
• pytest
• Code to put it all together

Benefits

Being able to write examples that are verifiable using linting tools to ensure they stay up to date, but still only being able to show the relevant parts that make sense for the documentation, avoiding distracting, boilerplate code.

Solution Idea

I would like code blocks to accept selecting which lines to show, similar to what the snippet line selection or the highlight specific lines feature does. For example:

Example:
    ``` py show_lines="4:"
    # Bolierplate setup code
    # More boiler plate setup code
    # And some more
    this is the_juicy_part
    # more useful code
    ```

I'm suggesting the snippet extension syntax (with :) instead of the highlight specific lines feature syntax because it is more natural for specifying open-ended ranges.

Optional: There could be some button to expand the example and show also the boilerplate code, so users could copy&paste the whole example and run it if they want.

[facelessuser]

The requested feature is quite niche, though I see why you would find it useful for your purposes, and I can see a select amount of people using it as well. One of the reasons I added custom fences to SuperFences was to allow people to create specialized behavior for their code blocks without me having to implement and maintain every niche case that was desired.

What I can provide is some guidance as to how you can accomplish what you want with custom fences. You can simply override the code blocks you desire adding in the options to specify the lines to show. Lastly, you'd preprocess the code, stripping out any lines you did not wish to show before passing them along to the highlighter.

You may have to wait until I find the time over the weekend to code up an example for you, but you can start reading about custom fences here: https://facelessuser.github.io/pymdown-extensions/extensions/superfences/#custom-fences.

[facelessuser]

Here's an example. We override all languages with * and define our custom formatter and validator. In our custom validator we validate our new option and then feed the great of the inputs through the default code validator. Then in our formatter, we check for our new option, modify the source as needed, and send it through the default highlight logic. That's it. It should work on any language then.

"""Example custom fence."""
import markdown
from pymdownx.superfences import highlight_validator


def remove_validator(language, inputs, options, attrs, md):
    """Highlight validator."""

    # Add validation for special new options
    for k, v in inputs.items():
        if k == 'show_lines':
            try:
                options[k] = set(int(i) for i in v.split(','))
            except Exception:
                # Validation failed
                return False

            # Remove handled option from inputs
            del inputs[k]
            break

    # Run default validator
    res = highlight_validator(language, inputs, options, attrs, md)
    return res


def remove_formatter(src="", language="", class_name=None, options=None, md="", **kwargs):
    """Formatter wrapper."""

    # Alter source here
    if 'show_lines' in options:
        show = options['show_lines']
        lines = []
        for e, line in enumerate(src.split('\n'), 1):
            if e in show:
                lines.append(line)
        src = '\n'.join(lines)

    # Run through default highlighter
    return md.preprocessors['fenced_code_block'].highlight(
        src=src,
        class_name=class_name,
        language=language,
        md=md,
        options=options,
        **kwargs
    )


MD = """
``` py show_lines="1,2,3,8"
import something
import something_else

# Some comments
# that we
# want to
# avoid
something.go()
```
"""

html = markdown.markdown(
    MD,
    extensions=['pymdownx.superfences'],
    extension_configs={
        "pymdownx.superfences": {
            "custom_fences": [
                {
                    'name': '*',
                    'class': 'highlight',
                    'validator': remove_validator,
                    'format': remove_formatter
                }
            ]
        }
    }
)

print(html)

[llucax]

Hi @facelessuser, thanks for the explanation and example. I really appreciate it. I will give it a try as soon as possible.

I have one more question though, since I want to use this in mkdocs-material, I'm not exactly sure how this is plugged into that. I'm also not sure if this is a question I should ask here or in mkdocs-material.

Thanks again!

[facelessuser]

I can show you how I run them:

I currently have one in my tools folder: https://github.com/facelessuser/pymdown-extensions/blob/main/tools/pymdownx_md_render.py. I place an __init__.py in tools to treat it as a local module.

I specify it in mkdocs.yml like so: https://github.com/facelessuser/pymdown-extensions/blob/main/mkdocs.yml#L119. For yours, I would obviously use the name of * and class of highlight.

Lastly, to find local modules I have to run mkdocs with the -m option: python3 -m mkdocs serve: https://github.com/facelessuser/pymdown-extensions/blob/main/.github/workflows/deploy.yml#L36.

I don't use Material's docker image, so if you are using that, you might have to ask on mkdocs-material if it is not obvious how to run mkdocs with -m.

The alternative is that you make a pip installable package with your custom code and then you don't have to load local modules, you would just reference it as a normal installed module.

Also, the example I provided is just a quick and dirty example. You will have to tweak the code to have it work exactly how you want, but it is a good starting point.

[facelessuser]

This is by design. In general, if unexpected Markdown syntax is used, Markdown does not fail, it just doesn't parse the content that it doesn't understand, or some other step of the parser will attempt to parse it when another step fails.

If you want your custom fences to throw an exception that causes SuperFences to fail, check out the docs which explains what to do: http://localhost:8000/pymdown-extensions/extensions/superfences/#exception-handling.

[llucax]

I see, thanks, but still I don't think I want to raise the exception, I want to print a warning and let the original handler handle the block, ignoring my customization.

So, right now if parsing show_lines fails, and I return False in the validation, I get this:

Which doesn't make a lot of sense, I guess nobody wants to have that in their documentation and never be notified about it.

What I think I want to do is to try to parse my show_lines and if it fails, show a warning (a type of warning that will be converted to an error if --strict is used in mkdocs), and ignore the broken option and still highlight_validator(language, inputs, options, attrs, md) normally.

Does this make sense? I will try to find out how to emit a warning mkdocs understand, but if you know how to do it, it would be awesome if you can share it.

PS: I'll probably release this as a small python package. I plan to reference this issue as a source of inspiration, as the base of the code is derived from your example. I plan to release it with MIT license. Would that be OK? Should I put you in the copyright?

[facelessuser]

Then capture the exception and print a warning.

import logging
log = logging.getLogger('mkdocs')
log.warning('whatever')

Remember, none of these extensions care about MkDocs. MkDocs uses them, but they don't know the user is MkDocs, so there is no specific MkDocs code in any of these extensions.

If you want to raise an exception, raise an exception. If you want to silently let the parser move on, let it move on. If you want a warning, then print a warning. The choice is yours.

[llucax]

OK, thanks. For my use case it is really important that it plays well with mkdocs, so I decided for an ugly hack I'm not proud of, but it is practical and seems to work (inspect the stack to see if we are running inside mkdocs to use it as logger name or not).

[facelessuser]

That's fair

[llucax]

Just in case anybody else bumps into this, I create a package to implement it:

• Docs: https://frequenz-floss.github.io/pymdownx-superfence-filter-lines-python/
• Pypi: https://pypi.org/project/pymdownx-superfence-filter-lines/

[geoffreyweal]

Hey @llucax and @facelessuser . Thanks for this post. I just wanted to mention that this is very useful to me as I like to explain python scripts line section by section to show how to run a python script with my custom program (for an example, see https://organisms.readthedocs.io/en/latest/Using_Run.html which was made with sphinx). I am currently using this package from @llucax which is working as expected. This is amazing thanks heaps!

I was wondering if the installation requirement for pymdownx-superfence-filter-lines has to be for python >= 3.11? I am currently using Python 3.9 because one of my programs uses another programs that has to be run in either python 3.7 or 3.9?

Thanks again for this! pymdownx-superfence-filter-lines is amazing!

[facelessuser]

I have nothing to do with pymdownx-superfence-filter-lines and pymdown-extensions does not have a lower limit of 3.11.

[llucax]

@geoffreyweal I don't think it would be difficult to make pymdownx-superfence-filter-lines work with Python 3.9 or even 3.7, I just used that version because is what I use and was too lazy to test it with older Python version. Please create an issue in https://frequenz-floss.github.io/pymdownx-superfence-filter-lines-python/ about it, so we don't spam this discussion (or better send a PR! :D).

And happy to hear it is useful for other people too!

[geoffreyweal]

Hey @llucax Awesome thanks for that. I will do that now. Thanks!