Console output folding?

[jeffwright13]

Hello,

I am doing some investigation into the possibility of implementing a 'folding' capability for Pytest's console output, and wanted to ask here about the feasibility of this idea. My main motivator is the fact that, in the face of multiple test failures, and with the default behavior in place, the output can span many pages, requiring the tester to scroll back and hunt for the first failed test in the run. It is easy to get lost, and hard to pinpoint precisely where things started to go wrong.

So I thought it would be helpful to have a capability that implements some sort of console 'folding', such that all the detailed Pytest console content is still available, just hidden by default - yet still able to be expanded in some way with operator interaction. [I don't know yet if "operator interaction" would be done with keyboard, mouse, or what.] I do know that Pytest already allows changing the level of verbosity, traceback info, etc. I am also aware of the pytest-html plugin, which is pretty much exactly what I am looking for, just in-console.

Would appreciate some comments/advice...is this even do-able? I can't be the first person to dream of such a thing!

Thanks, Jeff

[nicoddemus]

Hi @jeffwright13,

That seems really useful, however it would require pytest's output to become interactive, which would be a separate application I believe. This certainly seems possible to implement as a plugin, perhaps using Rich? Anyway that is a neat idea.

[jeffwright13]

@nicoddemus wrote:

it would require pytest's output to become interactive

Could you elaborate on that bit more? The way I was visualizing it, the "pytest_fold" code would consume the normal output from Pytest (as it is on its way to stderr/stdout); "mark" any lines that should be folded; and then send the marked text to the console for rendering in a way that allows the user to fold or unfold the marked lInes. So in my mind, there would be no coupling between Pytest's output stream and the folding code. Of course, there would be a necessary coupling between the output from "pytest_fold" and the console, and maybe that is where Rich would come into play?

Just thinking out loud here...I freely admit the mental picture I have constructed of how Pytest works and how this might be implemented are quite fuzzy...so feel free to correct me where I am wrong. :-)

[nicoddemus]

Sorry, I just meant that the output code would need to be changed to handle interaction in some way (clicking on the marked lines would expand them).

Regardless definitely seems like this can be experimented in a plugin (for a plugin which completely changes pytest's output, there's pytest-sugar which can be used as a starting point).

[The-Compiler]

I don't think rich can handle interactive UIs. However, rich's developer is currently working on textual which builds on Rich and does allow to do so. Right now, the closest thing to some framework like that is probably Urwid.

But that'd be a quite big step, and not sure if I'd use it personally, rather than using some kind of IDE integration (VS Code in my case). Ideally we'd indeed be able to do this as part of normal "linear" terminal output as you suggest, but that's not possible with any terminal emulator I'm aware of, unfortunately - see my comment below for some thoughts.

[nicoddemus]

rich's developer is currently working on textual

Sorry, that's what I had in mind hehehe.

[flub]

can't prompt-toolkit handle stuff like this as well?

[The-Compiler]

Taking this here from above since it's a different approach at tackling the problem:

"mark" any lines that should be folded; and then send the marked text to the console for rendering in a way that allows the user to fold or unfold the marked lInes.

This would be amazing! Unfortunately, to my knowledge, there's no terminal which supports such a thing. That's why @nicoddemus said above that it essentially requires a pytest wrapper or plugin which moves away from the current console UI, instead providing an interactive, TUI-like interface.

Perhaps @kovidgoyal with kitty would be interested in such a thing, given that it already has a lot of nice, advanced features (such as graphics rendering) other terminals typically don't? 😉

For some prior art, both Travis CI and GitHub Actions have such a feature, and there's a pytest-travis-fold plugin using it (see abusalimov/pytest-travis-fold#9 for GHA, as an aside).

[jeffwright13]

Good discussion everyone,m thanks for all the input so far.

@flub Since I am not an expert on the inner workings of the Pytest codebase (yet...), would you be able to point me to something specific? Like, when you say "consume the internal report representation directly," are you talking about something like the TestReport class?

@kovidgoyal - do we not need to be careful about adding anything to the output? I am scared that adding a fold marker might cause some terminals to misbehave, so choosing the fold marker would have to be done carefully.

[nicoddemus]

Like, when you say "consume the internal report representation directly," are you talking about something like the TestReport class?

Yes, by implementing pytest_runtest_logreport in a plugin. 👍

[kovidgoyal]

@kovidgoyal - do we not need to be careful about adding anything to the output? I am scared that adding a fold marker might cause some terminals to misbehave, so choosing the fold marker would have to be done carefully.

I meant foldmarkers for the editor, like this

test_name {{{
...
}}}

then when you open the report in the editor, all the tests will be folded except for the ones that have errors, since you can choose not to put fold markers around them.

[jeffwright13]

then when you open the report in the editor

Ahh, I see what you are saying. I somehow missed the part about using an editor to look at the results. ==> I blame not having had any coffee yet today. <==

My original idea was to implement something that allowed folding in the same console in which the pytest invocation was done in the first place - something that did not require having to open an external program. Because by the time I do that, I could have just as easily used pytest-html and opened the results in a browser, no?

[jeffwright13]

Just to revisit @kovidgoyal's post about folding in vim: I was actually able to very easily do that with a minor modification to my code (just using his recommended fold markers `{{{ ... }}}``. But I was not able to get the encoded ANSI text to render properly, even after trying some vim ANSI plugins. Also, from an end-user's perspective, (1) vim is not exactly user friendly, so using it will limit the user base; (2) since you're now in an editor and the results from a test run, it is easy to accidentally alter the results, so extra care must be taken.

I am still working on a good TUI solution. I keep waiting for Textual to drop its much-anticipated "css" branch. In the meantime I have an Asciimatics version (very rudimentary) and a Textual version (nicer, but a bit slow). I will make sure to post more as I improve the UI.

[kovidgoyal]

On Fri, Jul 02, 2021 at 04:57:08AM -0700, Jeff Wright wrote: > then when you open the report in the editor Ahh, I see what you are saying. I somehow missed the part about using an editor to look at the results. ==> I blame not having had any coffee yet today. <== My original idea was to implement something that allowed folding in the same console in which the `pytest` invocation was done in the first place - something that did not require having to open an external program. Because by the time I do that, I could have just as easily used pytest-html and opened the results in a browser, no?

The editor runs in the same console as pytest, its a terminal program. You are going to need *something* that draws a UI in the console. You can of course write your own code to do it, but seems a lot of work and people that actually use terminals heavily would probably prefer navigating the scrollback in an editor they use anyway. This is why kitty has the facility to open its scrollback in a pager/editor.

[jeffwright13]

You are going to need something that draws a UI in the console.

Isn't the console already its own UI? Perhaps we are misunderstanding each other.

The way in which I envisioned this working, I would be in-console (I use a Mac and iTerm2). There, I would invoke Pytest (e.g. pytest test_example.py) and then I would be presented with the results in-console. In the case where there were many failures, the console would give me many more lines than I could see in the existing 80x40 panel, and the earliest failure (which is typically what I would want to investigate) would be farthest back. So in order to see where the failures started I would have to scroll back many pages and hunt for the small pytest command among all the other stuff on the command line.

With this envisioned new feature/plugin/whatever, I might be able to say something like `pytest --fold test_example.py' and even though many failed test results might be shown in the console, at least they are now only one line each, instead of 12 lines each, and now (here is the kicker), I can unfold/fold results as needed. Very much like the way Robot Framework does it, if you are familiar with that tool.

In my mind, at least, this would be a time-saver and much more convenient than having to launch a browser or an editor to look at the results.

[kovidgoyal]

On Fri, Jul 02, 2021 at 06:33:23AM -0700, Jeff Wright wrote: > You are going to need *something* that draws a UI in the console. With this envisioned new feature/plugin/whatever, I might be able to say something like `pytest --fold test_example.py' and even though many failed test results might be shown in the console, at least they are now only one line each, instead of 12 lines each, and now (here is the kicker), I can unfold/fold results as needed. Very much like the way Robot Framework does it, if you are familiar with that tool. In my mind, at least, this would be a time-saver and much more convenient than having to launch a browser or an editor to look at the results.

py test would open the editor *for you automatically*, with all but failing tests folded, with the cursor positioned at the first failed test, *in the console*. You would not have to do anything but marvel at its awesomeness.

[jeffwright13]

OK, so that is cool, at least from a "MVP" (minimum viable product) standpoint. Might be a good place to start to get my feet wet, at least.

Questions and concerns:

• Regarding @flub 's comment that it is not generally a good idea to rely on any specific output format - I would like to take that into account, especially since the user can modify the output format drastically with execution flags like -v, --tb, etc. Do you have any insight into this, with your proposed solution in mind?
• I'm not in love with the idea of prompting a question to the tester at the end of a run, so perhaps just setting the EDITOR environment variable, as you suggested, or maybe specifying the editor via command line flag, would be an alternative...and the editor would automagically appear if conmfigured, otherwise the output would be as usual.
• Any special considerations for non-Linux/Mac users, who won't have vi/vim available?
• Any special considerations for DOS/WIndows vs. Linux/Mac EOL characters?
• probably more as I think on this ...

[flub]

What @nicoddemus said:

Yes, by implementing pytest_runtest_logreport in a plugin.

[kovidgoyal]

On Fri, Jul 02, 2021 at 07:14:18AM -0700, Jeff Wright wrote: OK, so that is cool, at least from a "MVP" (minimum viable product) standpoint. Might be a good place to start to get my feet wet, at least. Questions and concerns: - Regarding @flub 's comment that it is not generally a good idea to rely on any specific output format - I would like to take that into account, especially since the user can modify the output format drastically with execution flags like `-v`, `--tb`, etc. Do you have any insight into this, with your proposed solution in mind?

I dont actually use pytest so I have no clue, my solution requires all foldable parts of the output to be surrounded by fold markers.

- I'm not in love with the idea of prompting a question to the tester at the end of a run, so perhaps just setting the EDITOR environment variable, as you suggested, or maybe specifying the editor via command line flag, would be an alternative...and the editor would automagically appear if conmfigured, otherwise the output would be as usual.

Sure use a command line option.

- Any special considerations for non-Linux/Mac users, who won't have vi/vim available?

Have an option/config/env var (aka EDITOR) to specify the editor. If you cant find the editor to execute just print an error about it at the end of the normal output.

- Any special considerations for DOS/WIndows vs. Linux/Mac EOL characters?

Most editors handle those automatically.

[jeffwright13]

I finally had some time to put together a POC (Proof of Concept) for this. Please see https://github.com/jeffwright13/pytest-fold for source code. Yes, I know, very hacky in places. :-)

And here is a screencast of pytest-fold in action: https://drive.google.com/file/d/1kbyI_e5lPQmVBv_v-QvyTDNT4zlRZtIW/view?usp=sharing

I've been up-close with this for so long now that I don't even know if it's anything anybody will ever want to use. Feedback welcome!

[bbelderbos]

Fantastic, kudos on persisting through!