Serial Monitor filters does not work with "solution" for colorized output

[BOBAH1248]

Many people love this wonderful tool, but sometimes they want more, especially since it is at least according to the documentation.

I found that it is based on miniterm part of serial.tools.
And then found that "solution" for colorize just write text as binary data:
https://github.com/pyserial/pyserial/blob/master/serial/tools/miniterm.py#L511-L517
So, colorization of output is done by OS terminal as expected.

Everything would be fine, but this solution also discards all filters! Include the best one esp32_exception_decoder.

Initially I faced this issue in PlatformIO IDE for VSCode under ESP32.
But research show that issue in platformio.exe device monitor, and in dependent 3th party submodule serial.tools.miniterm.

Configuration

Operating system:
Microsoft Windows 11 Pro
Version 10.0.22621 Build 22621

PlatformIO Version (platformio --version):
PlatformIO Core, version 6.1.6

Description of problem

The Serial Monitor filters are not work with "colorize solution":

monitor_filters = esp32_exception_decoder, direct, colorize
monitor_raw = yes   ; this is "colorize solution"

Steps to Reproduce

1. Use ESP32/ESP8266 solution with enabled colorized output to uart, with throwing an "Guru meditation error" (like "assert", "abort" and so)
2. When used platformio.ini with rows as in description, the colors works as expected, but Backtrace: from output is not parsed
3. If comment monitor_raw in platformio.ini, the colors don't works, but Backtrace: works as expected

[BOBAH1248]

It looks like, I managed to find way to avoid this issue on system, that use console with support of ANSI colors (ps: the VS Code support it) refs pull request #1039

[ivankravets]

What is your goal? You want to use filters but you disabled them with monitor_raw. See docs https://docs.platformio.org/en/latest/projectconf/sections/env/options/monitor/monitor_raw.html

The esp32_exception_decoder filter already sends ANSI codes (colors), you don't need extra colorize.

Just try this:

monitor_filters = direct, esp32_exception_decoder

[BOBAH1248]

What is your goal?

My goal is make it usable (in the most possible cases) and clarify the documentation.

You want to use filters but you disabled them with monitor_raw.

I am sure that it will be ugly of me to refer to what you wrote in other topics:
platformio/platformio-vscode-ide#2750 (comment)
https://community.platformio.org/t/monitor-configure-colors-for-line-with-tags/8625/11
extra (the full thread is interesting, not just the last message): https://community.platformio.org/t/some-bizare-characters-prints-on-serial-monitor-with-esp-idf-hello-world-example/5025/37

The esp32_exception_decoder filter already sends ANSI codes (colors), you don't need extra colorize.

According to docs, colorize ("Apply different colors for received and echo") is useful with user output, not device input only

Anyway, "The esp32_exception_decoder filter already sends ANSI codes (colors)" it's good joke

[ivankravets]

The colorize filter does "Apply different colors for received and echo". It just colorizes input/output with different colors. Should we improve our docs? We would be thankful if you point us to the place where is not clear which filter you should to use.

[BOBAH1248]

To be able to see what I sent and what I received from the device in different colors, this should in no way affect what the device sends me, additionally tinting its parts of the lines.
Moreover, all the logs from the device begin to tint the line at the very beginning of the line and cancel it at the very end of the line.
So it turns out that colorize should add the color character after the newline (<CR>/<LF>), and mark the color immediately before the next linefeed.
That is, it should "frame" the output from the device, which means it cannot technically affect the colors inside the line.

[ivankravets]

How about if we just remove colorize from the docs? I see it adds more problems than helps developers. Please note that we can't fix the colorize filter because it is the part of PySerial package.

[BOBAH1248]

Agreed, the colorize (as well some another like direct) is part of miniterm and cannot be easily changed. And due to it adds problems it will be good to remove from docs or marked as "use at your own risk"

[ivankravets]

Resolved in platformio/platformio-docs@ee27d13

[HB-Stratos]

This has not been fully resolved. It has been removed from the docs, but the info string in the terminal still states

--- Terminal on COM5 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, plotter, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H

I excitedly enabled the colorize filter, thinking it was some fancy option that applies trickery to give a useful output, but was then confused when it did nothing, and did not exist in the docs.

I would argue that this should also be removed from the available filters listing here, or at least marked with unsupported or similar.