Add the PivotGuide

[thejud]

Per issue #2313 Add a new PivotGuide

[saulpw]

Hey @thejud, I don't think we should have these bulky tables in the Guides. The VisiData renderer can't handle them anyway. Do you think there's a way to condense this? This one might be pretty tricky.

[thejud]

Hey @saulpw: Yes, I was torn about the tables. However, for melt and pivot in particular, seeing what happens was the best way for me to understand what was going on. I actually previewed them within VD to ensure that they would render.

Do you mean that these guides are simply too text heavy, or that the tables themselves are problematic because of wrappings, etc.., or both :)

Do you have suggestions about what you'd like to see?

[thejud]

I'll try the following. Please met me know if this makes sense, or if you prefer alternatives:

1. Remove the Melt section... there is a separate guide for that anyway.

2. Remove the tables.

   • Alternatively, would it be ok to simply condense them, e..g 2 char column names, less space, etc.. so they only take up, say 40 chars of width max?
   • Do you not want to have this sort of visual complexity (text tables) in the guides at all?

3. Condense/clean up some of the explanatory text. Looks like the intent of the guide is more like a quickref: show HOW to do specific things that the user knows about, not explain capabilities or concepts like pivot tables.

[saulpw]

Yes, that makes sense. "See also Melt" to reference the (currently non-existant) page. And it's true, I don't like the visual clutter of the tables in the Guides, but I don't use the Guides so we're asking on Discord about whether the tables should be condensed or removed; 40 wide by 8 tall would be my suggested max, and using underline for the header and no outer border (borders only between cells).

Also, yes, I think we should only have explanatory text for features that are special/unique to VisiData. If it's a term that can be easily looked up with a thousand examples on the web, then we don't need to be a tutorial (and that text gets in the way for people who already know about data, which are most VisiData users).

[thejud]

Something like:

date       | color | price
-----------+-------+------
2024-09-01 | Red   | 30   
2024-09-02 | Blue  | 28   
2024-09-02 | Green | 64   
2024-09-03 | Red   | 100  
2024-09-03 | Blue  | 33   
2024-09-03 | Blue  | 99   

or even more minimal like:

date       Red Blue Green
2024-09-01 1   0    0
2024-09-02 0   1    1
2024-09-03 1   2    0

Note that I chose the original format because it is the most explicit, and because it is both easy to parse and easy to generate using miller (mlr)... I can easily write filters to clean it up to whatever, however.

As a (possible) point of interest, I generated and edited the table in visidata, copied it to the system clipboard as TSV (gs gY), pasted it into vim where I am writing the guide, and then filtered it through mlr --t2p --barred cat to transform TSV into pretty format with bars.

[saulpw]

Could we do:

[:underline]date      | Red | Blue | Green[:]
2024-09-01 | 1 |  0  |  0
2024-09-02 | 0 |  1  |  1
2024-09-03 | 1 |  2  |  0

with spaces such that everything lines up?

[reagle]

When I was working on a guide I asked about this in #2456 when I realized Guide's didn't support URLs (#2466), nested lists, nor simple tables. Actual markdown support (while adopting a syntax for visidata internals, such as commands) would be ideal, especially if Guides are supposed to be the source of downstream documentation, but that might be a significant effort in itself. (I imagine there are simple libraries in python with permissive licenses.)

With an eye toward markdown compatibility and future support, perhaps use a standard/extent formats for tables format rather than inventing a new one (e.g., [:underline] for heading row).

[thejud]

yes, I definitely prefer a 'plain-text compatible' format for tables. That's one of the reasons that I avoid the markdown table flavors: columns aren't always visually aligned. The pandoc format seems reasonable and does offer formatting benefits (left, center and right + multi-line cells.) Pandoc also appears to support variants of the "pipe format" that I used.

I wanted something that could be automatically converted back and forth for ease of maintenance, both during initial authoring and for future updates. I'm not quite sure about the tooling flow for converting TSV/CSV <-> pandoc tables, as I haven't worked with pandoc much.

That's part of the reason I shared the mlr workflow, since it seems like a standardized scriptable way of generating AND parsing the table. I haven't played with VisiData's fixed-width input format, but using VD itself to parse the table is intriguing.

[anjakefala]

Hi @thejud Do you have everything you need to update this guide? There seemed to be a definite consensus of tables being welcome in the guides. Have you settled on a style from the conversations?

[thejud]

@anjakefala thanks for putting it to the vote up for comments :)

I added a simplified version of the tables. Hopefully this is reasonable middle ground. For what it's worth, I ended up running the TSV output through the python tabulate command with pbpaste | tabulate -s '\t'. There were some minor formatting cleanups required, but it seems reasonable given the infrequent need to update them.

I also moved the examples to a separate section at the end of the document with the idea that they could be additive, not interfere with quick lookup. This is an approach that pandas takes for their examples, which I find extremely helpful.

LMK if the example table should be added to the example data repository, or if it's fine existing only in the docs.

Thank you and @saulpw for working through these issues with me.

[anjakefala]

I loved the example @thejud and the step-by-step guidance! We tried to clarify the description of Pivot Tables (they are so challenging to describe), and restructure the example prose to be key (longname) to Foo.

Those were the main things. Thank you so much for tackling these chunkier guides!