question: Does csview fixed the stream rendering?

[zhiburt]

Hi @wfxr

I bump into csview by

phsym/prettytable-rs#104
#19

I think it's a great idea to not allocate the huge chunk of memory but rather do things lazily.

And I was thinking how to do it myself zhiburt/tabled#123
I was thinking about requiring Clone for iterator so 1 pass for demension the second pass when print.
But it seems to be restrictive to me.

So I looked at your ideas.

And I just wonder if you've fixed the original issue.

Specifically my concern is about this Vec allocation.

csview/src/table/mod.rs

Line 20 in 5a1e6d9

let (widths, buf) = sniff_widths(&mut rdr, header.as_ref(), sniff_rows)?;

csview/src/table/mod.rs

Line 79 in 5a1e6d9

let mut buf = Vec::new();

Am I wrong that essentially we will allocate n * m memory anyway here.

Take care.
Thank you for cross issue references :)

Check list

• I have read through the README
• I have searched through the existing issues

Environment info

• OS
  • Linux
  • Mac OS X
  • Windows
  • Others:

Version

Problem / Steps to reproduce

[wfxr]

Hi @zhiburt , sniff_rows is for detecting reasonable widths for each column. It is provided by the cli options --sniff (default 1000). And you are right, we still need to pre-allocate some memory to store the rows during sniffing because we cannot read them twice. But the memory usage will no longer grow once sniff finished. It's not a big deal since we only need to store constant number of rows.

The benchmarks section shows the memory usage in some common scenarios:
https://github.com/wfxr/csview#benchmark

[wfxr]

csview/src/table/mod.rs

Line 20 in 5a1e6d9

let (widths, buf) = sniff_widths(&mut rdr, header.as_ref(), sniff_rows)?;

csview/src/table/mod.rs

Line 79 in 5a1e6d9

let mut buf = Vec::new();

Am I wrong that essentially we will allocate n * m memory anyway here.

buf here is to store the sniffed rows, not all rows from input.

[zhiburt]

Hi @zhiburt , sniff_rows is for detecting reasonable widths for each column. It is provided by the cli options --sniff (default 1000). And you are right, we still need to pre-allocate some memory to store the rows during sniffing because we cannot read them twice. But the memory usage will no longer grow once sniff finished. It's not a big deal since we only need to store constant number of rows.

buf here is to store the sniffed rows, not all rows from input.

Understood

But when the --sniff argument is not provided we essentially will allocate memory for all rows right?

PS: Correct me if I am wrong but in case --sniff was 1000 but 1001 row "very big" compared to the ones in first 1000 chunk the displayed table will be broken?

[zhiburt]

As I understand it is assumed that --sniff always be set to some value right?

csview/src/table/mod.rs

Line 26 in 5a1e6d9

let widths = &self.widths;

csview/src/cli.rs

Lines 41 to 42 in 5a1e6d9

	#[clap(long, default_value_t = 1000, name = "LIMIT")]
	pub sniff: usize,

[wfxr]

As I understand it is assumed that --sniff always be set to some value right?

@zhiburt Right. If sniff not set by user it will take the default value 1000. You can also assign it to 0 which will disable sniff mechanism and in this case csview will load all rows in memory to calculate the exact width of each column before printing.

[wfxr]

PS: Correct me if I am wrong but in case --sniff was 1000 but 1001 row "very big" compared to the ones in first 1000 chunk the displayed table will be broken?

In this case the "very big" field will be truncated:

csview/src/table/row.rs

Line 29 in 5a1e6d9

.map(|(cell, &width)| cell.unicode_pad(width, Alignment::Left, true))

[zhiburt]

I like your approach 👍

Thank you for the clarification.
Sorry for bothering

And once again take care.