spurious error message when using qsv header

[njamescouk]

I'm on windows 7 and getting this "error"
constantly although the pipeline produces
correct output.

qsv excel -s 0 random.xlsx | qsv headers
[succesful output redacted]
Broken pipe: The pipe has been ended. (os error 109)

Am I using qsv headers wrongly?

hugely impressive program btw.

[jqnatividad]

Rust's String type requires UTF-8 encoding, and since CSV files are Text Files that are primarily processed as Strings, they need to be UTF-8 encoded.

This is typically not a problem with other operating systems because UTF-8 is the de facto encoding standard.

Unfortunately, Window's default encoding is UTF-16LE, which presents a problems when you redirect/pipe qsv's output, as anytime you do so, the Windows operating system changes the encoding to UTF-16LE. Ironically, even Microsoft's own Excel cannot properly open a UTF-16LE encoded CSV file.

Regardless, the easiest way around this limitation is to use qsv's --output option which creates a UTF-8 encoded file and refrain from piping/redirecting input/output in Windows 7, e.g.

qsv excel -s 0 random.xlsx --output random.csv
qsv headers random.csv

Having said that, qsv is actually quite tolerant of non UTF-8 encoded files as it often uses csv::ByteRecords when reading CSV files, handling the input as raw bytes (count, stats, index, headers, etc). However, qsv commands that require string manipulation like apply will fail if the input is not UTF-8 encoded, and those that do work may show spurious errors like the one you saw with headers.

If you need to stay on Windows 7 and want to use piping and redirection with qsv to take advantage of its command line composability, you may want to install an alternative Command Line Interface (CLI) environment like Cygwin.

It allows you to get that Linux feeling (and more importantly, its default UTF8-encoding!) - on Windows 😄

You can just use Cygwin as your "qsv command-line environment" - no need to install any extra packages beyond bash and bash-completion, so you also get the benefit of tab completion (pressing the Tab key auto-completes commands and arguments), and even install qsv's tab completion support, if you're so inclined.

And I'm glad you find it impressive. Most qsv users are on Linux and Mac, and I only do testing on Windows 11.

I'm actually quite surprised it still runs on Windows 7! The spurious message you're seeing may even be partly because of that... as qsv is built on all the latest stuff - for Rust (1.67.1) and all its crate dependencies since Windows 7 is EOLed and even the extended security updates have finally ended after being extended several times.

I hope you're not browsing the web using it... 😅

[njamescouk]

thanks for the reply.

the last time it was on the web was to download service pack 1! strictly air gapped now.

I had a go with cygwin years ago but it was a bit of a faff hopping between platforms. I've got all the msys2 gear which is very handy for the unixy stuff.

I've just had a try putting iconv in the pipeline, but no joy, likewise chcp 65001 doesn't seem to have any effect. I think the best bet is to get it in an sqlite database asap.

if it doesn't see me out, the next computer will be linux / bsd so my problems will be just beginning over.

irrespective of all this qsv is a lifesaver, I've spent the last 9 years intermittently enhancing a csv rewriter based off a lex / yacc combo which has finally succumbed to featuritis. I can get on with some work now!

[njamescouk]

windows 7 / gitbash combo works without complaint.