How to use "qsv to" to generate parquet file?

[aborruso]

Hi,
when I run qsv to parquet mydir tmp.csv I have:

Could not match 'to' with any of the allowed variants: ["apply", "behead", "cat", "count", "dedup", "diff", "enum", "excel", "exclude", "explode", "extdedup", "extsort", "fetch", "fetchpost", "fill", "fixlengths", "flatten", "fmt", "foreach", "frequency", "generate", "headers", "help", "index", "input", "join", "jsonl", "partition", "pseudo", "rename", "replace", "reverse", "safenames", "sample", "schema", "search", "searchset", "select", "slice", "snappy", "sniff", "sort", "sortcheck", "split", "stats", "table", "transpose", "tojsonl", "validate"]

My QSV version: qsv 0.106.0-mimalloc-apply;fetch;foreach;generate;magic-0.13.0;self_update-20-20;6.09 GiB-1.98 GiB-5.65 GiB-7.61 GiB (x86_64-unknown-linux-musl compiled with Rust 1.70.0) prebuilt

How to use "qsv to" to generate parquet file?

Thank you

[jqnatividad]

Hi @aborruso , the must prebuilt doesn't have to and polars features enabled as GitHub's action runners seem to run out of space while building, so I ended up disabling them on musl.

I suggest you install/compile from source if you want to use those features.

As for generating parquet files, you may also want to look into the sqlp command, as it can save query results in parquet, json and arrow formats.

[aborruso]

As for generating parquet files, you may also want to look into the sqlp command, as it can save query results in parquet, json and arrow formats.

To use it do I must install/compile from source, or is it in the prebuilt ?

[jqnatividad]

to is not in the musl prebuilt, so you must install/compile from source if you want to use the to command.

Alternatively, you can download the x86_64-unknown-linux-gnu zip as it has the sqlp command which allows parquet export.

The qsv_glibc-2.31 should run in your system. You can rename it to just qsv once you confirm it runs on your system.

[aborruso]

Thank you

[aborruso]

Alternatively, you can download the x86_64-unknown-linux-gnu zip as it has the sqlp command which allows parquet export.

Where could I find a complete simple example, that uses sqlp, to run something like "select * from mydata", to convert CSV to parquet?

Thank you

[jqnatividad]

Hi @aborruso , I just added an examples section to sqlp:

89f0065

[aborruso]

Hi @aborruso , I just added an examples section to sqlp:

with these examples, I'm able to understand it too, that I'm a bit dumb. A thousand thanks

[aborruso]

Hi @aborruso , I just added an examples section to sqlp:

How to apply this kind of options?

You might want to try:
- increasing `infer_schema_length` (e.g. `infer_schema_length=10000`),
- specifying correct dtype with the `dtypes` argument
- setting `ignore_errors` to `True`,

Thank you

[jqnatividad]

Those options are not exposed right now @aborruso .

Can you create a feature request issue explaining your use case?

[aborruso]

Can you create a feature request issue explaining your use case?

#1047