0.135.0

Highlights

JSON Schema validation just got a whole lot more powerful with the introduction of qsv's custom dynenum keyword!
With dynenum, you can now dynamically lookup valid enum values from a CSV (on the filesystem or on a URL), allowing for more flexible and responsive data validation.

Unlike the standardenum keyword, dynenum does not require hardcoding valid values at schema definition time, and can be used to validate data against a changing set of valid values.

For an example, see #1872 (reply in thread).

In an upcoming qsv pro release, we're planning on making dynenum even more powerful by allowing you to easily specify high-value reference data (e.g. US Census data, World Bank data, data.gov, etc.) that is maintained at data.dathere.com and other CKAN instances.

This release also add the custom currency JSON Schema format, which enables currency validation according to the ISO 4217 standard.

The Polars engine was also upgraded to 0.43.1 at the py-1.81.1 tag - making for various under-the-hood improvements for the sqlp, joinp and count commands, as we set the stage for more Polars-powered features in future releases.

---

Added

• foreach: enabled foreach command on Windows prebuilt binaries def9c8f
• lens: added support for QSV_SNIFF_DELIMITER env var and snappy auto-decompression 8340e89
• sample: add --max-size option e845a3c
• validate: added dynenum custom JSON Schema keyword for dynamic validation lookups #2166
• tests: add tests for https://100.dathere.com/lessons/2 by @rzmk in #2141
• added stats_sorted and frequency_sorted benchmarks
• added validate_dynenum benchmarks

Changed

• json: add error for empty key and update usage text by @rzmk in #2167
• prompt: gate prompt command behind prompt feature #2163
• validate: expanded currency JSON Schema custom format to support ISO 4217 currency codes and alternate formats 5202508
• validate: migrate to new jsonschema crate api 5d65054
• Update ubuntu version for deb package by @tino097 in #2126
• contrib(completions): update completions for qsv v0.134.0 and fix subcommand options by @rzmk in #2135
• contrib(completions): add --max-size completion for sample by @rzmk in #2142
• deps: bump to polars 0.43.1 at py-1.81.1 #2130
• deps: switch back to calamine upstream instead of our fork 677458f
• build(deps): bump actix-governor from 0.5.0 to 0.6.0 by @dependabot in #2146
• build(deps): bump anyhow from 1.0.87 to 1.0.88 by @dependabot in #2132
• build(deps): bump arboard from 3.4.0 to 3.4.1 by @dependabot in #2137
• build(deps): bump bytes from 1.7.1 to 1.7.2 by @dependabot in #2148
• build(deps): bump geosuggest-core from 0.6.3 to 0.6.4 by @dependabot in #2153
• build(deps): bump geosuggest-utils from 0.6.3 to 0.6.4 by @dependabot in #2154
• build(deps): bump jql-runner from 7.1.13 to 7.2.0 by @dependabot in #2165
• build(deps): bump jsonschema from 0.18.1 to 0.18.2 by @dependabot in #2127
• build(deps): bump jsonschema from 0.18.2 to 0.18.3 by @dependabot in #2134
• build(deps): bump jsonschema from 0.18.3 to 0.19.1 by @dependabot in #2144
• build(deps): bump jsonschema from 0.19.1 to 0.20.0 by @dependabot in #2152
• build(deps): bump pyo3 from 0.22.2 to 0.22.3 by @dependabot in #2143
• build(deps): bump rfd from 0.14.1 to 0.15.0 by @dependabot in #2151
• build(deps): bump simple-expand-tilde from 0.4.0 to 0.4.2 by @dependabot in #2129
• build(deps): bump qsv_currency from 0.6.0 to 0.7.0 by @dependabot in #2159
• build(deps): bump qsv_docopt from 1.7.0 to 1.8.0 by @dependabot in #2136
• build(deps): bump redis from 0.26.1 to 0.27.0 by @dependabot in #2133
• build(deps): bump simdutf8 from 0.1.4 to 0.1.5 by @dependabot in #2164
• bump indirect dependencies
• apply select clippy lint suggestions
• several usage text/documentation improvements
• bump MSRV to 1.81.0

Fixed

• validate: correct fail_validation_error! macro; reformat error messages to use hyphens as the JSONschema error message already starts with "error:" 9a25524
• moved --help output from stderr to stdout as per GNU CLI guidelines #2138
• lens: fixed parsing of lens options 1cdd1bc
• searchset: fixed usage text for <regexset-file> 9a60fb0
• used patched forks of arrow, csvlens and xlsxwriter crates that replaces a dependency on an old version of lexical-core with known soundness issues - https://rustsec.org/advisories/RUSTSEC-2023-0086. Once those crates have updated their lexical-coredependency, we will revert to the original crates.

Removed

• removed prompt command from qsvlite #2163
• publish: remove lens feature from i686 targets as it does not compile 959ca76
• deps: remove anyhow dependency #2150

Full Changelog: 0.134.0...0.135.0