Skip to content

0.124.1
Compare
Choose a tag to compare
@jqnatividad jqnatividad released this 15 Mar 22:05
· 2521 commits to master since this release
0.124.1
1dbfadb
This commit was signed with the committer’s verified signature.
jqnatividad Joel Natividad
SSH Key Fingerprint: KWusDMD3QSuqvkE6hobKlx49ZVw5StyYwBaKvLC8n6c Learn about vigilant mode.

Datapusher+ "Speed of Insight" Release! 🚀🚀🚀

This release is all about speed, speed, speed! We've made qsv even faster by leveraging Polars' multithreaded, mem-mapped CSV reader to get near-instant row counts of large CSV files, and near instant SQL queries and aggregations with Datapusher+ - automagically inferring metadata and giving you quick insights into your data in seconds!

We're demoing our qsv-powered Datapusher+ at the March 2024 installment of CKAN Montly Live on March 20, 2024, 13:00-14:00 UTC. Join us!

Beyond pushing data reliably at speed into your CKAN Datastore (it pushes real good! 😉), DP+ does some extended analysis, processing and enrichment of the data so it can be readily Used.

Both fetch and fetchpost commands now also have a --disk-cache option and are fully synched - forming the foundation for high-speed data enrichment from Web Services - including datHere's forthcoming, fully-integrated Data Enrichment Service.

🏇🏽 Hi-ho Quicksilver, away! 🏇🏽

Added

  • count: automatically use Polars multithreaded, mem-mapped CSV reader when polars feature is enabled to get near-instant row counts of large CSV files even without an index #1656
  • qsvdp: added polars support to Datapusher+-optimized binary variant, so we can do near instant SQL queries and aggregations during DP+ processing #1664
  • fetchpost: added --disk-cache options and synced usage options with fetch #1671
  • extended .infile-list to skip empty and commented lines, and to validate file paths
    20a45c8 and
    2650930

Changed

  • sqlp: automatically disable read_csv() fast path optimization when a custom delimiter is specified #1648
  • refactored util::count_rows() helper to also use polars if available 1e09e17 and 8d321fe
  • publish: updated Windows MSI publish GH Action workflow to use Wix 3.14 from 3.11 75894ef
  • deps: bump polars from 0.38.1 to 0.38.2 5faf90e
  • deps: update Luau from 0.614 to 0.616 eb197fe and 52331da
  • build(deps): bump sysinfo from 0.30.6 to 0.30.7 by @dependabot in #1650
  • build(deps): bump chrono from 0.4.34 to 0.4.35 by @dependabot in #1651
  • build(deps): bump strum from 0.26.1 to 0.26.2 by @dependabot in #1658
  • build(deps): bump qsv-stats from 0.12.0 to 0.13.0 by @dependabot in #1663
  • build(deps): bump anyhow from 1.0.80 to 1.0.81 by @dependabot in #1662
  • build(deps): bump reqwest from 0.11.25 to 0.11.26 by @dependabot in #1667
  • applied select clippy recommendations
  • updated several indirect dependencies
  • added several benchmarks for new/changed commands

Fixed

  • dedup: fixed #1665 dedup not handling numeric values properly by adding a --numeric option #1666
  • joinp: reenable join validation tests now that Polars 0.38.2 join validation is working again 5faf90e and fcfc75b
  • count: broken in unreleased 0.124.0. Polars-powered count require a "clean" CSV file as it infers the schema based on the first 1000 rows of a CSV. This will sometimes result in an invalid "error" (e.g. it infers a column is a number column, when its not). 0.124.1 fixes this by adding a fallback to the "regular" CSV reader if a Polars error occurs a2c0869

Removed

  • gender_guesser 0.2.0 has been released. Remove patch.crates-io entry
    97873a5

Full Changelog: 0.123.0...0.124.1

Contributors

dependabot
Assets 15
Loading