v5.0.0

New Feature

• Supports UDAF falling back.
• Supports native round-robin partitioner.
• Supports native range partitioner.
• Supports native WindowGroupLimitExec introduced in Spark-3.5.
• Supports SHJ falling back to SMJ when built side is too big.
• Fully supports to Apache Celeborn shuffle service.
• Initial supports to Apache Uniffle shuffle service.
• Initial supports to Apache Paimon datasource.

Improvement

Improved memory management in AggExec/SortMergeJoinExec, reducing number of OOMs.
Imptoved metric statistics.

Bug fixes

• Fixed inconsistent string to data casting.
• Fixed inconsistent bloom filter join when bloom filter is generated by Spark.
• Fixed incorrect sort ordering when writing tables with dynamic partitions.
• Fixed inconsistent sha2x functions.
• Fixed a lot of bugs those might lead to query failure, see What's Changed.

What's Changed

• release version v4.0.1 by @richox in #690
• fix incorrect expression conversion: Days should be DayOfMonth by @richox in #691
• fix ci: cache spark binaries by @richox in #696
• Bump smallvec from 2.0.0-alpha.7 to 2.0.0-alpha.8 by @dependabot in #692
• Bump prost from 0.13.3 to 0.13.4 by @dependabot in #688
• Dev repartitioning by @gy11233 in #693
• Add Blaze icon and issue navigation In IDEA by @cxzl25 in #699
• [BLAZE-700] Minor nit fix for hyperlink by @merrily01 in #701
• [BLAZE-706] Fix year/month/day functions data type by @wForget in #703
• [BLAZE-704] Specify name for spark ext function by @wForget in #705
• Fix some incorrect module name mapping in docker compose file by @harveyyue in #709
• fix ci: trigger ci when opening PR by @richox in #711
• Support native scan hive paimon cow table by @harveyyue in #708
• Automatically use the protoc version downloaded by the maven plugin by @cxzl25 in #702
• [BLAZE-707][FOLLOWUP] NativePaimonTableScanExec should use shimed PartitionedFile and min partition number by @SteNicholas in #713
• fix ci: trigger ci when opening/changing PR by @richox in #714
• [BLAZE-287][FOLLOWUP] BlazeCelebornShuffleWriter should use mapped shuffle id for rerunning stage of fetch failure by @SteNicholas in #712
• Bump sonic-rs from 0.3.16 to 0.3.17 by @dependabot in #694
• Bump smallvec from 2.0.0-alpha.8 to 2.0.0-alpha.9 by @dependabot in #698
• Bump foldhash from 0.1.3 to 0.1.4 by @dependabot in #710
• feat(spill): Align with the multi IO compression codec in spill by @zuston in #657
• bug fixes by @richox in #717
• Fix OrcScan reads missing data column by @ASiegeLion in #716
• fix test failures by @richox in #720
• [BLAZE-725] Bump Spark from 3.5.3 to 3.5.4 by @SteNicholas in #726
• Fix MacOS compile by @cxzl25 in #724
• Apply spotless by @cxzl25 in #728
• Remove bug_report unnecessary information by @cxzl25 in #727
• fix mvn build helper by @richox in #735
• Duplicated project schema will cause index out of bounds exception in orc_exec by @harveyyue in #723
• [BLAZE-729] Fix a typo in the Shebang line of the shell script by @merrily01 in #730
• Fix orc map type entries field naming issue by @harveyyue in #732
• Bump itertools from 0.13.0 to 0.14.0 by @dependabot in #733
• close inactive issues by @richox in #738
• [BLAZE-736] Write time should increment for mapperEnd in CelebornPart… by @HYBG-1126 in #739
• fix ci: use huaweicloud mirror to download spark binaries by @richox in #742
• Bump tempfile from 3.14.0 to 3.15.0 by @dependabot in #741
• Bump async-trait from 0.1.83 to 0.1.84 by @dependabot in #740
• fix performance issues by @richox in #743
• [BLAZE-744] Bump Celeborn version from 0.5.2 to 0.5.3 by @SteNicholas in #745
• Dev repartitioning by @gy11233 in #734
• Bump Paimon version from 0.9.0 to 1.0.0 by @harveyyue in #751
• [BLAZE-747] Enhance the ArrowFFIExporter.exportNextBatch method to execute conditionally by @merrily01 in #748
• fix range repartitioning proto issue by @gy11233 in #752
• Bump async-trait from 0.1.84 to 0.1.85 by @dependabot in #746
• Bump uuid from 1.11.0 to 1.11.1 by @dependabot in #754
• Bump tokio from 1.42.0 to 1.43.0 by @dependabot in #755
• fix ci: update to actions/upload-artifact@v4 by @richox in #756
• fix ci: update to actions/upload-artifact@v4 by @richox in #757
• fix ci: add --all-opens for supporting jdk17 by @richox in #758
• fix ci: use cached spark-bin directory to walk around permission denied issue by @richox in #766
• fix-ci: use specified jdk version by @richox in #767
• fix-ci: adjust memory configuration by @richox in #768
• Bump uuid from 1.11.1 to 1.12.0 by @dependabot in #765
• Bump log from 0.4.22 to 0.4.25 by @dependabot in #764
• [BLAZE-747][FOLLOW-UP] Fix user changed in FFI NextBatch by @Flyangz in #769
• Bump smallvec from 2.0.0-alpha.9 to 2.0.0-alpha.10 by @dependabot in #770
• [BLAZE-762] Return null when log function input is negative by @wForget in #763
• [BLAZE-760] Fallback shuffle exchange when range partitioning with unsupported type by @wForget in #761
• supports falling back hash join to sort merge join when hash table is too big by @richox in #753
• fix-ci: remote incorrect cache by @richox in #779
• fix-ci: rust fmt by @richox in #780
• Add comma to line in README file by @xleoken in #778
• bug fixes by @richox in #777
• [BLAZE-775] Support float type for sum function by @wForget in #776
• [BLAZE-773] Support long type for floor function by @wForget in #774
• fix-ci: pull_request_target -> pull_request by @richox in #782
• fix build error and code style by @wForget in #781
• Bump uuid from 1.12.0 to 1.12.1 by @dependabot in #783
• [BLAZE-786] Mark big decimal value convertion as unsupported by @wForget in #787
• use better aggregate OwnedKey construction by @richox in #784
• [BLAZE-790] Support LZ4_RAW compression codec for parquet by @SteNicholas in #791
• Add support of mac aarch64 for tpcds data generator by @zuston in #792
• Automatic cancel previous CI tests when newly commit comes for per PR by @zuston in #794
• Add support of pprof dump for rust execution by @zuston in #793
• Bump tempfile from 3.15.0 to 3.16.0 by @dependabot in #802
• Bump serde from 1.0.216 to 1.0.217 by @dependabot in #800
• Bump poem from 1.3.59 to 3.1.6 by @dependabot in #799
• Bump rand from 0.8.5 to 0.9.0 by @dependabot in #801
• Bump bytes from 1.9.0 to 1.10.0 by @dependabot in #811
• Bump async-trait from 0.1.85 to 0.1.86 by @dependabot in #810
• Add support of Apache Uniffle for remote shuffle service by @zuston in #796
• use separated thread in ffi exporter by @richox in #788
• Fix the rootless-docker action failure when building the jar in github action by @zuston in #813
• Bump uuid from 1.12.1 to 1.13.1 by @dependabot in #814
• Add support of building native with --features by @zuston in #797
• Add support of memory profile by @zuston in #798
• [BLAZE-808] Support statistics of ExecutionPlan for WindowExec by @SteNicholas in #809
• [BLAZE-805] Support statistics of ExecutionPlan for SortExec by @steni...

v4.0.1

New Feature

• Initial supports to ORC input file format.
• Initial supports to RSS framework and Apache Celeborn shuffle service.

Improvement

• Optimize AggExec by supporting Implement columnar-based aggregation.
• Use custom implemented hashmap implement for aggregation.
• Supports specialized count(0).
• Optimize bloom filter by reusing same bloom filter in the same executor.
• Optimize bloom filter by supporting shrinking.
• Optimize reading parquet files by supporting parallel reading.
• Improve spill file deletion logics.

Bug fixes

• Fix file not found for path with url encoded character.
• Fix Hashaggregate convert job throwing ScalaReflectionException.
• Fix pruning error while reading parquet files with multiple row groups.
• Fix incorrect number of tasks due to missing shuffleOrigin.
• Fix record batch creating error when hash joining with empty input.

Other

• Upgrade datafusion/arrow dependency to v42/v53.
• Replace gxhash with foldhash for better compatibility on some hardwares.
• Other minor improvement & fixes.

What's Changed

• update to datafusion-v42 by @richox in #574
• doc: update tpc-h benchmark result by @richox in #614
• fix running error for Spark 3.2.0 and 3.2.1 by @XorSum in #602
• [BLAZE-287][FOLLOWUP] Use JavaUtils#newConcurrentHashMap to speed up ConcurrentHashMap#computeIfAbsent by @SteNicholas in #615
• fix pruning error while reading parquet files with multiple row groups by @richox in #616
• use custom implemented hashmap for aggregation by @richox in #617
• supports specialized count(0) by @richox in #619
• optimize bloom filter by @richox in #620
• [BLAZE-627] Make ORC and Parquet format detection more generic by @dixingxing0 in #628
• Bump object_store from 0.11.0 to 0.11.1 by @dependabot in #622
• improve memory management by @richox in #621
• Bump bytes from 1.7.2 to 1.8.0 by @dependabot in #625
• Bump tokio from 1.40.0 to 1.41.0 by @dependabot in #629
• Bump uuid from 1.10.0 to 1.11.0 by @dependabot in #618
• replace gxhash with foldhash by @richox in #624
• Bump sonic-rs from 0.3.13 to 0.3.14 by @dependabot in #623
• add new blaze logo by @richox in #633
• improve NativeOrcScan by @richox in #631
• fix Hashaggregate convert job throw ScalaReflectionException by @leizhang5s in #637
• tpcd benchmarkrunner : add orc format support by @leizhang5s in #639
• Bump bigdecimal from 0.4.5 to 0.4.6 by @dependabot in #638
• [BLAZE-573][FOLLOWUP] Bump Spark from 3.4.3 to 3.4.4 by @SteNicholas in #640
• Bump tempfile from 3.13.0 to 3.14.0 by @dependabot in #641
• parquet reading improvements by @richox in #650
• Bump tokio from 1.41.0 to 1.41.1 by @dependabot in #642
• Bump sonic-rs from 0.3.14 to 0.3.16 by @dependabot in #647
• AggExec: implement columnar accumulator states. by @richox in #646
• chore: Make spotless plugin happy by @zuston in #653
• fix(spill): Explicitly delete spill file for FileBasedSpillBuf after release by @zuston in #654
• fix(spill): Delete spill file when dropping for rust FileSpill by @zuston in #660
• [BLAZE-664] Bump Celeborn version from 0.5.1 to 0.5.2 by @SteNicholas in #665
• [MINOR] Avoid NPE when native lib is not found by @wForget in #668
• Fix build on windows by @wForget in #666
• fix(shuffle): Progagate shuffle origin to native exchange exec to make AQE rebalance valid by @zuston in #663
• improvement: Add numOfPartitions metrics for exchange exec to align with vanilla spark by @zuston in #669
• code refactoring by @richox in #658
• Introduce base blaze sql test suite by @wForget in #674
• Bump bytes from 1.8.0 to 1.9.0 by @dependabot in #671
• Bump tokio from 1.41.0 to 1.41.1 by @dependabot in #676
• code refactoring by @richox in #677
• Fix file not found for path with url encoded character by @wForget in #679
• Followup to #674, add -r for rm by @wForget in #681
• Create RecordBatch with num_rows option to avoid bhj error caused by empty output_schema by @wForget in #683
• Bump tokio from 1.41.1 to 1.42.0 by @dependabot in #680
• reduce jar package size and fix CI check error by @richox in #687

New Contributors

• @dixingxing0 made their first contribution in #628
• @leizhang5s made their first contribution in #637
• @wForget made their first contribution in #668

Full Changelog: v4.0.0...v4.0.1

v4.0.0

New features

• supports spark3.0/3.1/3.2/3.3/3.4/3.5.
• supports integrating with Apache Celeborn.
• supports native ORC input format.
• supports bloom filter join introduced in spark 3.5.
• supports forceShuffledHashJoin for running tpch/tpcds benchmarks.
• new supported native expression/functions: year, month, day, md5.

Bug fixes

• add missing UDTF.terminate() invokes.
• fix NPE while executing some native spark physical plans.

Performance

• use custom implemented hash table for faster joining, supporting SIMD, bulk searching, memory prefetching, etc.
• improve shuffle write performance.
• reuse FSDataInputStream for same input file.

What's Changed

• spark351 + supports bloom filter join by @richox in #532
• Support Spark 3.2.0 by @XorSum in #538
• Bump sonic-rs from 0.3.8 to 0.3.9 by @dependabot in #533
• Bump tonic-build from 0.12.0 to 0.12.1 by @dependabot in #525
• Bump prost from 0.13.0 to 0.13.1 by @dependabot in #521
• supports UDTF.terminate by @richox in #536
• Bump bytes from 1.6.1 to 1.7.0 by @dependabot in #539
• Bump tokio from 1.38.0 to 1.39.2 by @dependabot in #534
• Bump tempfile from 3.10.1 to 3.12.0 by @dependabot in #542
• Bump bytes from 1.7.0 to 1.7.1 by @dependabot in #540
• supports forceShuffledHashJoin. by @richox in #545
• init RSS framework by @richox in #551
• Bump radsort from 0.1.0 to 0.1.1 by @dependabot in #550
• Bump postcard from 1.0.8 to 1.0.10 by @dependabot in #552
• Bump tonic-build from 0.12.1 to 0.12.2 by @dependabot in #554
• Bump sonic-rs from 0.3.9 to 0.3.12 by @dependabot in #555
• improve shuffle performance by @richox in #559
• Bump prost from 0.13.1 to 0.13.2 by @dependabot in #556
• Bump async-trait from 0.1.81 to 0.1.82 by @dependabot in #557
• Bump tokio from 1.39.2 to 1.40.0 by @dependabot in #558
• delete walkaround_sliced_boolean_array_issue by @DDDominik in #560
• fix CI by @richox in #562
• optimize hash joins by @richox in #563
• support year/month/day functions by @richox in #565
• Support Spark 3.1.3 by @harveyyue in #561
• Bump sonic-rs from 0.3.12 to 0.3.13 by @dependabot in #566
• WIP: improve shuffle write performance by @richox in #564
• remove variables from artifacts in pom.xml by @richox in #569
• add --allow-no-vcs to cargo fix command by @richox in #570
• use memory prefetch in hash map building by @richox in #571
• support native scan orc format by @harveyyue in #544
• add spark prefix to tpcds.gen.defaultParallel to take affect by @TJX2014 in #572
• support md5 command by @KnightChess in #581
• Bump bytes from 1.7.1 to 1.7.2 by @dependabot in #578
• update doc to add protobuf installation guides by @richox in #583
• Bump prost from 0.13.2 to 0.13.3 by @dependabot in #586
• make RUSTFLAGS configurable by @XorSum in #585
• [BLAZE-587] Replace spark pattern with spark- for maven profile and shim name by @SteNicholas in #588
• use hadoop fs positioned-reading by @richox in #582
• [BLAZE-573] Support Spark 3.4 version by @SteNicholas in #589
• docs/fix: fix error message by @caicancai in #591
• reuse FSDataInputStream for same input file by @richox in #592
• make RUSTFLAGS effective before running cargo commands by @harveyyue in #594
• [BLAZE-287] Integrate Blaze with Celeborn by @RexXiong in #596
• fix spark3.1 compile error by @richox in #598
• fix timer in AggExec by @richox in #597
• update docs (celeborn related) by @richox in #599
• Change the output qualifier from var to def will avoid invoking NPE by @harveyyue in #608
• OrcExec should read the specified file range data by @harveyyue in #605
• Bump tempfile from 3.12.0 to 3.13.0 by @dependabot in #604
• Bump once_cell from 1.20.0 to 1.20.1 by @dependabot in #603
• Bump tonic-build from 0.12.2 to 0.12.3 by @dependabot in #595
• Bump async-trait from 0.1.82 to 0.1.83 by @dependabot in #590
• Bump futures-util from 0.3.30 to 0.3.31 by @dependabot in #609
• Bump futures from 0.3.30 to 0.3.31 by @dependabot in #610
• Bump once_cell from 1.20.1 to 1.20.2 by @dependabot in #611

New Contributors

• @XorSum made their first contribution in #538
• @DDDominik made their first contribution in #560
• @TJX2014 made their first contribution in #572
• @KnightChess made their first contribution in #581
• @SteNicholas made their first contribution in #588
• @caicancai made their first contribution in #591
• @RexXiong made their first contribution in #596

Full Changelog: v3.0.1...v4.0.0

v3.0.1

blaze-v3.0.1

Features

• Supports spark3.0/3.2/3.3.

Performance

fix GetJsonObject conversion, supporting faster get_json_object with sonic-rs.

Bugfix

• fix childOrderingRequiredTag computation logic.

v3.0.0 [yanked]

blaze-v3.0.0 [yanked]

Features

• Supports using spark.io.compression.codec for shuffle/broadcast compression
• Supports date type casting
• Refactor join implementations to support existence joins and BHJ building hash map on driver side

Performance

• Fixed performance issues when running on spark3 with default configurations
• Use cached parquet metadata
• Refactor native broadcast to avoid duplicated broadcast jobs
• Supports spark333 batch shuffle reading

Bugfix

• Fix in_list conversion in from_proto.rs

v2.0.9.1

release version 2.0.9.1 (#470)

Co-authored-by: zhangli20 <[email protected]>

v2.0.9

v2.0.9

v2.0.8

v2.0.8

v2.0.7

update blaze version 2.0.7-SNAPSHOT (#312)

Co-authored-by: zhangli20 <[email protected]>

v2.0.6

update blaze version 2.0.6-SNAPSHOT