Apache DataSketches plugin for Trino #27563

sopel39 · 2025-12-05T17:07:15Z

Plugin to query Apache DataSketches (https://datasketches.apache.org/docs/Theta/ThetaSketchFramework.html). This includes merge and estimate function for theta sketch.

Description

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## General
* Add DataSketches functions. ({issue}`issuenumber`)

Supercedes #6643

...ino-datasketches/src/main/java/io/trino/plugin/datasketches/theta/SketchFunctionsPlugin.java

docs/src/main/sphinx/functions/datasketches.md

plugin/trino-datasketches/src/main/java/io/trino/plugin/datasketches/theta/UnionWithParams.java

findepi · 2025-12-08T09:21:07Z

docs/src/main/sphinx/functions/datasketches.md

+```sql
+SELECT
+  o_orderdate AS date,
+  theta_sketch_estimate(theta_sketch_union(o_custkey_sketch)) AS unique_user_count,


I see only functions that take a sketch as a parameter.
How to i make my first sketch?

Sketches are produced either by other engines or ingestion pipelines (e.g. Spark or Hive) rather than Trino itself. Apache DataSketches are used quite commonly.

Plugin to query Apache DataSketches (https://datasketches.apache.org/docs/Theta/ThetaSketchFramework.html). This includes merge and estimate function for theta sketch.

findepi · 2025-12-10T12:47:04Z

...n/trino-datasketches/src/test/java/io/trino/plugin/datasketches/theta/TestMergeEstimate.java

+    }
+
+    @Test
+    public void testSimpleMerge()


This unit test should not be in AbstractTestQueryFramework class.
it doesn't use QueryRunner, so running this test should not require paying the cost of QR setup

findepi · 2025-12-10T12:47:30Z

...n/trino-datasketches/src/test/java/io/trino/plugin/datasketches/theta/TestMergeEstimate.java

+        state1 = factory.createSingleState();
+        state2 = factory.createSingleState();


is this shared mutable state in a concurrent test class?

findepi · 2025-12-10T12:49:36Z

plugin/trino-datasketches/src/main/java/io/trino/plugin/datasketches/theta/UnionWithParams.java

+        state.setNominalEntries(nominalEntries);
+        state.setSeed(seed);
+        state.setSketch(inputValue);


This looks the code overrides state in the @InputFunction function.
Isn't the @InputFunction function supposed to accumulate?

findepi · 2025-12-10T12:52:43Z

docs/src/main/sphinx/functions/datasketches.md

+# DataSketches functions
+
+DataSketches is a high-performance library of stochastic streaming
+algorithms, commonly called sketches. Sketches are small, stateful programs


This document uses "sketch" to describe an algorithm (here), but also a data structure (when used as a type name).
This is confusing.

findepi · 2025-12-10T12:53:41Z

docs/src/main/sphinx/functions/datasketches.md

+is available through {func}`theta_sketch_union` and
+{func}`theta_sketch_estimate`, typically used in `COUNT DISTINCT` queries.


If this is a typical usage, do you want to show it in an example?
Or did you mean "instead of count(DISTINCT ...) queries"?

findepi · 2025-12-10T12:55:59Z

docs/src/main/sphinx/functions/datasketches.md

+is available through {func}`theta_sketch_union` and
+{func}`theta_sketch_estimate`, typically used in `COUNT DISTINCT` queries.
+DataSketches can be created with Hive or Pig using their respective sketch
+APIs.


I'd rather see a concrete instructions user can follow to create these data sketches. IMO they should not need to go to a different system. It's really not a big lift especially that Trino already has an aggregation function for building them (IcebergThetaSketchForStats), it's just not user facing.

cc @mosabua for how to reference to other systems.

findepi · 2025-12-10T12:57:01Z

docs/src/main/sphinx/functions/datasketches.md

+
+Note: Trino does not create new sketches. Build the Theta sketches in your
+ingest pipeline or another engine (e.g., Spark or Hive) and store the
+serialized sketch bytes as a ``VARBINARY`` column. Trino functions operate on those serialized sketches.


DataSketches supports many sketch families and serialization format is to some extend configurable. This should say what's supported.

findepi · 2025-12-10T12:58:16Z

docs/src/main/sphinx/functions/datasketches.md

+
+## Functions
+
+:::{function} theta_sketch_union(sketches [, nominal_entries, seed]) -> varbinary


Why is it aggregation function? should there be a corresponding scalar function for unioning two sketches?

Also sketches -> sketch? or sketches -> varbinary? cc @mosabua

findepi · 2025-12-10T13:05:18Z

...ino-datasketches/src/main/java/io/trino/plugin/datasketches/state/SketchStateSerializer.java

+            SliceOutput sliceOutput = new DynamicSliceOutput(SIZE_OF_LONG + SIZE_OF_INT + SIZE_OF_INT + slice.length());
+            sliceOutput.appendInt(state.getNominalEntries());
+            sliceOutput.appendLong(state.getSeed());
+
+            sliceOutput.appendInt(slice.length());
+            sliceOutput.appendBytes(slice);
+
+            VARBINARY.writeSlice(out, sliceOutput.slice());


Compare with

trino/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/aggregation/DataSketchStateSerializer.java

Lines 51 to 55 in da1de5e

checkArgument(state.getUpdateSketch() == null || state.getCompactSketch() == null, "A state must not have both transient accumulator and combined form set");

CompactSketch compactSketch = Optional.ofNullable(state.getCompactSketch())

.orElseGet(() -> state.getUpdateSketch().compact());

Slice slice = Slices.wrappedBuffer(compactSketch.toByteArray());

VARBINARY.writeSlice(out, slice);

sopel39 requested review from findepi and raunaqmorarka December 5, 2025 17:07

cla-bot bot added the cla-signed label Dec 5, 2025

github-actions bot added the docs label Dec 5, 2025

sopel39 force-pushed the ks/data_sketches branch from e892109 to 09190ed Compare December 5, 2025 17:07

martint requested changes Dec 5, 2025

View reviewed changes

...ino-datasketches/src/main/java/io/trino/plugin/datasketches/theta/SketchFunctionsPlugin.java Outdated Show resolved Hide resolved

findepi reviewed Dec 8, 2025

View reviewed changes

docs/src/main/sphinx/functions/datasketches.md Outdated Show resolved Hide resolved

findepi reviewed Dec 8, 2025

View reviewed changes

plugin/trino-datasketches/src/main/java/io/trino/plugin/datasketches/theta/UnionWithParams.java Outdated Show resolved Hide resolved

findepi reviewed Dec 8, 2025

View reviewed changes

plugin/trino-datasketches/src/main/java/io/trino/plugin/datasketches/theta/UnionWithParams.java Show resolved Hide resolved

findepi reviewed Dec 8, 2025

View reviewed changes

Apache DataSketches plugin for Trino

497b043

Plugin to query Apache DataSketches (https://datasketches.apache.org/docs/Theta/ThetaSketchFramework.html). This includes merge and estimate function for theta sketch.

sopel39 force-pushed the ks/data_sketches branch from 09190ed to 497b043 Compare December 8, 2025 14:55

sopel39 requested review from findepi and martint December 8, 2025 14:56

findepi reviewed Dec 10, 2025

View reviewed changes

findepi requested a review from mosabua December 10, 2025 13:05

		state1 = factory.createSingleState();
		state2 = factory.createSingleState();

		is available through {func}`theta_sketch_union` and
		{func}`theta_sketch_estimate`, typically used in `COUNT DISTINCT` queries.


		## Functions

		:::{function} theta_sketch_union(sketches [, nominal_entries, seed]) -> varbinary

	checkArgument(state.getUpdateSketch() == null \|\| state.getCompactSketch() == null, "A state must not have both transient accumulator and combined form set");
	CompactSketch compactSketch = Optional.ofNullable(state.getCompactSketch())
	.orElseGet(() -> state.getUpdateSketch().compact());
	Slice slice = Slices.wrappedBuffer(compactSketch.toByteArray());
	VARBINARY.writeSlice(out, slice);

Apache DataSketches plugin for Trino #27563

Are you sure you want to change the base?

Apache DataSketches plugin for Trino #27563

Uh oh!

Conversation

sopel39 commented Dec 5, 2025

Description

Additional context and related issues

Release notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants