Skip to content

Conversation

hdikeman
Copy link
Contributor

@hdikeman hdikeman commented Sep 2, 2025

Description

The exchange compression format session property controls the encoding across exchanges, but also the encoding for results returning via the serialized result interface

There are a few reasons we may want to independently control these:

  1. result compression occurs on the coordinator, which likely has different resource constraints than worker nodes
  2. the transfer from coordinator to client may operate over a higher latency or lower bandwidth link: for this reason the expectation is that a more rigorous compression format may be preferable to that used for worker-worker or worker-coordinator exchanges

Motivation and Context

Returning results can be the bottleneck for queries which return significant amounts of data, especially if the client is strapped for bandwidth. We want to enable compressing query results without affecting the exchange compression format.

Impact

Setting exchange_compression_codec will no longer affect result compression

Setting result_compression_codec will set serialized result compression format

Test Plan

[add unit test reference, manual]

Contributor checklist

  • Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

== NO RELEASE NOTE ==

Differential Revision: D81504585

Summary:
The exchange compression format session property controls the encoding across exchanges, but also the encoding for results returning via the serialized result interface

There are a few reasons we may want to independently control these:

1. result compression occurs on the coordinator, which likely has different resource constraints than worker nodes
2. the transfer from coordinator to client may operate over a higher latency or lower bandwidth link: for this reason the expectation is that  a more rigorous compression format may be preferable to that used for worker-worker or worker-coordinator exchanges

Differential Revision: D81504585
@facebook-github-bot
Copy link
Collaborator

This pull request was exported from Phabricator. Differential Revision: D81504585

@hdikeman hdikeman marked this pull request as draft September 2, 2025 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants