Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC: Support S2 compression #9322

Merged
merged 4 commits into from
Sep 25, 2024
Merged

gRPC: Support S2 compression #9322

merged 4 commits into from
Sep 25, 2024

Conversation

aknuds1
Copy link
Contributor

@aknuds1 aknuds1 commented Sep 18, 2024

What this PR does

Support S2 (a Snappy extension) gRPC compression, via custom gRPC compressor support in dskit.

Based on grafana/dskit#583.

I've done a test deployment internally, and everything seems to work.

See favourable test results vs Snappy.

Which issue(s) this PR fixes or relates to

Supersedes #9292. Fixes #8522.

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@aknuds1 aknuds1 added enhancement New feature or request type/performance labels Sep 18, 2024
@aknuds1 aknuds1 mentioned this pull request Sep 18, 2024
4 tasks
@aknuds1 aknuds1 force-pushed the arve/grpc-s2 branch 4 times, most recently from 833b840 to d72a6ac Compare September 19, 2024 08:03
@aknuds1 aknuds1 marked this pull request as ready for review September 19, 2024 08:06
Copy link
Contributor

@tacole02 tacole02 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@aknuds1 aknuds1 force-pushed the arve/grpc-s2 branch 3 times, most recently from 99502b7 to bc0ea78 Compare September 24, 2024 09:12
@aknuds1
Copy link
Contributor Author

aknuds1 commented Sep 24, 2024

Dropping the Snappy compatible mode, since I don't see a use for it in practice. AFAICT for it to be at all useful, it has to replace the Snappy compressor (same name). When using the "s2-snappy" compressor name, unpatched Mimir instances don't recognize it and fail to accept the connection.

Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@@ -76,6 +77,7 @@ func (cfg *Config) RegisterFlags(f *flag.FlagSet, logger log.Logger) {
f.StringVar(&cfg.Addr, "query-frontend.instance-addr", "", "IP address to advertise to the querier (via scheduler) (default is auto-detected from network interfaces).")
f.IntVar(&cfg.Port, "query-frontend.instance-port", 0, "Port to advertise to querier (via scheduler) (defaults to server.grpc-listen-port).")

cfg.GRPCClientConfig.CustomCompressors = []string{s2.Name}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uh, so we need to pass this to all clients? I somehow thought it would be enough to register it in dskit "globally".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can find a better solution after this PR?

pkg/util/grpcencoding/s2/s2.go Show resolved Hide resolved
Support S2 and S2 in Snappy compatible mode gRPC compression.

Signed-off-by: Arve Knudsen <[email protected]>
Signed-off-by: Arve Knudsen <[email protected]>
@deniszh
Copy link
Contributor

deniszh commented Sep 25, 2024 via email

@aknuds1
Copy link
Contributor Author

aknuds1 commented Sep 25, 2024

Thanks, added your tests/benchmarks.

@aknuds1 aknuds1 enabled auto-merge (squash) September 25, 2024 15:54
Signed-off-by: Arve Knudsen <[email protected]>
@aknuds1 aknuds1 enabled auto-merge (squash) September 25, 2024 16:14
@aknuds1 aknuds1 merged commit b7fd6e1 into main Sep 25, 2024
29 checks passed
@aknuds1 aknuds1 deleted the arve/grpc-s2 branch September 25, 2024 16:27
@narqo narqo mentioned this pull request Sep 26, 2024
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

zstd compression support between distributors and ingesters
5 participants