[feat][storage] Store Traces in ClickHouse Based on Jaeger V2 #6725
Conversation
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
from
a65794d to
bacbf97
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6725 +/- ##
==========================================
- Coverage 96.03% 95.71% -0.32%
==========================================
Files 364 369 +5
Lines 20675 21002 +327
==========================================
+ Hits 19855 20102 +247
- Misses 626 700 +74
- Partials 194 200 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
from
bacbf97 to
21c3fab
Compare
zzzk1
changed the title
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
5 times, most recently
from
a16370b to
72d91cc
Compare
|
@yurishkuro ClickHouse integration test not working in CI. Is there anything I might have missed? |
| @@ -0,0 +1,58 @@ | |||
| // Copyright (c) 2025 The Jaeger Authors. | |||
There was a problem hiding this comment.
All of these methods require a reliable connection to the ClickHouse server; they should not be unit tested.
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
from
72d91cc to
51b3f8f
Compare
Signed-off-by: zzzk1 <[email protected]>
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
from
51b3f8f to
b8915ef
Compare
Signed-off-by: zzzk1 <[email protected]>
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
from
572c1a2 to
7718d65
Compare
zzzk1
changed the title
Signed-off-by: zzzk1 <[email protected]>
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
from
7718d65 to
19bc811
Compare
Signed-off-by: zzzk1 <[email protected]>
zzzk1
force-pushed
the
write-path-for-clickhouse
branch
from
19bc811 to
c408a25
Compare
Signed-off-by: zzzk1 <[email protected]>
|
I would like to implement all basic features, ensure that the integration tests pass, and then return to address other low-priority tasks. |
| workflow_call: | ||
|
|
||
| concurrency: | ||
| group: cit-kafka-${{ github.workflow }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }} |
There was a problem hiding this comment.
| group: cit-kafka-${{ github.workflow }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }} | |
| group: cit-clickhouse-${{ github.workflow }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }} |
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| jaeger-version: [v2] |
There was a problem hiding this comment.
we don't plan to support CH in v1, so this doesn't need to be part of the matrix
| run: bash scripts/e2e/clickhouse.sh | ||
|
|
||
| - uses: ./.github/actions/verify-metrics-snapshot | ||
| if: matrix.jaeger-version == 'v2' |
| ports: | ||
| - 9000:9000 | ||
| volumes: | ||
| - ./init.sql:/docker-entrypoint-initdb.d/init.sql |
There was a problem hiding this comment.
we are trying to move away from having external steps for setting up database - Jaeger should be able to run when started against a blank database. That means the schema creation logic should be embedded in the implementation (similar to #5922).
| } | ||
|
|
||
| func ignoreStartAutoCloseIdleConnections() goleak.Option { | ||
| return goleak.IgnoreTopFunction("github.com/ClickHouse/clickhouse-go/v2.(*clickhouse).startAutoCloseIdleConnections") |
There was a problem hiding this comment.
adding exclusions here should be the last resort. Are there known upstream bugs that prevent clean shutdown of these goroutines?
| if c.Pool != nil { | ||
| c.Pool.Close() | ||
| } | ||
| if c.Conn != nil { |
| // Copyright (c) 2025 The Jaeger Authors. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package wrapper |
There was a problem hiding this comment.
pkg/clickhouse/wrapper/wrapper.go
please keep everything CH-related under internal/storage/v2/clickhouse
| client clickhouse.Client | ||
| } | ||
|
|
||
| func NewFactoryWithConfig(configuration *config.Configuration, logger *zap.Logger) (*Factory, error) { |
There was a problem hiding this comment.
| func NewFactoryWithConfig(configuration *config.Configuration, logger *zap.Logger) (*Factory, error) { | |
| func NewFactory(cfg *config.Configuration, logger *zap.Logger) (*Factory, error) { |
| } | ||
| } | ||
|
|
||
| func (c *Configuration) NewClient(logger *zap.Logger) (clickhouse.Client, error) { |
There was a problem hiding this comment.
this should be moved to a separate package (perhaps close to wrapper) so that config logic can be 100% testable and not dependent on runtime connections to db.
| testutils.VerifyGoLeaksOnce(t) | ||
| }) | ||
| s := &ClickhouseIntegrationTestSuite{} | ||
| s.initialize(t) |
There was a problem hiding this comment.
this doesn't actually run the test?
|
@yurishkuro The key here is that the design of the schema:
#auto run DDL script
auto: true
client:
database: jaeger
username: default
password: default
#ch-go
writer:
address: "127.0.0.1:9200"
pool:
max_connection_lifetime: 3600000000000
max_connection_idle_time: 1800000000000
#CPU Core number
min_connections: 4
#CPU Core number * 2
max_connections: 8
health_check_period: 60000000000
#clickhouse-go
reader:
#no cluster just a different field here.
addresses: ["node00:9200","node01:9200","node02:9200"]The directory structure has been adjusted as follows:
|
Which problem is this PR solving?
Desgin doc: Jaeger V2: Support for ClickHouse as Storage Backend
Part of #5058
Description of the changes
How was this change tested?
Checklist
jaeger:make lint testjaeger-ui:npm run lintandnpm run test