-
Notifications
You must be signed in to change notification settings - Fork 940
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add metrics for batch pending max elapse time #6829
Conversation
acc99b1
to
30eb694
Compare
30eb694
to
1e93dd2
Compare
95a8de6
to
6c82970
Compare
6c82970
to
1680175
Compare
d2a490d
to
ab7b6db
Compare
val filter = MetadataFilter( | ||
sessionType = SessionType.BATCH, | ||
engineType = batchType, | ||
username = batchUser, | ||
state = batchState, | ||
requestName = batchName, | ||
createTime = createTime, | ||
createTime = createTimeFilter, | ||
endTime = endTime) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not related to this PR, but createTime
and endTime
are confusing, should be minCreateTime
and maxEndTime
, and say if they are inclusive or exclusive in the comments, with an explanation for specific values
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update: the predication name might be more intuitive if we use greatThanXXX
lessThanOrEqualsXXX
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will address it in another PR
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
Outdated
Show resolved
Hide resolved
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
Outdated
Show resolved
Hide resolved
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
Outdated
Show resolved
Hide resolved
thanks for the comments, will address it today |
e8aa576
to
d9bc335
Compare
d9bc335
to
bf8169a
Compare
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
Outdated
Show resolved
Hide resolved
### Why are the changes needed? 1. add metrics `kyuubi.operartion.batch_pending_max_elapse` for the batch pending max elapse time, which is helpful for batch health monitoring, and we can send alert if the batch pending elapse time too long 2. For `GET /api/v1/batches` api, limit the max time window for listing batches, which is helpful that, we want to reserve more metadata in kyuubi server end, for example: 90 days, but for list batches, we just want to allow user to search the last 7 days. It is optional. And if `create_time` is specified, order by `create_time` instead of `key_id`. https://github.com/apache/kyuubi/blob/68a6f48da53dd0ad2e20b450a41ca600b8c1e1d2/kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql#L32 ### How was this patch tested? GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #6829 from turboFei/batch_pending_time. Closes #6829 ee4f931 [Wang, Fei] docs bf8169a [Wang, Fei] comments f493a2a [Wang, Fei] new config ab7b6db [Wang, Fei] ut 1680175 [Wang, Fei] in memory session 510a30b [Wang, Fei] batchSearchWindow opt 1e93dd2 [Wang, Fei] save Authored-by: Wang, Fei <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 3167692) Signed-off-by: Cheng Pan <[email protected]>
Thanks, merged to master/1.10 |
Why are the changes needed?
kyuubi.operartion.batch_pending_max_elapse
for the batch pending max elapse time, which is helpful for batch health monitoring, and we can send alert if the batch pending elapse time too longGET /api/v1/batches
api, limit the max time window for listing batches, which is helpful that, we want to reserve more metadata in kyuubi server end, for example: 90 days, but for list batches, we just want to allow user to search the last 7 days. It is optional. And ifcreate_time
is specified, order bycreate_time
instead ofkey_id
.kyuubi/kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql
Line 32 in 68a6f48
How was this patch tested?
GA.
Was this patch authored or co-authored using generative AI tooling?
No.