scheduler: logical plan fragmenter #7018
Conversation
rubywtl
force-pushed
the
scheduler/logicalplan-fragmenter
branch
from
8b4f789 to
02fd607
Compare
yeya24
left a comment
yeya24
left a comment
There was a problem hiding this comment.
Thanks for the work. There is some room for improvement in the test but it looks good in general.
|
|
||
| if len(fragments) > 0 { | ||
| return fragments, nil | ||
| } else { |
There was a problem hiding this comment.
Nit. You can remove this else as we already return above
| require.Equal(t, tc.expectedFragments, len(res)) | ||
|
|
||
| // check the metadata of the fragments of binary expressions | ||
| if len(res) == 3 { |
There was a problem hiding this comment.
Why do we need this check? This only works for the second test case?
There was a problem hiding this comment.
I added checks for the 7 fragment case (when we have 4 binary expressions) and a new test case for 5-fragment case (when we have 3 binary expressions) to ensure that the order of child fragment IDs and number of child fragments are correct.
| query: "sum(rate(http_requests_total{job=\"api\"}[5m])) + sum(rate(http_requests_total{job=\"web\"}[5m])) + sum(rate(http_requests_total{job=\"cache\"}[5m])) + sum(rate(http_requests_total{job=\"db\"}[5m]))", | ||
| start: now, | ||
| end: now, | ||
| expectedFragments: 7, |
There was a problem hiding this comment.
The test should try to compare the output fragments instead of number of fragments to be safer
rubywtl
force-pushed
the
scheduler/logicalplan-fragmenter
branch
from
02fd607 to
ef17bb3
Compare
Signed-off-by: rubywtl <[email protected]>
rubywtl
force-pushed
the
scheduler/logicalplan-fragmenter
branch
from
ef17bb3 to
2e4789c
Compare
yeya24
left a comment
yeya24
left a comment
There was a problem hiding this comment.
Thanks. This PR looks good to me.
Before I merge, I would like to get more reviews from other people who might be interested in this topic. Maybe @harry671003 @justinjung04 @SungJin1212?
|
LGTM, it would be nicer if add some tracing code later. |
3 participants
What this PR does:
This PR implements a logical plan fragmenter for distributed query execution. The fragmenter identifies fragments based on remote node markers in the logical plan and creates independent execution units that can be distributed across queriers. It maintains execution dependencies between parent and child fragments
Which issue(s) this PR fixes:
related to distributed query execution proposal
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]