schedule: impl balance range scheduler #9005

bufferflies · 2025-01-17T01:47:03Z

What problem does this PR solve?

Issue Number: Close #9006

What is changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Code changes

Side effects

Related changes

Release note

None.

Signed-off-by: 童剑 <[email protected]>

ti-chi-bot · 2025-01-17T01:47:05Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: 童剑 <[email protected]>

bufferflies · 2025-02-17T08:12:31Z

/test pull-integration-realcluster-test

ti-chi-bot · 2025-02-17T08:12:33Z

@bufferflies: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build

/test pull-integration-realcluster-test

The following commands are available to trigger optional jobs:

/debug pull-unit-test

/test pull-integration-copr-test

Use /test all to run the following jobs that were automatically triggered:

tikv/pd/ghpr_build

tikv/pd/pull_integration_realcluster_test

In response to this:

/test pull-integration-realcluster-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: 童剑 <[email protected]>

pkg/schedule/schedulers/balance_range.go

rleungx · 2025-02-17T10:10:52Z

pkg/schedule/schedulers/balance_range.go

+	}
+
+	opInfluence := s.OpController.GetOpInfluence(cluster.GetBasicCluster(), operator.WithRangeOption(job.Ranges))
+	plan, err := s.prepare(cluster, opInfluence, job)


For every time we schedule, it will scan the range from the beginning?

yes, the info on the regions and stores in the given ranges may be updated.

Will it affect the other requests like get region or heartbeat if the range is large?

yes, the operator of scanning regions acquires Rlock which shares with the region heartbeat. We could reuse the previous distribution by interval duration, not every scheduler.

pkg/schedule/schedulers/balance_range.go

pkg/schedule/operator/operator_controller.go

pkg/schedule/schedulers/balance_range.go

Signed-off-by: 童剑 <[email protected]>

okJiang · 2025-02-20T06:46:24Z

pkg/schedule/schedulers/balance_range.go

+	balanceRangeCounter.Inc()
+	job := s.conf.peek()
+	if job == nil {
+		balanceRangeNoJobCounter.Inc()


Seems we need not to count this. It is enough that set it true when there is no job.

It will count if all the jobs are finished.

How about using RunningJobCounter rather than NoJobCounter?

if conf.peek() is nil, it indicates that all the jobs in the configuration are finished, so the metrics just tell DBA that there is no unable job in this scheduler.

okJiang · 2025-02-20T07:06:59Z

pkg/schedule/schedulers/balance_range.go

+	now := time.Now()
+	job.Start = &now
+	job.Status = running
+	if err := conf.save(); err != nil {


It is not recommended that

modify the input param

call conf.save() to persist the modified input struct

I prefer more explicit implement. For example, we can use jobs map[JobID]balanceRangeSchedulerJob and input JobID here rather than Job.

@rleungx the functions must ensure that jobs list must contains the given job.

pkg/schedule/schedulers/balance_range.go

bufferflies · 2025-02-21T01:13:10Z

pkg/schedule/schedulers/balance_range.go

+
+	targetInfluence := p.opInfluence.GetStoreInfluence(p.targetStoreID())
+	targetInf := p.job.Role.getStoreInfluence(targetInfluence)
+	if targetInf < 0 {


yes, if the store has some remove-peer operators

I must consider the scenario involving removing-peer , so the targeting is negative, the target score is small. This causes too many peers to be added to this store.

okJiang · 2025-02-20T07:32:01Z

pkg/schedule/schedulers/balance_range.go

+	}
+	targetScore := p.targetScore + targetInf + p.tolerate
+
+	shouldBalance := sourceScore >= targetScore


Can you add some comments to explan the principle? Why is the score calculated this way? What is the score difference from the balanceRegionScheduler? How are conflicts between different balance schedulers handled?

The balanceRegionScheduler and balanceLeaderScheduler only consider the global regions not the given key ranges, so the store status can calculate their score. But the balance-range-scheduler is different, the given key ranges should set it

rleungx · 2025-03-04T09:21:48Z

pkg/schedule/schedulers/balance_range.go

+func (conf *balanceRangeSchedulerConfig) begin(index int) *balanceRangeSchedulerJob {
+	conf.Lock()
+	defer conf.Unlock()
+	job := conf.jobs[index]


Seems we have already had the job, why do we need to get it again?

rleungx · 2025-03-04T09:23:03Z

pkg/schedule/schedulers/balance_range.go

+func (conf *balanceRangeSchedulerConfig) finish(index int) *balanceRangeSchedulerJob {
+	conf.Lock()
+	defer conf.Unlock()
+	job := conf.jobs[index]


pkg/schedule/schedulers/balance_range.go

rleungx · 2025-03-04T09:44:56Z

pkg/schedule/schedulers/balance_range.go

+	switch r {
+	case leader:
+		return influence.LeaderCount
+	case follower:


Will it conflict with balance region?

yes，the next MR will solve it

pkg/schedule/schedulers/balance_range.go

Signed-off-by: 童剑 <[email protected]>

rleungx

Overall LGTM

rleungx · 2025-03-05T08:16:11Z

pkg/schedule/schedulers/balance_range.go

@@ -463,7 +463,7 @@ func (s *balanceRangeScheduler) prepare(cluster sche.SchedulerCluster, opInfluen
 		averageScore:     averageScore,
 		job:              job,
 		opInfluence:      opInfluence,
-		tolerate:         tolerate,
+		tolerate:         tolerantSizeRatio,


tolerantSizeRatio

ti-chi-bot · 2025-03-05T08:18:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: okJiang, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rleungx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-03-05T08:18:09Z

[LGTM Timeline notifier]

Timeline:

2025-02-26 09:39:20.184004847 +0000 UTC m=+435108.137163113: ☑️ agreed by okJiang.
2025-03-05 08:18:08.724748243 +0000 UTC m=+429601.853667984: ☑️ agreed by rleungx.

bufferflies added 8 commits January 7, 2025 18:24

add scheduler config

e693b3e

Signed-off-by: 童剑 <[email protected]>

add new scheduler for key range

23ff7d0

Signed-off-by: 童剑 <[email protected]>

pass ut

1e6d628

Signed-off-by: 童剑 <[email protected]>

lint

d1da5b5

Signed-off-by: 童剑 <[email protected]>

pass ut

d0cfc2d

Signed-off-by: 童剑 <[email protected]>

draft

fb723a0

Signed-off-by: 童剑 <[email protected]>

rename balance-key-range to balance-range

d86148f

Signed-off-by: 童剑 <[email protected]>

use hex encode

8bdb7bc

Signed-off-by: 童剑 <[email protected]>

Merge branch 'balance_key_range' into balance_key_range_scheduler

cb2bd58

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 17, 2025

bufferflies added 10 commits January 21, 2025 15:05

rename

0696ba6

Signed-off-by: 童剑 <[email protected]>

resolve conflict

46bd44d

Signed-off-by: 童剑 <[email protected]>

impl

66f70c2

Signed-off-by: 童剑 <[email protected]>

add table configuration

5d5ee0f

Signed-off-by: 童剑 <[email protected]>

Merge branch 'balance_key_range' into balance_key_range_scheduler

5aaa79f

add test for getPeets

6a440ba

Signed-off-by: 童剑 <[email protected]>

Merge branch 'master' into balance_key_range_scheduler

97c4da2

add test

1e1d934

Signed-off-by: 童剑 <[email protected]>

sync origin

fdd2279

Signed-off-by: 童剑 <[email protected]>

add test

d0f2cae

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the balance_key_range_scheduler branch 2 times, most recently from 50d905f to 7f0ed67 Compare February 13, 2025 08:33

add more test

2854acd

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the balance_key_range_scheduler branch from 120a2bf to 2854acd Compare February 17, 2025 08:54

rleungx reviewed Feb 17, 2025

View reviewed changes

pkg/schedule/operator/operator_controller.go Outdated Show resolved Hide resolved

pkg/schedule/schedulers/balance_range.go Outdated Show resolved Hide resolved

pkg/schedule/schedulers/balance_range.go Show resolved Hide resolved

bufferflies force-pushed the balance_key_range_scheduler branch from 1628d72 to 40f0f56 Compare February 18, 2025 09:44

address comment

246aaa6

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the balance_key_range_scheduler branch from 40f0f56 to 246aaa6 Compare February 19, 2025 02:20

bufferflies requested a review from nolouch February 19, 2025 02:49

okJiang reviewed Feb 20, 2025

View reviewed changes

bufferflies force-pushed the balance_key_range_scheduler branch 3 times, most recently from ccaa591 to 2e61a74 Compare February 26, 2025 07:26

okJiang approved these changes Feb 26, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Feb 26, 2025

bufferflies force-pushed the balance_key_range_scheduler branch from 2e61a74 to 132d854 Compare March 4, 2025 08:21

rleungx reviewed Mar 4, 2025

View reviewed changes

address comment

5795f44

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the balance_key_range_scheduler branch from 132d854 to 5795f44 Compare March 5, 2025 06:52

fmt pak

1f3e261

Signed-off-by: 童剑 <[email protected]>

rleungx approved these changes Mar 5, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 5, 2025

ti-chi-bot bot added the approved label Mar 5, 2025

ti-chi-bot bot added 2 commits March 5, 2025 08:18

Merge branch 'master' into balance_key_range_scheduler

d55aca4

Merge branch 'master' into balance_key_range_scheduler

9fd7839

ti-chi-bot bot merged commit bdd857e into tikv:master Mar 5, 2025
23 of 25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schedule: impl balance range scheduler #9005

schedule: impl balance range scheduler #9005

bufferflies commented Jan 17, 2025 •

edited

Loading

ti-chi-bot bot commented Jan 17, 2025

bufferflies commented Feb 17, 2025

ti-chi-bot bot commented Feb 17, 2025

rleungx Feb 17, 2025

bufferflies Feb 18, 2025

rleungx Feb 18, 2025

bufferflies Feb 19, 2025 •

edited

Loading

okJiang Feb 20, 2025

bufferflies Feb 21, 2025

okJiang Feb 25, 2025

bufferflies Feb 26, 2025

okJiang Feb 20, 2025

bufferflies Mar 5, 2025

This comment was marked as resolved.

bufferflies Feb 21, 2025

This comment was marked as resolved.

bufferflies Feb 26, 2025

okJiang Feb 20, 2025

bufferflies Feb 21, 2025

rleungx Mar 4, 2025

rleungx Mar 4, 2025

rleungx Mar 4, 2025

bufferflies Mar 5, 2025

rleungx left a comment

rleungx Mar 5, 2025

ti-chi-bot bot commented Mar 5, 2025

ti-chi-bot bot commented Mar 5, 2025

schedule: impl balance range scheduler #9005

schedule: impl balance range scheduler #9005

Conversation

bufferflies commented Jan 17, 2025 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Jan 17, 2025

bufferflies commented Feb 17, 2025

ti-chi-bot bot commented Feb 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bufferflies Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rleungx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Mar 5, 2025

ti-chi-bot bot commented Mar 5, 2025

[LGTM Timeline notifier]

bufferflies commented Jan 17, 2025 •

edited

Loading

bufferflies Feb 19, 2025 •

edited

Loading