Add Fallback option `behavior` for dynamic fallback calculation by rickbrouwer · Pull Request #6464 · kedacore/keda

rickbrouwer · 2025-01-05T17:50:37Z

When you are running a large number of pods and something goes wrong, Keda uses a fallback (if configured). However, the difference between the fallback.replicas and the current number of replicas can be large.
"Something going wrong" can also be temporary, which means it can suddenly scale down considerably and then return to the correct number of replicas.

With this PR, I want to make a new option behavior for dynamic fallback calculation.

You can indicate that it should use the current number of replicas (CurrentReplicasIfHigher ) as fallback if current replicas is higher. If the given fallback.replicas is higher than the current replicas, then the fallback.replicas will be used.
Also you can indicate that it should use the current number of replicas (CurrentReplicasIfLower ) as fallback if current replicas is lower. If the given fallback.replicas is higher than the current replicas, then the current replicas will be used.

A new parameter has been added to the fallback struct (behavior) with three options Static (default), CurrentReplicasIfHigher and CurrentReplicasIfLower.
An adjustment has been made in the fallback.go to retrieve the current number of replicas
In the scale_scaledobjects.go there was code that determined what the current number of replicas was. This has been moved to the scale_resolvers.go
Fallback modified to determine whether to fall back to the fallback.replicas or the current number of replicas

If for any reason this option is not desired, please let me know.

Checklist

I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #6450
Docs: kedacore/keda-docs#1519
Relates to discussion: #6334

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

wozniakjan

lgtm, I like this a lot!

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

wozniakjan

a minor suggestion for simplification, I think semantically this should be equivalent, just less code to maintain :)

Let me know if you find any discrepancies

Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com> Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

JorTurFer · 2025-01-18T19:28:50Z

Sorry because I'm late, I've posted a comment in the issue about the approach, PTAL and share your thoughts

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

zroubalik

I think that we should also think about e2e test for this feature

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

rickbrouwer · 2025-02-07T14:43:55Z

I think that we should also think about e2e test for this feature

added an e2e test

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

wozniakjan · 2025-02-20T16:43:26Z

/run-e2e fallback
Update: You can check the progress here

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

wozniakjan · 2025-02-21T09:03:45Z

/run-e2e fallback
Update: You can check the progress here

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

wozniakjan · 2025-02-21T12:55:47Z

/run-e2e fallback
Update: You can check the progress here

JorTurFer

❤️

zroubalik

LGTM, great stuff!

…core#6464) Signed-off-by: rickbrouwer <rickbrouwer@gmail.com> Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com> Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com> Signed-off-by: mittalvaibhav1 <mittalvaibhavandroid@gmail.com>

…core#6464) Signed-off-by: rickbrouwer <rickbrouwer@gmail.com> Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com> Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com> Signed-off-by: David Pochopsky <david.pochopsky@united.com>

rickbrouwer added 3 commits January 4, 2025 16:24

Add Fallback option UseCurrentReplicasAsMinimum

1b0d21b

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Update the tests

67d8be1

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Update changelog

13c95df

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

rickbrouwer requested a review from a team as a code owner January 5, 2025 17:50

rickbrouwer marked this pull request as draft January 5, 2025 17:50

semgrep-app Bot reviewed Jan 5, 2025

View reviewed changes

Comment thread pkg/scaling/executor/scale_scaledobjects_test.go Outdated

semgrep-app Bot reviewed Jan 5, 2025

View reviewed changes

Comment thread pkg/scaling/executor/scale_scaledobjects_test.go Outdated

semgrep-app Bot reviewed Jan 5, 2025

View reviewed changes

Comment thread pkg/scaling/executor/scale_scaledobjects_test.go Outdated

Fix

7513c7f

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

rickbrouwer force-pushed the fallback branch from f6fbcf4 to 7513c7f Compare January 6, 2025 08:23

rickbrouwer added 2 commits January 6, 2025 09:41

Fix

05ccab8

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

Fix

2bc1117

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

rickbrouwer marked this pull request as ready for review January 6, 2025 09:17

rickbrouwer mentioned this pull request Jan 6, 2025

Add Fallback option behavior for dynamic fallback calculation kedacore/keda-docs#1519

Merged

1 task

wozniakjan approved these changes Jan 13, 2025

View reviewed changes

Comment thread pkg/fallback/fallback.go Outdated

Comment thread apis/keda/v1alpha1/scaledobject_types.go Outdated

rickbrouwer marked this pull request as draft January 15, 2025 16:31

rickbrouwer force-pushed the fallback branch 3 times, most recently from caa059c to 34abee5 Compare January 15, 2025 18:03

rickbrouwer marked this pull request as ready for review January 15, 2025 18:04

rickbrouwer changed the title ~~Use CurrentReplicas as a minimum for fallback replicas~~ Add Fallback option UseCurrentReplicasAsMinimum Jan 16, 2025

rickbrouwer force-pushed the fallback branch from 34abee5 to e6a0cd2 Compare January 16, 2025 12:52

Update after feedback

c0b1d55

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

rickbrouwer force-pushed the fallback branch from e6a0cd2 to c0b1d55 Compare January 16, 2025 13:22

wozniakjan reviewed Jan 16, 2025

View reviewed changes

Comment thread apis/keda/v1alpha1/scaledobject_types.go

Comment thread apis/keda/v1alpha1/scaledobject_types.go Outdated

rickbrouwer and others added 3 commits January 17, 2025 15:47

Update apis/keda/v1alpha1/scaledobject_types.go

ea4698d

Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com> Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Update apis/keda/v1alpha1/scaledobject_types.go

93e29ce

Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com> Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Update after feedback

6fbcb56

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

rickbrouwer force-pushed the fallback branch from e8c12c5 to 6fbcb56 Compare January 17, 2025 17:38

rickbrouwer force-pushed the fallback branch 2 times, most recently from fe1d3a0 to 76d35eb Compare February 6, 2025 14:43

Updates from feedback

c472516

Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com>

rickbrouwer force-pushed the fallback branch from 76d35eb to c472516 Compare February 6, 2025 14:57

zroubalik reviewed Feb 6, 2025

View reviewed changes

Add e2e test

6545ef7

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

rickbrouwer force-pushed the fallback branch from c6b0d8b to 6545ef7 Compare February 7, 2025 14:30

JorTurFer reviewed Feb 8, 2025

View reviewed changes

Comment thread apis/keda/v1alpha1/scaledobject_types.go

rickbrouwer added 2 commits February 18, 2025 13:46

Merge branch 'main' into fallback

dfe21b0

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Remove unnecessary tests

5ce7599

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

JorTurFer reviewed Feb 20, 2025

View reviewed changes

Comment thread tests/internals/fallback/fallback_test.go

Add more e2e tests

dd86f7b

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

rickbrouwer force-pushed the fallback branch from 547091b to 3780b83 Compare February 21, 2025 09:03

Add static e2e

aef8b42

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

rickbrouwer force-pushed the fallback branch from 3780b83 to aef8b42 Compare February 21, 2025 09:13

rickbrouwer added 2 commits February 21, 2025 10:25

Adjust failureThreshold

98227b2

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

Change time settings

350af40

Signed-off-by: rickbrouwer <rickbrouwer@gmail.com>

JorTurFer approved these changes Feb 21, 2025

View reviewed changes

zroubalik approved these changes Feb 22, 2025

View reviewed changes

zroubalik merged commit 4cd84f3 into kedacore:main Feb 22, 2025

rickbrouwer deleted the fallback branch February 22, 2025 11:47

rickbrouwer mentioned this pull request Mar 27, 2025

Fallback overrides HPA's internal triggers #6659

Closed

rickbrouwer mentioned this pull request Jun 2, 2025

feat: add fallback support for value metric type #6655

Merged

7 tasks

y-rabie mentioned this pull request Aug 19, 2025

fix: sigsegv accessing scaleClient in non-static fallback modes #6992

Merged

7 tasks

Conversation

rickbrouwer commented Jan 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wozniakjan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wozniakjan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JorTurFer commented Jan 18, 2025

Uh oh!

zroubalik left a comment

Choose a reason for hiding this comment

Uh oh!

rickbrouwer commented Feb 7, 2025

Uh oh!

Uh oh!

wozniakjan commented Feb 20, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

wozniakjan commented Feb 21, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wozniakjan commented Feb 21, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JorTurFer left a comment

Choose a reason for hiding this comment

Uh oh!

zroubalik left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rickbrouwer commented Jan 5, 2025 •

edited

Loading

wozniakjan commented Feb 20, 2025 •

edited by github-actions Bot

Loading

wozniakjan commented Feb 21, 2025 •

edited by github-actions Bot

Loading

wozniakjan commented Feb 21, 2025 •

edited by github-actions Bot

Loading