Adding flags for ignore-retry-cause and default-max-attempts #596

hamersaw · 2023-07-26T15:17:29Z

TL;DR

Adding flags to simplify retry behavior. ignore-retry-cause means that system and user failures are treated the same and default-max-attempts sets a default number of retries for tasks.

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

^^^

Tracking Issue

NA

Follow-up issue

NA

Signed-off-by: Daniel Rammer <[email protected]>

Signed-off-by: Dennis Keck <[email protected]>

Signed-off-by: Daniel Rammer <[email protected]>

hamersaw · 2023-09-14T15:51:48Z

In the current implementation enabling ignoreRetryCause also turns the interruptibleFailureThreshold configuration into a complementary value. IMO this will result in unintended behavior and should be handled separately. Unit tests will need to be completed / updated accordingly.

Signed-off-by: Daniel Rammer <[email protected]>

codecov · 2023-09-14T22:47:09Z

Codecov Report

❗ No coverage uploaded for pull request base (master@0fcc1da). Click here to learn what that means.
The diff coverage is 91.42%.

❗ Current head 1af296d differs from pull request most recent head 6af5f8f. Consider uploading reports for the commit 6af5f8f to get more accurate results

Additional details and impacted files

pkg/controller/nodes/node_exec_context.go

pkg/controller/nodes/executor.go

* Cleanup retry behavior Signed-off-by: Dennis Keck <[email protected]> * Fix interruptible retry threshold for odl behavior Signed-off-by: Dennis Keck <[email protected]> * Add tests for BuildNodeExecutionContext Signed-off-by: Dennis Keck <[email protected]> * Fix IsElgibileForRetries Tests Signed-off-by: Dennis Keck <[email protected]> --------- Signed-off-by: Dennis Keck <[email protected]>

Signed-off-by: Daniel Rammer <[email protected]>

pingsutw · 2023-09-29T23:42:55Z

pkg/controller/nodes/node_exec_context.go

+		// interruptible machine
+		maxAttempts := uint32(config.GetConfig().NodeConfig.DefaultMaxAttempts)
+		if n.GetRetryStrategy() != nil && n.GetRetryStrategy().MinAttempts != nil && *n.GetRetryStrategy().MinAttempts != 1 {
+			maxAttempts = uint32(*n.GetRetryStrategy().MinAttempts)


Should we deprecate MinAttempts and add MaxAttempts 😂

eapolinario · 2023-10-03T02:13:54Z

Hi, we are moving all Flyte development to a monorepo. In order to help the transition period, we're moving open PRs to monorepo automatically and your PR was moved to flyteorg/flyte#4153. Notice that if there are any conflicts in the resulting PR they most likely happen due to the change in the import path of the flyte components.

added flags for ignore-retry-cause and default-max-attempts

b201c6b

Signed-off-by: Daniel Rammer <[email protected]>

fellhorn mentioned this pull request Aug 23, 2023

Feature: Switch to non interrupible when using the simplified retry behavior #610

Merged

8 tasks

fg91 mentioned this pull request Sep 10, 2023

Document simplified retry behaviour introduced in #3902 flyteorg/flyte#4022

Merged

3 tasks

Merge branch 'master' into feature/simplify-retries

060ff3a

Signed-off-by: Daniel Rammer <[email protected]>

hamersaw marked this pull request as ready for review September 14, 2023 14:34

hamersaw requested review from kumare3 and EngHabu as code owners September 14, 2023 14:34

fellhorn and others added 3 commits September 14, 2023 09:35

Switch non interruptible @ unified retry behavior (#610)

56eb391

Signed-off-by: Dennis Keck <[email protected]>

fixed lint issues

6c91faa

Signed-off-by: Daniel Rammer <[email protected]>

began unit tests

210975c

Signed-off-by: Daniel Rammer <[email protected]>

hamersaw mentioned this pull request Sep 14, 2023

updating interruptibleFailureThreshold to int32 to allow negative values flyteorg/flyteplugins#403

Merged

8 tasks

hamersaw added 2 commits September 14, 2023 17:12

enabling negative interruptibleFailureThreshold

b754c3f

Signed-off-by: Daniel Rammer <[email protected]>

fixed unit tests

6a73b29

Signed-off-by: Daniel Rammer <[email protected]>

fg91 reviewed Sep 15, 2023

View reviewed changes

pkg/controller/nodes/node_exec_context.go Outdated Show resolved Hide resolved

fg91 reviewed Sep 15, 2023

View reviewed changes

pkg/controller/nodes/executor.go Outdated Show resolved Hide resolved

fellhorn and others added 6 commits September 29, 2023 09:39

Merge branch 'master' into feature/simplify-retries

397120b

Signed-off-by: Daniel Rammer <[email protected]>

updated flyteplugins

d0b7b27

Signed-off-by: Daniel Rammer <[email protected]>

Merge branch 'master' into feature/simplify-retries

b728529

Signed-off-by: Daniel Rammer <[email protected]>

fixed lint issue

f919ad6

Signed-off-by: Daniel Rammer <[email protected]>

Merge branch 'master' into feature/simplify-retries

6af5f8f

Signed-off-by: Daniel Rammer <[email protected]>

pingsutw reviewed Sep 29, 2023

View reviewed changes

pingsutw approved these changes Sep 29, 2023

View reviewed changes

eapolinario closed this Oct 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding flags for ignore-retry-cause and default-max-attempts #596

Adding flags for ignore-retry-cause and default-max-attempts #596

hamersaw commented Jul 26, 2023 •

edited

Loading

hamersaw commented Sep 14, 2023 •

edited

Loading

codecov bot commented Sep 14, 2023 •

edited

Loading

pingsutw Sep 29, 2023

eapolinario commented Oct 3, 2023

Adding flags for ignore-retry-cause and default-max-attempts #596

Adding flags for ignore-retry-cause and default-max-attempts #596

Conversation

hamersaw commented Jul 26, 2023 • edited Loading

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

hamersaw commented Sep 14, 2023 • edited Loading

codecov bot commented Sep 14, 2023 • edited Loading

Codecov Report

pingsutw Sep 29, 2023

Choose a reason for hiding this comment

eapolinario commented Oct 3, 2023

hamersaw commented Jul 26, 2023 •

edited

Loading

hamersaw commented Sep 14, 2023 •

edited

Loading

codecov bot commented Sep 14, 2023 •

edited

Loading