-
Notifications
You must be signed in to change notification settings - Fork 288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement flaky test retries for Test NG #6347
Merged
nikita-tkachenko-datadog
merged 2 commits into
master
from
nikita-tkachenko/retry-flaky-tests-testng
Jan 3, 2024
Merged
Implement flaky test retries for Test NG #6347
nikita-tkachenko-datadog
merged 2 commits into
master
from
nikita-tkachenko/retry-flaky-tests-testng
Jan 3, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikita-tkachenko-datadog
added
the
comp: ci visibility
Continuous Integration Visibility
label
Dec 11, 2023
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 8 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049366
Total [baseline] (9.324 s) : 0, 9323702
Agent [candidate] (1.053 s) : 0, 1053041
Total [candidate] (9.366 s) : 0, 9365725
section appsec
Agent [baseline] (1.153 s) : 0, 1152708
Total [baseline] (9.437 s) : 0, 9436933
Agent [candidate] (1.151 s) : 0, 1150530
Total [candidate] (9.421 s) : 0, 9421110
section iast
Agent [baseline] (1.172 s) : 0, 1172116
Total [baseline] (9.619 s) : 0, 9618880
Agent [candidate] (1.176 s) : 0, 1175662
Total [candidate] (9.639 s) : 0, 9638871
section profiling
Agent [baseline] (1.254 s) : 0, 1253625
Total [baseline] (9.615 s) : 0, 9614555
Agent [candidate] (1.249 s) : 0, 1248589
Total [candidate] (9.601 s) : 0, 9601452
gantt
title petclinic - break down per module: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (650.014 ms) : 0, 650014
BytebuddyAgent [candidate] (651.335 ms) : 0, 651335
GlobalTracer [baseline] (306.72 ms) : 0, 306720
GlobalTracer [candidate] (308.806 ms) : 0, 308806
AppSec [baseline] (50.456 ms) : 0, 50456
AppSec [candidate] (50.826 ms) : 0, 50826
Remote Config [baseline] (670.046 µs) : 0, 670
Remote Config [candidate] (682.297 µs) : 0, 682
Telemetry [baseline] (7.184 ms) : 0, 7184
Telemetry [candidate] (7.172 ms) : 0, 7172
section appsec
BytebuddyAgent [baseline] (653.615 ms) : 0, 653615
BytebuddyAgent [candidate] (651.529 ms) : 0, 651529
GlobalTracer [baseline] (308.265 ms) : 0, 308265
GlobalTracer [candidate] (307.973 ms) : 0, 307973
AppSec [baseline] (148.782 ms) : 0, 148782
AppSec [candidate] (149.116 ms) : 0, 149116
Remote Config [baseline] (645.52 µs) : 0, 646
Remote Config [candidate] (640.754 µs) : 0, 641
Telemetry [baseline] (6.963 ms) : 0, 6963
Telemetry [candidate] (6.944 ms) : 0, 6944
section iast
BytebuddyAgent [baseline] (772.242 ms) : 0, 772242
BytebuddyAgent [candidate] (772.807 ms) : 0, 772807
GlobalTracer [baseline] (285.985 ms) : 0, 285985
GlobalTracer [candidate] (288.155 ms) : 0, 288155
AppSec [baseline] (52.601 ms) : 0, 52601
AppSec [candidate] (50.947 ms) : 0, 50947
Remote Config [baseline] (566.209 µs) : 0, 566
Remote Config [candidate] (570.264 µs) : 0, 570
Telemetry [baseline] (6.45 ms) : 0, 6450
Telemetry [candidate] (7.301 ms) : 0, 7301
IAST [baseline] (19.856 ms) : 0, 19856
IAST [candidate] (21.535 ms) : 0, 21535
section profiling
BytebuddyAgent [baseline] (664.736 ms) : 0, 664736
BytebuddyAgent [candidate] (662.138 ms) : 0, 662138
GlobalTracer [baseline] (380.307 ms) : 0, 380307
GlobalTracer [candidate] (378.445 ms) : 0, 378445
AppSec [baseline] (51.077 ms) : 0, 51077
AppSec [candidate] (50.97 ms) : 0, 50970
Remote Config [baseline] (981.023 µs) : 0, 981
Remote Config [candidate] (974.56 µs) : 0, 975
Telemetry [baseline] (7.189 ms) : 0, 7189
Telemetry [candidate] (7.234 ms) : 0, 7234
ProfilingAgent [baseline] (94.474 ms) : 0, 94474
ProfilingAgent [candidate] (94.348 ms) : 0, 94348
Profiling [baseline] (94.5 ms) : 0, 94500
Profiling [candidate] (94.374 ms) : 0, 94374
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1058622
Total [baseline] (8.816 s) : 0, 8815615
Agent [candidate] (1.065 s) : 0, 1065004
Total [candidate] (8.776 s) : 0, 8776210
section iast
Agent [baseline] (1.177 s) : 0, 1177276
Total [baseline] (9.256 s) : 0, 9256003
Agent [candidate] (1.174 s) : 0, 1173549
Total [candidate] (9.257 s) : 0, 9256547
section iast_TELEMETRY_OFF
Agent [baseline] (1.164 s) : 0, 1163634
Total [baseline] (9.286 s) : 0, 9286310
Agent [candidate] (1.161 s) : 0, 1161479
Total [candidate] (9.266 s) : 0, 9265749
gantt
title insecure-bank - break down per module: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (655.653 ms) : 0, 655653
BytebuddyAgent [candidate] (659.633 ms) : 0, 659633
GlobalTracer [baseline] (309.487 ms) : 0, 309487
GlobalTracer [candidate] (311.314 ms) : 0, 311314
AppSec [baseline] (50.956 ms) : 0, 50956
AppSec [candidate] (51.186 ms) : 0, 51186
Remote Config [baseline] (685.995 µs) : 0, 686
Remote Config [candidate] (703.082 µs) : 0, 703
Telemetry [baseline] (7.265 ms) : 0, 7265
Telemetry [candidate] (7.419 ms) : 0, 7419
section iast
BytebuddyAgent [baseline] (775.78 ms) : 0, 775780
BytebuddyAgent [candidate] (772.745 ms) : 0, 772745
GlobalTracer [baseline] (287.482 ms) : 0, 287482
GlobalTracer [candidate] (286.16 ms) : 0, 286160
AppSec [baseline] (53.339 ms) : 0, 53339
AppSec [candidate] (54.037 ms) : 0, 54037
IAST [baseline] (19.046 ms) : 0, 19046
IAST [candidate] (19.125 ms) : 0, 19125
Remote Config [baseline] (553.518 µs) : 0, 554
Remote Config [candidate] (574.396 µs) : 0, 574
Telemetry [baseline] (6.437 ms) : 0, 6437
Telemetry [candidate] (6.478 ms) : 0, 6478
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (764.611 ms) : 0, 764611
BytebuddyAgent [candidate] (763.72 ms) : 0, 763720
GlobalTracer [baseline] (286.124 ms) : 0, 286124
GlobalTracer [candidate] (285.98 ms) : 0, 285980
AppSec [baseline] (49.435 ms) : 0, 49435
AppSec [candidate] (49.25 ms) : 0, 49250
IAST [baseline] (21.951 ms) : 0, 21951
IAST [candidate] (21.156 ms) : 0, 21156
Remote Config [baseline] (658.218 µs) : 0, 658
Remote Config [candidate] (655.523 µs) : 0, 656
Telemetry [baseline] (6.482 ms) : 0, 6482
Telemetry [candidate] (6.473 ms) : 0, 6473
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
dateFormat X
axisFormat %s
section baseline
no_agent (1.362 ms) : 1343, 1380
. : milestone, 1362,
appsec (1.798 ms) : 1773, 1822
. : milestone, 1798,
iast (1.543 ms) : 1519, 1568
. : milestone, 1543,
profiling (1.57 ms) : 1544, 1597
. : milestone, 1570,
tracing (1.519 ms) : 1494, 1544
. : milestone, 1519,
section candidate
no_agent (1.357 ms) : 1337, 1377
. : milestone, 1357,
appsec (1.773 ms) : 1749, 1798
. : milestone, 1773,
iast (1.543 ms) : 1519, 1567
. : milestone, 1543,
profiling (1.584 ms) : 1558, 1611
. : milestone, 1584,
tracing (1.483 ms) : 1458, 1507
. : milestone, 1483,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
dateFormat X
axisFormat %s
section baseline
no_agent (363.639 µs) : 344, 384
. : milestone, 364,
iast (484.72 µs) : 464, 505
. : milestone, 485,
iast_FULL (542.327 µs) : 522, 563
. : milestone, 542,
iast_INACTIVE (454.307 µs) : 433, 476
. : milestone, 454,
iast_TELEMETRY_OFF (476.137 µs) : 455, 497
. : milestone, 476,
tracing (450.291 µs) : 429, 472
. : milestone, 450,
section candidate
no_agent (373.011 µs) : 352, 394
. : milestone, 373,
iast (477.667 µs) : 457, 498
. : milestone, 478,
iast_FULL (536.774 µs) : 517, 557
. : milestone, 537,
iast_INACTIVE (449.856 µs) : 428, 471
. : milestone, 450,
iast_TELEMETRY_OFF (470.309 µs) : 450, 491
. : milestone, 470,
tracing (449.645 µs) : 429, 471
. : milestone, 450,
|
Base automatically changed from
nikita-tkachenko/retry-flaky-tests-cucumber-junit-5
to
master
December 16, 2023 18:41
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/retry-flaky-tests-testng
branch
from
December 16, 2023 18:43
9e79857
to
37474d4
Compare
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/retry-flaky-tests-testng
branch
from
December 25, 2023 13:26
4f8a36a
to
912a0c0
Compare
juan-fernandez
approved these changes
Jan 3, 2024
nikita-tkachenko-datadog
deleted the
nikita-tkachenko/retry-flaky-tests-testng
branch
January 3, 2024 11:42
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Implements retrying flaky TestNG tests.
Motivation
Adding flaky test retries feature to CI Visibility.
Jira ticket: CIVIS-8243