Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement flaky test retries for Test NG #6347

Merged
merged 2 commits into from
Jan 3, 2024
Merged

Implement flaky test retries for Test NG #6347

merged 2 commits into from
Jan 3, 2024

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Dec 11, 2023 •
edited by jira bot
Loading

What Does This Do

Implements retrying flaky TestNG tests.

Motivation

Adding flaky test retries feature to CI Visibility.

Jira ticket: CIVIS-8243

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 11, 2023
@pr-commenter
Copy link

pr-commenter bot commented Dec 11, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/retry-flaky-tests-testng
git_commit_date 1703292532 1703510751
git_commit_sha dee42ed 912a0c0
release_version 1.27.0-SNAPSHOT~dee42ed64a 1.27.0-SNAPSHOT~912a0c096e
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1703513418 1703513418
ci_job_id 396456520 396456520
ci_pipeline_id 25595856 25595856
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 8 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049366
Total [baseline] (9.324 s) : 0, 9323702
Agent [candidate] (1.053 s) : 0, 1053041
Total [candidate] (9.366 s) : 0, 9365725
section appsec
Agent [baseline] (1.153 s) : 0, 1152708
Total [baseline] (9.437 s) : 0, 9436933
Agent [candidate] (1.151 s) : 0, 1150530
Total [candidate] (9.421 s) : 0, 9421110
section iast
Agent [baseline] (1.172 s) : 0, 1172116
Total [baseline] (9.619 s) : 0, 9618880
Agent [candidate] (1.176 s) : 0, 1175662
Total [candidate] (9.639 s) : 0, 9638871
section profiling
Agent [baseline] (1.254 s) : 0, 1253625
Total [baseline] (9.615 s) : 0, 9614555
Agent [candidate] (1.249 s) : 0, 1248589
Total [candidate] (9.601 s) : 0, 9601452
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.049 s -
Agent appsec 1.153 s 103.342 ms (9.8%)
Agent iast 1.172 s 122.75 ms (11.7%)
Agent profiling 1.254 s 204.259 ms (19.5%)
Total tracing 9.324 s -
Total appsec 9.437 s 113.231 ms (1.2%)
Total iast 9.619 s 295.178 ms (3.2%)
Total profiling 9.615 s 290.852 ms (3.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.053 s -
Agent appsec 1.151 s 97.489 ms (9.3%)
Agent iast 1.176 s 122.621 ms (11.6%)
Agent profiling 1.249 s 195.548 ms (18.6%)
Total tracing 9.366 s -
Total appsec 9.421 s 55.385 ms (0.6%)
Total iast 9.639 s 273.146 ms (2.9%)
Total profiling 9.601 s 235.727 ms (2.5%) 
gantt
    title petclinic - break down per module: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (650.014 ms) : 0, 650014
BytebuddyAgent [candidate] (651.335 ms) : 0, 651335
GlobalTracer [baseline] (306.72 ms) : 0, 306720
GlobalTracer [candidate] (308.806 ms) : 0, 308806
AppSec [baseline] (50.456 ms) : 0, 50456
AppSec [candidate] (50.826 ms) : 0, 50826
Remote Config [baseline] (670.046 µs) : 0, 670
Remote Config [candidate] (682.297 µs) : 0, 682
Telemetry [baseline] (7.184 ms) : 0, 7184
Telemetry [candidate] (7.172 ms) : 0, 7172
section appsec
BytebuddyAgent [baseline] (653.615 ms) : 0, 653615
BytebuddyAgent [candidate] (651.529 ms) : 0, 651529
GlobalTracer [baseline] (308.265 ms) : 0, 308265
GlobalTracer [candidate] (307.973 ms) : 0, 307973
AppSec [baseline] (148.782 ms) : 0, 148782
AppSec [candidate] (149.116 ms) : 0, 149116
Remote Config [baseline] (645.52 µs) : 0, 646
Remote Config [candidate] (640.754 µs) : 0, 641
Telemetry [baseline] (6.963 ms) : 0, 6963
Telemetry [candidate] (6.944 ms) : 0, 6944
section iast
BytebuddyAgent [baseline] (772.242 ms) : 0, 772242
BytebuddyAgent [candidate] (772.807 ms) : 0, 772807
GlobalTracer [baseline] (285.985 ms) : 0, 285985
GlobalTracer [candidate] (288.155 ms) : 0, 288155
AppSec [baseline] (52.601 ms) : 0, 52601
AppSec [candidate] (50.947 ms) : 0, 50947
Remote Config [baseline] (566.209 µs) : 0, 566
Remote Config [candidate] (570.264 µs) : 0, 570
Telemetry [baseline] (6.45 ms) : 0, 6450
Telemetry [candidate] (7.301 ms) : 0, 7301
IAST [baseline] (19.856 ms) : 0, 19856
IAST [candidate] (21.535 ms) : 0, 21535
section profiling
BytebuddyAgent [baseline] (664.736 ms) : 0, 664736
BytebuddyAgent [candidate] (662.138 ms) : 0, 662138
GlobalTracer [baseline] (380.307 ms) : 0, 380307
GlobalTracer [candidate] (378.445 ms) : 0, 378445
AppSec [baseline] (51.077 ms) : 0, 51077
AppSec [candidate] (50.97 ms) : 0, 50970
Remote Config [baseline] (981.023 µs) : 0, 981
Remote Config [candidate] (974.56 µs) : 0, 975
Telemetry [baseline] (7.189 ms) : 0, 7189
Telemetry [candidate] (7.234 ms) : 0, 7234
ProfilingAgent [baseline] (94.474 ms) : 0, 94474
ProfilingAgent [candidate] (94.348 ms) : 0, 94348
Profiling [baseline] (94.5 ms) : 0, 94500
Profiling [candidate] (94.374 ms) : 0, 94374
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1058622
Total [baseline] (8.816 s) : 0, 8815615
Agent [candidate] (1.065 s) : 0, 1065004
Total [candidate] (8.776 s) : 0, 8776210
section iast
Agent [baseline] (1.177 s) : 0, 1177276
Total [baseline] (9.256 s) : 0, 9256003
Agent [candidate] (1.174 s) : 0, 1173549
Total [candidate] (9.257 s) : 0, 9256547
section iast_TELEMETRY_OFF
Agent [baseline] (1.164 s) : 0, 1163634
Total [baseline] (9.286 s) : 0, 9286310
Agent [candidate] (1.161 s) : 0, 1161479
Total [candidate] (9.266 s) : 0, 9265749
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.059 s -
Agent iast 1.177 s 118.654 ms (11.2%)
Agent iast_TELEMETRY_OFF 1.164 s 105.013 ms (9.9%)
Total tracing 8.816 s -
Total iast 9.256 s 440.388 ms (5.0%)
Total iast_TELEMETRY_OFF 9.286 s 470.695 ms (5.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.065 s -
Agent iast 1.174 s 108.545 ms (10.2%)
Agent iast_TELEMETRY_OFF 1.161 s 96.475 ms (9.1%)
Total tracing 8.776 s -
Total iast 9.257 s 480.337 ms (5.5%)
Total iast_TELEMETRY_OFF 9.266 s 489.539 ms (5.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (655.653 ms) : 0, 655653
BytebuddyAgent [candidate] (659.633 ms) : 0, 659633
GlobalTracer [baseline] (309.487 ms) : 0, 309487
GlobalTracer [candidate] (311.314 ms) : 0, 311314
AppSec [baseline] (50.956 ms) : 0, 50956
AppSec [candidate] (51.186 ms) : 0, 51186
Remote Config [baseline] (685.995 µs) : 0, 686
Remote Config [candidate] (703.082 µs) : 0, 703
Telemetry [baseline] (7.265 ms) : 0, 7265
Telemetry [candidate] (7.419 ms) : 0, 7419
section iast
BytebuddyAgent [baseline] (775.78 ms) : 0, 775780
BytebuddyAgent [candidate] (772.745 ms) : 0, 772745
GlobalTracer [baseline] (287.482 ms) : 0, 287482
GlobalTracer [candidate] (286.16 ms) : 0, 286160
AppSec [baseline] (53.339 ms) : 0, 53339
AppSec [candidate] (54.037 ms) : 0, 54037
IAST [baseline] (19.046 ms) : 0, 19046
IAST [candidate] (19.125 ms) : 0, 19125
Remote Config [baseline] (553.518 µs) : 0, 554
Remote Config [candidate] (574.396 µs) : 0, 574
Telemetry [baseline] (6.437 ms) : 0, 6437
Telemetry [candidate] (6.478 ms) : 0, 6478
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (764.611 ms) : 0, 764611
BytebuddyAgent [candidate] (763.72 ms) : 0, 763720
GlobalTracer [baseline] (286.124 ms) : 0, 286124
GlobalTracer [candidate] (285.98 ms) : 0, 285980
AppSec [baseline] (49.435 ms) : 0, 49435
AppSec [candidate] (49.25 ms) : 0, 49250
IAST [baseline] (21.951 ms) : 0, 21951
IAST [candidate] (21.156 ms) : 0, 21156
Remote Config [baseline] (658.218 µs) : 0, 658
Remote Config [candidate] (655.523 µs) : 0, 656
Telemetry [baseline] (6.482 ms) : 0, 6482
Telemetry [candidate] (6.473 ms) : 0, 6473
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2023-12-25T13:49:25 2023-12-25T14:06:03
git_branch master nikita-tkachenko/retry-flaky-tests-testng
git_commit_date 1703292532 1703510751
git_commit_sha dee42ed 912a0c0
release_version 1.27.0-SNAPSHOT~dee42ed64a 1.27.0-SNAPSHOT~912a0c096e
start_time 2023-12-25T13:49:12 2023-12-25T14:05:50
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1703513418 1703513418
ci_job_id 396456520 396456520
ci_pipeline_id 25595856 25595856
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.362 ms) : 1343, 1380
.   : milestone, 1362,
appsec (1.798 ms) : 1773, 1822
.   : milestone, 1798,
iast (1.543 ms) : 1519, 1568
.   : milestone, 1543,
profiling (1.57 ms) : 1544, 1597
.   : milestone, 1570,
tracing (1.519 ms) : 1494, 1544
.   : milestone, 1519,
section candidate
no_agent (1.357 ms) : 1337, 1377
.   : milestone, 1357,
appsec (1.773 ms) : 1749, 1798
.   : milestone, 1773,
iast (1.543 ms) : 1519, 1567
.   : milestone, 1543,
profiling (1.584 ms) : 1558, 1611
.   : milestone, 1584,
tracing (1.483 ms) : 1458, 1507
.   : milestone, 1483,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.362 ms [1.343 ms, 1.38 ms] -
appsec 1.798 ms [1.773 ms, 1.822 ms] 435.93 µs (32.0%)
iast 1.543 ms [1.519 ms, 1.568 ms] 181.916 µs (13.4%)
profiling 1.57 ms [1.544 ms, 1.597 ms] 208.86 µs (15.3%)
tracing 1.519 ms [1.494 ms, 1.544 ms] 157.461 µs (11.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.357 ms [1.337 ms, 1.377 ms] -
appsec 1.773 ms [1.749 ms, 1.798 ms] 416.407 µs (30.7%)
iast 1.543 ms [1.519 ms, 1.567 ms] 185.699 µs (13.7%)
profiling 1.584 ms [1.558 ms, 1.611 ms] 227.421 µs (16.8%)
tracing 1.483 ms [1.458 ms, 1.507 ms] 125.67 µs (9.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.27.0-SNAPSHOT~912a0c096e, baseline=1.27.0-SNAPSHOT~dee42ed64a
    dateFormat X
    axisFormat %s
section baseline
no_agent (363.639 µs) : 344, 384
.   : milestone, 364,
iast (484.72 µs) : 464, 505
.   : milestone, 485,
iast_FULL (542.327 µs) : 522, 563
.   : milestone, 542,
iast_INACTIVE (454.307 µs) : 433, 476
.   : milestone, 454,
iast_TELEMETRY_OFF (476.137 µs) : 455, 497
.   : milestone, 476,
tracing (450.291 µs) : 429, 472
.   : milestone, 450,
section candidate
no_agent (373.011 µs) : 352, 394
.   : milestone, 373,
iast (477.667 µs) : 457, 498
.   : milestone, 478,
iast_FULL (536.774 µs) : 517, 557
.   : milestone, 537,
iast_INACTIVE (449.856 µs) : 428, 471
.   : milestone, 450,
iast_TELEMETRY_OFF (470.309 µs) : 450, 491
.   : milestone, 470,
tracing (449.645 µs) : 429, 471
.   : milestone, 450,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 363.639 µs [343.508 µs, 383.77 µs] -
iast 484.72 µs [464.294 µs, 505.146 µs] 121.082 µs (33.3%)
iast_FULL 542.327 µs [521.887 µs, 562.766 µs] 178.688 µs (49.1%)
iast_INACTIVE 454.307 µs [433.005 µs, 475.61 µs] 90.669 µs (24.9%)
iast_TELEMETRY_OFF 476.137 µs [455.253 µs, 497.022 µs] 112.499 µs (30.9%)
tracing 450.291 µs [428.754 µs, 471.829 µs] 86.653 µs (23.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 373.011 µs [352.236 µs, 393.787 µs] -
iast 477.667 µs [457.136 µs, 498.197 µs] 104.655 µs (28.1%)
iast_FULL 536.774 µs [516.544 µs, 557.004 µs] 163.763 µs (43.9%)
iast_INACTIVE 449.856 µs [428.497 µs, 471.216 µs] 76.845 µs (20.6%)
iast_TELEMETRY_OFF 470.309 µs [449.759 µs, 490.858 µs] 97.298 µs (26.1%)
tracing 449.645 µs [428.543 µs, 470.748 µs] 76.634 µs (20.5%)

Base automatically changed from nikita-tkachenko/retry-flaky-tests-cucumber-junit-5 to master December 16, 2023 18:41
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/retry-flaky-tests-testng branch from 9e79857 to 37474d4 Compare December 16, 2023 18:43
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/retry-flaky-tests-testng branch from 4f8a36a to 912a0c0 Compare December 25, 2023 13:26
@DataDog DataDog deleted a comment from github-actions bot Dec 25, 2023
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review December 25, 2023 14:37
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit bbb960b into master Jan 3, 2024
71 of 73 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/retry-flaky-tests-testng branch January 3, 2024 11:42
@github-actions github-actions bot added this to the 1.28.0 milestone Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juan-fernandez juan-fernandez juan-fernandez approved these changes

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility
Projects
None yet
Milestone
1.28.0
Development

Successfully merging this pull request may close these issues.
2 participants
@nikita-tkachenko-datadog @juan-fernandez