Disable continuous profiler in GenerateDumpIfDbgRequested #6497

kevingosse · 2025-01-03T09:41:53Z

Summary of changes

Disable the continuous profiler in GenerateDumpIfDbgRequested

Reason for change

GenerateDumpIfDbgRequested has been flaky for a while now and I've been unable to find the root cause. I suspect the failure might be caused by some weird interaction with the continuous profiler, and the CP isn't actually needed for this test, so I'm disabling it to see if the failure disappears.

andrewlock

🤞

datadog-ddstaging · 2025-01-03T10:13:54Z

Datadog Report

Branch report: kevin/cp_in_createdump_test
Commit report: 160a788
Test service: dd-trace-dotnet

✅ 0 Failed, 236140 Passed, 2557 Skipped, 20h 21m 32.06s Total Time

andrewlock · 2025-01-03T10:20:39Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6497) - mean (68ms)  : 65, 71
     .   : milestone, 68,
    master - mean (69ms)  : 65, 73
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6497) - mean (975ms)  : 951, 1000
     .   : milestone, 975,
    master - mean (980ms)  : 953, 1007
     .   : milestone, 980,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6497) - mean (108ms)  : 105, 110
     .   : milestone, 108,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6497) - mean (676ms)  : 660, 691
     .   : milestone, 676,
    master - mean (678ms)  : 661, 695
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6497) - mean (91ms)  : 89, 92
     .   : milestone, 91,
    master - mean (92ms)  : 89, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (6497) - mean (635ms)  : 615, 654
     .   : milestone, 635,
    master - mean (633ms)  : 614, 653
     .   : milestone, 633,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6497) - mean (194ms)  : 189, 199
     .   : milestone, 194,
    master - mean (195ms)  : 190, 199
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6497) - mean (1,101ms)  : 1068, 1134
     .   : milestone, 1101,
    master - mean (1,099ms)  : 1071, 1126
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6497) - mean (279ms)  : 275, 284
     .   : milestone, 279,
    master - mean (278ms)  : 274, 281
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6497) - mean (871ms)  : 843, 899
     .   : milestone, 871,
    master - mean (871ms)  : 839, 903
     .   : milestone, 871,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6497) - mean (267ms)  : 261, 272
     .   : milestone, 267,
    master - mean (267ms)  : 262, 273
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6497) - mean (853ms)  : 820, 886
     .   : milestone, 853,
    master - mean (852ms)  : 817, 887
     .   : milestone, 852,

andrewlock · 2025-01-03T10:50:06Z

Benchmarks Report for tracer 🐌

Benchmarks for #6497 compared to master:

1 benchmarks are faster, with geometric mean 1.152
1 benchmarks are slower, with geometric mean 1.116
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.3μs	47.3ns	335ns	0.0193	0.00773	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	55.2ns	292ns	0.0154	0.00514	0	5.8 KB
master	`StartStopWithChild`	net472	16.6μs	65.3ns	253ns	1.04	0.296	0.0905	6.22 KB
#6497	`StartStopWithChild`	net6.0	7.93μs	43.2ns	248ns	0.0154	0.00768	0	5.61 KB
#6497	`StartStopWithChild`	netcoreapp3.1	9.95μs	53.5ns	293ns	0.0153	0.0102	0	5.81 KB
#6497	`StartStopWithChild`	net472	16.4μs	51.5ns	193ns	1.05	0.313	0.0906	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	483μs	409ns	1.58μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	688μs	233ns	873ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	857μs	484ns	1.81μs	0.431	3.3 KB
#6497	`WriteAndFlushEnrichedTraces`	net6.0	493μs	327ns	1.22μs	0	2.7 KB
#6497	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	652μs	387ns	1.5μs	0	2.7 KB
#6497	`WriteAndFlushEnrichedTraces`	net472	861μs	671ns	2.6μs	0.434	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	935ns	9.25μs	0.168	14.47 KB
master	`SendRequest`	netcoreapp3.1	186μs	1.78μs	17.7μs	0.175	17.27 KB
master	`SendRequest`	net472	0.00291ns	0.00119ns	0.00462ns	0	0 b
#6497	`SendRequest`	net6.0	151μs	975ns	9.6μs	0.158	14.47 KB
#6497	`SendRequest`	netcoreapp3.1	169μs	1.05μs	10.4μs	0.194	17.27 KB
#6497	`SendRequest`	net472	0.00565ns	0.00199ns	0.00772ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	587μs	3.32μs	21.8μs	0.291	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	672μs	3.63μs	19.5μs	0.388	0	0	41.81 KB
master	`WriteAndFlushEnrichedTraces`	net472	834μs	3.96μs	15.3μs	8.45	2.53	0.422	53.28 KB
#6497	`WriteAndFlushEnrichedTraces`	net6.0	560μs	1.99μs	7.46μs	0.563	0	0	41.52 KB
#6497	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	672μs	3.37μs	17.2μs	0.329	0	0	41.65 KB
#6497	`WriteAndFlushEnrichedTraces`	net472	873μs	4.47μs	20.5μs	8.3	2.62	0.437	53.33 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.3μs	1.36ns	5.28ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.71μs	2.06ns	7.71ns	0.0133	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.1μs	2.58ns	10ns	0.156	0.00106	987 B
#6497	`ExecuteNonQuery`	net6.0	1.32μs	0.609ns	2.28ns	0.0139	0	1.02 KB
#6497	`ExecuteNonQuery`	netcoreapp3.1	1.84μs	1.68ns	6.51ns	0.0135	0	1.02 KB
#6497	`ExecuteNonQuery`	net472	2.06μs	2.18ns	8.44ns	0.156	0.00102	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.3μs	1.81ns	6.77ns	0.0134	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	2.42ns	9.36ns	0.013	976 B
master	`CallElasticsearch`	net472	2.47μs	1.18ns	4.43ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	0.441ns	1.65ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	0.577ns	2.16ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.6μs	0.815ns	2.82ns	0.167	1.05 KB
#6497	`CallElasticsearch`	net6.0	1.32μs	0.429ns	1.6ns	0.0139	976 B
#6497	`CallElasticsearch`	netcoreapp3.1	1.55μs	0.661ns	2.47ns	0.013	976 B
#6497	`CallElasticsearch`	net472	2.45μs	1.47ns	5.5ns	0.157	995 B
#6497	`CallElasticsearchAsync`	net6.0	1.28μs	0.637ns	2.38ns	0.0133	952 B
#6497	`CallElasticsearchAsync`	netcoreapp3.1	1.66μs	1.93ns	7.23ns	0.0133	1.02 KB
#6497	`CallElasticsearchAsync`	net472	2.52μs	0.547ns	1.97ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.37μs	0.395ns	1.42ns	0.013	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	0.859ns	3.33ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.82μs	0.896ns	3.47ns	0.145	915 B
#6497	`ExecuteAsync`	net6.0	1.39μs	0.76ns	2.94ns	0.0133	952 B
#6497	`ExecuteAsync`	netcoreapp3.1	1.62μs	1.48ns	5.73ns	0.013	952 B
#6497	`ExecuteAsync`	net472	1.78μs	0.283ns	1.06ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.46μs	1.65ns	6.37ns	0.0314	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.4μs	1.84ns	7.13ns	0.0379	2.85 KB
master	`SendAsync`	net472	7.61μs	1.62ns	6.29ns	0.493	3.12 KB
#6497	`SendAsync`	net6.0	4.47μs	1.96ns	7.61ns	0.0312	2.31 KB
#6497	`SendAsync`	netcoreapp3.1	5.11μs	1.09ns	3.79ns	0.0382	2.85 KB
#6497	`SendAsync`	net472	7.33μs	2.1ns	7.85ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.48μs	0.708ns	2.65ns	0.0229	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.2μs	1.25ns	4.69ns	0.022	1.64 KB
master	`EnrichedLog`	net472	2.65μs	2.56ns	9.93ns	0.249	1.57 KB
#6497	`EnrichedLog`	net6.0	1.44μs	0.807ns	3.02ns	0.0231	1.64 KB
#6497	`EnrichedLog`	netcoreapp3.1	2.17μs	1.72ns	6.19ns	0.0226	1.64 KB
#6497	`EnrichedLog`	net472	2.66μs	0.825ns	2.97ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	135ns	525ns	0.059	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	157ns	607ns	0	0	4.28 KB
master	`EnrichedLog`	net472	150μs	114ns	426ns	0.677	0.226	4.46 KB
#6497	`EnrichedLog`	net6.0	118μs	308ns	1.19μs	0	0	4.28 KB
#6497	`EnrichedLog`	netcoreapp3.1	122μs	242ns	936ns	0	0	4.28 KB
#6497	`EnrichedLog`	net472	150μs	205ns	793ns	0.676	0.225	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.91μs	0.908ns	3.52ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.36μs	2.3ns	8.61ns	0.0284	2.2 KB
master	`EnrichedLog`	net472	4.86μs	1.99ns	7.69ns	0.321	2.02 KB
#6497	`EnrichedLog`	net6.0	3.04μs	0.492ns	1.84ns	0.0303	2.2 KB
#6497	`EnrichedLog`	netcoreapp3.1	4.44μs	1.89ns	7.31ns	0.0291	2.2 KB
#6497	`EnrichedLog`	net472	4.89μs	1.41ns	5.46ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.66ns	2.56ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.73μs	0.74ns	2.86ns	0.0156	1.14 KB
master	`SendReceive`	net472	2.09μs	0.989ns	3.83ns	0.184	1.16 KB
#6497	`SendReceive`	net6.0	1.41μs	0.788ns	2.84ns	0.0162	1.14 KB
#6497	`SendReceive`	netcoreapp3.1	1.72μs	0.811ns	3.14ns	0.0155	1.14 KB
#6497	`SendReceive`	net472	2.19μs	1.59ns	6.14ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.67μs	0.721ns	2.79ns	0.0227	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.84μs	0.924ns	3.2ns	0.0211	1.65 KB
master	`EnrichedLog`	net472	4.39μs	1.53ns	5.71ns	0.323	2.04 KB
#6497	`EnrichedLog`	net6.0	2.67μs	1.06ns	4.11ns	0.0226	1.6 KB
#6497	`EnrichedLog`	netcoreapp3.1	3.79μs	1.4ns	5.05ns	0.0208	1.65 KB
#6497	`EnrichedLog`	net472	4.29μs	2.68ns	10.4ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6497

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.116	398.49	444.84

Faster 🎉 in #6497

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.152	961.33	834.69

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	399ns	0.229ns	0.887ns	0.00804	576 B
master	`StartFinishSpan`	netcoreapp3.1	610ns	0.455ns	1.7ns	0.00768	576 B
master	`StartFinishSpan`	net472	742ns	0.301ns	1.17ns	0.0915	578 B
master	`StartFinishScope`	net6.0	481ns	0.279ns	1.08ns	0.00991	696 B
master	`StartFinishScope`	netcoreapp3.1	808ns	0.536ns	2.08ns	0.00929	696 B
master	`StartFinishScope`	net472	961ns	0.532ns	2.06ns	0.104	658 B
#6497	`StartFinishSpan`	net6.0	446ns	0.916ns	3.55ns	0.00815	576 B
#6497	`StartFinishSpan`	netcoreapp3.1	603ns	0.374ns	1.4ns	0.00783	576 B
#6497	`StartFinishSpan`	net472	722ns	0.408ns	1.58ns	0.0916	578 B
#6497	`StartFinishScope`	net6.0	517ns	0.241ns	0.869ns	0.00988	696 B
#6497	`StartFinishScope`	netcoreapp3.1	757ns	0.548ns	2.12ns	0.00944	696 B
#6497	`StartFinishScope`	net472	834ns	0.653ns	2.53ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	588ns	0.358ns	1.38ns	0.00986	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	858ns	0.299ns	1.12ns	0.00947	696 B
master	`RunOnMethodBegin`	net472	1.14μs	0.509ns	1.97ns	0.104	658 B
#6497	`RunOnMethodBegin`	net6.0	587ns	0.3ns	1.16ns	0.00986	696 B
#6497	`RunOnMethodBegin`	netcoreapp3.1	874ns	0.747ns	2.89ns	0.00962	696 B
#6497	`RunOnMethodBegin`	net472	1.12μs	0.436ns	1.69ns	0.104	658 B

Disable CP in GenerateDumpIfDbgRequested

160a788

kevingosse requested a review from a team as a code owner January 3, 2025 09:41

github-actions bot added the area:profiler Issues related to the continous-profiler label Jan 3, 2025

andrewlock approved these changes Jan 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable continuous profiler in GenerateDumpIfDbgRequested #6497

Disable continuous profiler in GenerateDumpIfDbgRequested #6497

kevingosse commented Jan 3, 2025

andrewlock left a comment

datadog-ddstaging bot commented Jan 3, 2025 •

edited

Loading

andrewlock commented Jan 3, 2025

andrewlock commented Jan 3, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6497

Faster 🎉 in #6497

Raw results

Raw results

Disable continuous profiler in GenerateDumpIfDbgRequested #6497

Are you sure you want to change the base?

Disable continuous profiler in GenerateDumpIfDbgRequested #6497

Conversation

kevingosse commented Jan 3, 2025

Summary of changes

Reason for change

andrewlock left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Jan 3, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 3, 2025

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 3, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6497

Faster 🎉 in #6497

Raw results

Raw results

datadog-ddstaging bot commented Jan 3, 2025 •

edited

Loading