Build the test sample projects in a separate CI stage #6498

andrewlock · 2025-01-03T14:25:08Z

Summary of changes

Simplify the sample build, and move the building of the samples to a separate stage.

Reason for change

Currently, the samples are built at the start of each integration stage (i.e. for each OS/arch/framework). This typically takes ~10+ minutes, depending on the details. If a test fails due to flake, the whole job must be restarted, which means you must pay that cost again.

Additionally, building the samples at this point puts them in the critical path for the whole CI run. By moving them to a separate stage, which can start immediately, we can have the samples built ready by the time the integration test stage starts

Finally, this is an incremental step in attempting to overhaul the Nuke build by reducing the number of different targets, and the complexity of the build in general.

Implementation details

One of the key factors is creating a dedicated (autogenerated) "samples" solution, that you can just run dotnet build on. This should be a lot faster than iterating through all the projects to find them, but it also just simplifies the build stage in general. It's built rom the "main" Datadog.Trace.sln, and can easily be updated by running .\tracer\build.ps1 RegenerateSolutions. I have some other ideas around this approach

Note that I looked at trying to use Microsoft.Build.Traversal but I just couldn't get it to work on our solution, and this seemed easier.

In the pipeline, I chose to build all the samples for all TFMs (in parallel). This mostly for simplicity, as some stages test other TFMs to the "default ones" (e.g. arm64 tests .NET 5 on PRs), but we can look to improve that later. The sample build runs on Windows, as I suspected that to give the fewest issues longer term. I also added a dedicate macos sample build to make sure we don't commit a sample that only builds on Windows! 😄

The biggest problem was CI Visibility. There some weirdness that happens with the Microsoft.TestPlatform.TestHost package. Namely, that on Windows, the package copies an arch-specific binary into the output folder. Unfortunately, this causes the tests to fail when it runs on arm64 (i.e. a different arch). I spent a long time trying to work around this, and in the end, just disabled the default copying for Windows. It looks like the CI Vis tests don't currently run on Windows (⚠️ @tonyredondo) so this "works" but will need to be addressed somehow later if we want to run them on Windows (we should).

Finally, I removed a lot of the old Nuke targets around building samples. This isn't complete yet, but is a step in the right direction I think.

Test coverage

I've done a lot of runs of the CI, including checking it still works for ASM and debugger etc. This is one of the latest runs.

I have also tried to keep the local dev experience the same for now, i.e. BuildAndRunWindowsIntegrationTests etc still work the same as they did before.

Other details

There's a lot of additional work and caveats about this work:

The fully built samples are big: 20GB. We probably could/should dig into that at some point...
Currently the default samples (i.e. not the multi-version samples) are only built not published. That means they don't contain all their dependencies, and rely on the packages folder.
- We don't currently restore that folder between stages.
- That doesn't matter currently, because we're doing a "full solution" restore in the build stage, and are restoring it there
- In the (near) future I'd like to stop doing the global restore (and simplify the "normal projects" build using a similar generated sln). At that point we'll need to look into handling the samples differently (either restoring the packages folder, or doing a publish instead of build)
The samples don't depend on Datadog.Trace which is why we can take this approach. However, some do depend on Datadog.Trace.Manual. Consequently, that project is rebuilt in the samples stage. It's not ideal, but not a big deal IMO
We don't build all the samples that we currently reference in the project. Some of this is for simplicity (e.g. aspnet samples) and some is because we simply don't test some of the samples (ExpenseItDemo).
Some stages are still doing their "own" building of samples. This is fine for now, but it would be nice to remove that rebuilding as we further simplify the Nuke build in the future.
We can now relatively easy start doing the multi-version testing on Windows if we want, as the samples are already built and available.
We need to start testing ci Visibility on Windows 😬
In general, you'll need to run RegenerateSolutions when you add a new sample. We could/should add a github action that flags this for you, but can do that in a follow up PR

https://datadoghq.atlassian.net/browse/AIT-6624
https://datadoghq.atlassian.net/browse/LANGPLAT-5

…er" solution file. This makes it easier to build the samples without complex globbing etc in a single shot.

… platforms, and built standalone Note that _currently_ this will require calling the dedicated CompileSamples target until I can tidy it up further

Ensure we build to the dedicated artifacts directory

We now have the same location for each platform (which makes it easier to reason about in general)

…d on Windows

…t broken

andrewlock · 2025-01-03T15:07:30Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (69ms)  : 65, 73
     .   : milestone, 69,
    master - mean (69ms)  : 65, 73
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (1,029ms)  : 907, 1151
     .   : milestone, 1029,
    master - mean (980ms)  : 953, 1007
     .   : milestone, 980,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (107ms)  : 104, 110
     .   : milestone, 107,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (671ms)  : 656, 687
     .   : milestone, 671,
    master - mean (678ms)  : 661, 695
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (92ms)  : 89, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (632ms)  : 619, 645
     .   : milestone, 632,
    master - mean (633ms)  : 614, 653
     .   : milestone, 633,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (194ms)  : 189, 198
     .   : milestone, 194,
    master - mean (195ms)  : 190, 199
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (1,099ms)  : 1069, 1129
     .   : milestone, 1099,
    master - mean (1,099ms)  : 1071, 1126
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (278ms)  : 273, 283
     .   : milestone, 278,
    master - mean (278ms)  : 274, 281
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (866ms)  : 841, 891
     .   : milestone, 866,
    master - mean (871ms)  : 839, 903
     .   : milestone, 871,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (267ms)  : 263, 271
     .   : milestone, 267,
    master - mean (267ms)  : 262, 273
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (847ms)  : 812, 883
     .   : milestone, 847,
    master - mean (852ms)  : 817, 887
     .   : milestone, 852,

datadog-ddstaging · 2025-01-03T15:19:06Z

Datadog Report

Branch report: andrew/ci/build-samples-separately
Commit report: 14a2cf9
Test service: dd-trace-dotnet

✅ 0 Failed, 226531 Passed, 2079 Skipped, 19h 8m 27.44s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

EarlyFlakeDetection - Datadog.Trace.ClrProfiler.IntegrationTests.CI.PipesXUnitEvpTests - Last Failure
Expand for error
```
 
 Actual:   112 Failure
```

andrewlock · 2025-01-03T15:23:09Z

Benchmarks Report for appsec 🐌

Benchmarks for #6498 compared to master:

2 benchmarks are slower, with geometric mean 1.179
2 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6498

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0	1.210	185,000.56	223,814.50
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472	1.150	248,602.75	285,863.12

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	185μs	97.4ns	365ns	2.69	0	188.9 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	284μs	106ns	396ns	2.69	0	196.23 KB
master	`AllCycleSimpleBody`	net472	249μs	347ns	1.35μs	35.9	1.99	225.95 KB
master	`AllCycleMoreComplexBody`	net6.0	190μs	93.5ns	350ns	2.66	0	192.41 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	290μs	146ns	545ns	2.6	0	199.65 KB
master	`AllCycleMoreComplexBody`	net472	255μs	174ns	627ns	36.4	2.04	229.46 KB
master	`ObjectExtractorSimpleBody`	net6.0	138ns	0.0524ns	0.196ns	0.00396	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	190ns	0.153ns	0.573ns	0.00377	0	272 B
master	`ObjectExtractorSimpleBody`	net472	163ns	0.155ns	0.599ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.91μs	1.81ns	6.77ns	0.0535	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.88μs	1.72ns	6.19ns	0.0506	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.61μs	3.68ns	13.8ns	0.602	0.00539	3.8 KB
#6498	`AllCycleSimpleBody`	net6.0	224μs	70.4ns	273ns	2.63	0	188.76 KB
#6498	`AllCycleSimpleBody`	netcoreapp3.1	300μs	109ns	408ns	2.7	0	196.09 KB
#6498	`AllCycleSimpleBody`	net472	286μs	136ns	489ns	35.8	1.99	225.78 KB
#6498	`AllCycleMoreComplexBody`	net6.0	210μs	118ns	459ns	2.72	0	192.26 KB
#6498	`AllCycleMoreComplexBody`	netcoreapp3.1	320μs	120ns	449ns	2.72	0	199.5 KB
#6498	`AllCycleMoreComplexBody`	net472	279μs	201ns	726ns	36.4	2.08	229.3 KB
#6498	`ObjectExtractorSimpleBody`	net6.0	138ns	0.15ns	0.56ns	0.00396	0	280 B
#6498	`ObjectExtractorSimpleBody`	netcoreapp3.1	188ns	0.0938ns	0.351ns	0.00373	0	272 B
#6498	`ObjectExtractorSimpleBody`	net472	162ns	0.136ns	0.527ns	0.0446	0	281 B
#6498	`ObjectExtractorMoreComplexBody`	net6.0	2.89μs	1.54ns	5.75ns	0.0535	0	3.78 KB
#6498	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.76μs	3.6ns	13.9ns	0.0489	0	3.69 KB
#6498	`ObjectExtractorMoreComplexBody`	net472	3.61μs	2.9ns	10.9ns	0.603	0.0054	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	36.7μs	17.5ns	67.8ns	0.457	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	53.9μs	22.6ns	87.5ns	0.428	0	32.4 KB
master	`EncodeArgs`	net472	66.1μs	51.1ns	198ns	5.16	0.0666	32.5 KB
master	`EncodeLegacyArgs`	net6.0	76μs	57.4ns	222ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	107μs	312ns	1.21μs	0	0	2.14 KB
master	`EncodeLegacyArgs`	net472	155μs	79ns	306ns	0.312	0	2.15 KB
#6498	`EncodeArgs`	net6.0	36.9μs	20.2ns	78.2ns	0.442	0	32.4 KB
#6498	`EncodeArgs`	netcoreapp3.1	53.8μs	23.8ns	92ns	0.432	0	32.4 KB
#6498	`EncodeArgs`	net472	66.8μs	49.9ns	193ns	5.15	0.0668	32.5 KB
#6498	`EncodeLegacyArgs`	net6.0	74.5μs	60.3ns	209ns	0	0	2.14 KB
#6498	`EncodeLegacyArgs`	netcoreapp3.1	107μs	201ns	777ns	0	0	2.15 KB
#6498	`EncodeLegacyArgs`	net472	156μs	70.5ns	264ns	0.309	0	2.15 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	174μs	71.6ns	277ns	0	2.44 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	187μs	154ns	596ns	0	2.39 KB
master	`RunWafRealisticBenchmark`	net472	200μs	112ns	433ns	0.3	2.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	116μs	210ns	814ns	0	1.47 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	125μs	135ns	524ns	0	1.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	133μs	50.8ns	197ns	0.2	1.49 KB
#6498	`RunWafRealisticBenchmark`	net6.0	174μs	275ns	1.06μs	0	2.44 KB
#6498	`RunWafRealisticBenchmark`	netcoreapp3.1	191μs	392ns	1.52μs	0	2.39 KB
#6498	`RunWafRealisticBenchmark`	net472	200μs	182ns	706ns	0.298	2.46 KB
#6498	`RunWafRealisticBenchmarkWithAttack`	net6.0	117μs	56.8ns	220ns	0	1.47 KB
#6498	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	124μs	57.7ns	223ns	0	1.46 KB
#6498	`RunWafRealisticBenchmarkWithAttack`	net472	133μs	55.9ns	209ns	0.2	1.49 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6498

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	256.22 KB	252.33 KB	-3.9 KB	-1.52%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	59.12 KB	55.75 KB	-3.37 KB	-5.70%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	59.9μs	762ns	7.59μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	62.8μs	780ns	7.72μs	42.64 KB
master	`StringConcatBenchmark`	net472	36.9μs	64.6ns	233ns	59.12 KB
master	`StringConcatAspectBenchmark`	net6.0	332μs	1.81μs	9.9μs	255.4 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	346μs	1.72μs	11μs	256.22 KB
master	`StringConcatAspectBenchmark`	net472	275μs	5.59μs	52.7μs	278.53 KB
#6498	`StringConcatBenchmark`	net6.0	62.1μs	853ns	8.49μs	43.44 KB
#6498	`StringConcatBenchmark`	netcoreapp3.1	54.8μs	275ns	1.2μs	42.64 KB
#6498	`StringConcatBenchmark`	net472	37.3μs	71.1ns	256ns	55.75 KB
#6498	`StringConcatAspectBenchmark`	net6.0	316μs	1.76μs	12.7μs	255.49 KB
#6498	`StringConcatAspectBenchmark`	netcoreapp3.1	333μs	1.94μs	17.2μs	252.33 KB
#6498	`StringConcatAspectBenchmark`	net472	286μs	6.56μs	63μs	278.53 KB

andrewlock · 2025-01-03T15:49:21Z

Benchmarks Report for tracer 🐌

Benchmarks for #6498 compared to master:

3 benchmarks are faster, with geometric mean 1.129
2 benchmarks are slower, with geometric mean 1.143
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.3μs	47.3ns	335ns	0.0193	0.00773	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	55.2ns	292ns	0.0154	0.00514	0	5.8 KB
master	`StartStopWithChild`	net472	16.6μs	65.3ns	253ns	1.04	0.296	0.0905	6.22 KB
#6498	`StartStopWithChild`	net6.0	7.95μs	44.6ns	309ns	0.012	0.00401	0	5.61 KB
#6498	`StartStopWithChild`	netcoreapp3.1	10.3μs	56.5ns	339ns	0.0239	0.00956	0	5.8 KB
#6498	`StartStopWithChild`	net472	16.4μs	41.9ns	162ns	1.04	0.31	0.0898	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	483μs	409ns	1.58μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	688μs	233ns	873ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	857μs	484ns	1.81μs	0.431	3.3 KB
#6498	`WriteAndFlushEnrichedTraces`	net6.0	491μs	325ns	1.26μs	0	2.7 KB
#6498	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	652μs	334ns	1.25μs	0	2.7 KB
#6498	`WriteAndFlushEnrichedTraces`	net472	850μs	436ns	1.57μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	935ns	9.25μs	0.168	14.47 KB
master	`SendRequest`	netcoreapp3.1	186μs	1.78μs	17.7μs	0.175	17.27 KB
master	`SendRequest`	net472	0.00291ns	0.00119ns	0.00462ns	0	0 b
#6498	`SendRequest`	net6.0	158μs	1.01μs	9.99μs	0.142	14.47 KB
#6498	`SendRequest`	netcoreapp3.1	177μs	1.02μs	8.29μs	0.164	17.27 KB
#6498	`SendRequest`	net472	0.000655ns	0.000321ns	0.0012ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	587μs	3.32μs	21.8μs	0.291	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	672μs	3.63μs	19.5μs	0.388	0	0	41.81 KB
master	`WriteAndFlushEnrichedTraces`	net472	834μs	3.96μs	15.3μs	8.45	2.53	0.422	53.28 KB
#6498	`WriteAndFlushEnrichedTraces`	net6.0	566μs	2.94μs	16.6μs	0.548	0	0	41.56 KB
#6498	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	697μs	3.89μs	25.8μs	0.343	0	0	41.72 KB
#6498	`WriteAndFlushEnrichedTraces`	net472	833μs	3.38μs	13.1μs	8.08	2.55	0.425	53.29 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.3μs	1.36ns	5.28ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.71μs	2.06ns	7.71ns	0.0133	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.1μs	2.58ns	10ns	0.156	0.00106	987 B
#6498	`ExecuteNonQuery`	net6.0	1.38μs	1.65ns	6.38ns	0.0144	0	1.02 KB
#6498	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	1.2ns	4.33ns	0.0136	0	1.02 KB
#6498	`ExecuteNonQuery`	net472	2.09μs	1.65ns	5.7ns	0.156	0.00105	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6498

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0	1.119	1,303.51	1,164.74

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.3μs	1.81ns	6.77ns	0.0134	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	2.42ns	9.36ns	0.013	976 B
master	`CallElasticsearch`	net472	2.47μs	1.18ns	4.43ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	0.441ns	1.65ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	0.577ns	2.16ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.6μs	0.815ns	2.82ns	0.167	1.05 KB
#6498	`CallElasticsearch`	net6.0	1.17μs	0.507ns	1.96ns	0.014	976 B
#6498	`CallElasticsearch`	netcoreapp3.1	1.55μs	0.61ns	2.28ns	0.0134	976 B
#6498	`CallElasticsearch`	net472	2.61μs	2.8ns	10.8ns	0.157	995 B
#6498	`CallElasticsearchAsync`	net6.0	1.38μs	0.522ns	1.95ns	0.0132	952 B
#6498	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	0.7ns	2.43ns	0.014	1.02 KB
#6498	`CallElasticsearchAsync`	net472	2.73μs	1.95ns	7.55ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.37μs	0.395ns	1.42ns	0.013	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	0.859ns	3.33ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.82μs	0.896ns	3.47ns	0.145	915 B
#6498	`ExecuteAsync`	net6.0	1.3μs	2.9ns	10.8ns	0.0133	952 B
#6498	`ExecuteAsync`	netcoreapp3.1	1.6μs	1.21ns	4.67ns	0.0128	952 B
#6498	`ExecuteAsync`	net472	1.81μs	0.56ns	2.09ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.46μs	1.65ns	6.37ns	0.0314	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.4μs	1.84ns	7.13ns	0.0379	2.85 KB
master	`SendAsync`	net472	7.61μs	1.62ns	6.29ns	0.493	3.12 KB
#6498	`SendAsync`	net6.0	4.54μs	1.97ns	7.1ns	0.0318	2.31 KB
#6498	`SendAsync`	netcoreapp3.1	5.26μs	2.84ns	11ns	0.0368	2.85 KB
#6498	`SendAsync`	net472	7.35μs	2.37ns	9.16ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.48μs	0.708ns	2.65ns	0.0229	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.2μs	1.25ns	4.69ns	0.022	1.64 KB
master	`EnrichedLog`	net472	2.65μs	2.56ns	9.93ns	0.249	1.57 KB
#6498	`EnrichedLog`	net6.0	1.59μs	0.431ns	1.61ns	0.0231	1.64 KB
#6498	`EnrichedLog`	netcoreapp3.1	2.2μs	1.49ns	5.76ns	0.0228	1.64 KB
#6498	`EnrichedLog`	net472	2.68μs	1.56ns	6.06ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	135ns	525ns	0.059	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	157ns	607ns	0	0	4.28 KB
master	`EnrichedLog`	net472	150μs	114ns	426ns	0.677	0.226	4.46 KB
#6498	`EnrichedLog`	net6.0	120μs	252ns	976ns	0.059	0	4.28 KB
#6498	`EnrichedLog`	netcoreapp3.1	121μs	221ns	856ns	0.0608	0	4.28 KB
#6498	`EnrichedLog`	net472	154μs	207ns	802ns	0.691	0.23	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.91μs	0.908ns	3.52ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.36μs	2.3ns	8.61ns	0.0284	2.2 KB
master	`EnrichedLog`	net472	4.86μs	1.99ns	7.69ns	0.321	2.02 KB
#6498	`EnrichedLog`	net6.0	3.15μs	1.16ns	4.51ns	0.0299	2.2 KB
#6498	`EnrichedLog`	netcoreapp3.1	4.13μs	0.944ns	3.4ns	0.0292	2.2 KB
#6498	`EnrichedLog`	net472	4.97μs	0.843ns	3.27ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.66ns	2.56ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.73μs	0.74ns	2.86ns	0.0156	1.14 KB
master	`SendReceive`	net472	2.09μs	0.989ns	3.83ns	0.184	1.16 KB
#6498	`SendReceive`	net6.0	1.36μs	0.929ns	3.6ns	0.0157	1.14 KB
#6498	`SendReceive`	netcoreapp3.1	1.83μs	2.76ns	10.7ns	0.0154	1.14 KB
#6498	`SendReceive`	net472	2.09μs	1.24ns	4.79ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.67μs	0.721ns	2.79ns	0.0227	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.84μs	0.924ns	3.2ns	0.0211	1.65 KB
master	`EnrichedLog`	net472	4.39μs	1.53ns	5.71ns	0.323	2.04 KB
#6498	`EnrichedLog`	net6.0	2.68μs	0.88ns	3.29ns	0.0229	1.6 KB
#6498	`EnrichedLog`	netcoreapp3.1	3.82μs	2.21ns	7.98ns	0.0214	1.65 KB
#6498	`EnrichedLog`	net472	4.41μs	3.36ns	13ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6498

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.149	481.21	552.70

Faster 🎉 in #6498

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.154	808.11	700.02
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.113	609.62	547.66

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	399ns	0.229ns	0.887ns	0.00804	576 B
master	`StartFinishSpan`	netcoreapp3.1	610ns	0.455ns	1.7ns	0.00768	576 B
master	`StartFinishSpan`	net472	742ns	0.301ns	1.17ns	0.0915	578 B
master	`StartFinishScope`	net6.0	481ns	0.279ns	1.08ns	0.00991	696 B
master	`StartFinishScope`	netcoreapp3.1	808ns	0.536ns	2.08ns	0.00929	696 B
master	`StartFinishScope`	net472	961ns	0.532ns	2.06ns	0.104	658 B
#6498	`StartFinishSpan`	net6.0	417ns	0.375ns	1.45ns	0.00814	576 B
#6498	`StartFinishSpan`	netcoreapp3.1	547ns	0.443ns	1.72ns	0.00776	576 B
#6498	`StartFinishSpan`	net472	686ns	1.82ns	7.03ns	0.0918	578 B
#6498	`StartFinishScope`	net6.0	552ns	0.347ns	1.34ns	0.00973	696 B
#6498	`StartFinishScope`	netcoreapp3.1	700ns	0.659ns	2.55ns	0.00954	696 B
#6498	`StartFinishScope`	net472	884ns	0.319ns	1.24ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6498

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1	1.137	857.63	974.88

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	588ns	0.358ns	1.38ns	0.00986	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	858ns	0.299ns	1.12ns	0.00947	696 B
master	`RunOnMethodBegin`	net472	1.14μs	0.509ns	1.97ns	0.104	658 B
#6498	`RunOnMethodBegin`	net6.0	614ns	0.312ns	1.17ns	0.00985	696 B
#6498	`RunOnMethodBegin`	netcoreapp3.1	975ns	0.699ns	2.71ns	0.0093	696 B
#6498	`RunOnMethodBegin`	net472	1.17μs	0.813ns	3.15ns	0.104	658 B

andrewlock added 9 commits January 3, 2025 13:35

Add a Nuke task that creates a "samples only" sln file from the "mast…

0b007e7

…er" solution file. This makes it easier to build the samples without complex globbing etc in a single shot.

Add new, dedicated CompileSamples Nuke targets that are shared across…

177f9ec

… platforms, and built standalone Note that _currently_ this will require calling the dedicated CompileSamples target until I can tidy it up further

Build the samples in a separate, dedicated stage

78f2e6a

Ensure we build to the dedicated artifacts directory

Fix the integration test sample search location

b584be3

We now have the same location for each platform (which makes it easier to reason about in general)

Fix line-ending issue in ASM test

37ed63a

Fix sample compilations that break when doing full multi-package buil…

e673af2

…d on Windows

Work around CI Visibility horribleness

62ca89b

Add a test for building the samples on macos, to ensure it doesn't ge…

7e8c997

…t broken

Remove now-unnecessary targets

14a2cf9

andrewlock requested review from a team as code owners January 3, 2025 14:25

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build the test sample projects in a separate CI stage #6498

Build the test sample projects in a separate CI stage #6498

andrewlock commented Jan 3, 2025 •

edited

Loading

andrewlock commented Jan 3, 2025

datadog-ddstaging bot commented Jan 3, 2025 •

edited

Loading

andrewlock commented Jan 3, 2025

Slower ⚠️ in #6498

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6498

Raw results

andrewlock commented Jan 3, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6498

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6498

Faster 🎉 in #6498

Raw results

Slower ⚠️ in #6498

Raw results

Build the test sample projects in a separate CI stage #6498

Are you sure you want to change the base?

Build the test sample projects in a separate CI stage #6498

Conversation

andrewlock commented Jan 3, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Jan 3, 2025

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Jan 3, 2025 • edited Loading

Datadog Report

New Flaky Tests (1)

andrewlock commented Jan 3, 2025

Benchmarks Report for appsec 🐌

Benchmark details

Slower ⚠️ in #6498

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6498

Raw results

andrewlock commented Jan 3, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6498

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6498

Faster 🎉 in #6498

Raw results

Slower ⚠️ in #6498

Raw results

andrewlock commented Jan 3, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 3, 2025 •

edited

Loading