Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet #2837

silkfire · 2025-10-10T15:50:51Z

This PR introduces an extensive set of analyzers that warns the user of incorrect usage of BenchmarkDotNet. This is something that has been asked since 2017 but has yet to be included as of this date. BDN has a set of validators that use reflection to detect errors but they are only triggered after the benchmark code has been compiled and is about to run.

I had the idea to implement this in 2022 but the testing framework back then wasn't trivial to use so I gave up in the end. Today, the Roslyn analyzer testing is completely testing framework-agnostic, making things considerably easier. It's also trivial to add multiple source files, references and framework assemblies in order to test your analyzer precisely the way you want.
All unit tests are implemented using xUnit v2.

With these analyzers, developers can detect errors early and solve them immediately. The descriptions are very clear and succinct, guiding the user and explaining the reasoning behind the specific rule.

Here's a list of currently implemented analyzers. There are still some remaining but I believe this is a good start and covers most common usage errors. The rest is up for grabs and can be added along the way.

Rule ID	Category	Severity	Title	Description	Comment
BDN1000	Usage	Error	Benchmark class has no annotated method(s)	The referenced benchmark class must have at least one method annotated with the [Benchmark] attribute	Triggers when invoking `BenchmarkRunner.Run<BenchmarkClass>()` and the benchmark class (`BenchmarkClass`) has no public methods marked with the `[Benchmark]` attribute
BDN1001	Usage	Error	Benchmark methods must be public	A method annotated with the [Benchmark] attribute must be public
BDN1002	Usage	Error	Benchmark methods must be non-generic	A method annotated with the [Benchmark] attribute must be non-generic
BDN1003	Usage	Error	Benchmark classes must be public	A benchmark class must be public
BDN1004	Usage	Error	Benchmark classes must be non-static	A benchmark class must be an instance class
BDN1005	Usage	Error	Benchmark classes must be non-abstract	A benchmark class must be non-abstract
BDN1006	Usage	Error	Benchmark classes not annotated with the [GenericTypeArguments] attribute must be non-generic	A benchmark class not annotated with the [GenericTypeArguments] attribute must be non-generic
BDN1007	Usage	Error	Benchmark classes annotated with the [GenericTypeArguments] attribute must be generic	A benchmark class annotated with the [GenericTypeArguments] attribute must be generic, having between one to three type parameters
BDN1008	Usage	Error	Number of type arguments passed to the [GenericTypeArguments] attribute must match the number of type parameters on the targeted benchmark class	The number of type arguments passed to the [GenericTypeArguments] attribute must match the number of type parameters on the targeted benchmark class
BDN1009	Usage	Error	Benchmark classes must be unsealed	A benchmark class bust be unsealed
BDN1010	Usage	Error	Only one benchmark method can be baseline	Only one benchmark method can be marked as baseline
BDN1011	Usage	Error	Only one parameter attribute can be applied to a field	Parameter attributes are mutually exclusive; only one of the attributes [Params], [ParamsSource] or [ParamsAllValues] can be applied to a field at any one time
BDN1012	Usage	Error	Only one parameter attribute can be applied to a property	Parameter attributes are mutually exclusive; only one of the attributes [Params], [ParamsSource] or [ParamsAllValues] can be applied to a property at any one time
BDN1013	Usage	Error	Fields annotated with a parameter attribute must be public	A field annotated with a parameter attribute must be public
BDN1014	Usage	Error	Properties annotated with a parameter attribute must be public	A property annotated with a parameter attribute must be public
BDN1015	Usage	Error	Fields annotated with a parameter attribute cannot be read-only	Parameter attributes are not valid on fields with a readonly modifier
BDN1016	Usage	Error	Parameter attributes are not valid on constant field declarations	Parameter attributes are not valid on constant field declarations
BDN1017	Usage	Error	Properties annotated with a parameter attribute cannot have an init-only setter	A property annotated with a parameter attribute must have a public, assignable setter i.e. { set; }
BDN1018	Usage	Error	Properties annotated with a parameter attribute must have a public setter	A property annotated with a parameter attribute must have a public setter; make sure that the access modifier of the setter is empty and that the property is not an auto-property or an expression-bodied property.
BDN1019	Usage	Error	The [Params] attribute must include at least one value	The [Params] attribute requires at least one value. No values were provided, or an empty array was specified.
BDN1020	Usage	Error	Type of all value(s) passed to the [Params] attribute must match the type of the annotated field or property	The type of each value provided to the [Params] attribute must match the type of the field or property it is applied to
BDN1021	Usage	Warning	Unnecessary single value passed to [Params] attribute	Providing a single value to the [Params] attribute is unnecessary. This attribute is only useful when provided two or more values.
BDN1022	Usage	Error	The [ParamsAllValues] attribute cannot be applied to fields or properties of enum types marked with [Flags]	The [ParamsAllValues] attribute cannot be applied to a field or property of an enum type marked with the [Flags] attribute. Use this attribute only with non-flags enum types, as [Flags] enums support bitwise combinations that cannot be exhaustively enumerated.
BDN1023	Usage	Error	The [ParamsAllValues] attribute is only valid on fields or properties of enum or bool type and nullable type for another allowed type	The [ParamsAllValues] attribute can only be applied to a field or property of enum or bool type (or nullable of these types)
BDN1024	Usage	Error	[Arguments] attribute can only be used on methods annotated with the [Benchmark] attribute	The [Arguments] attribute can only be used on methods annotated with the [Benchmark] attribute
BDN1025	Usage	Error	Benchmark methods without [Arguments] attribute(s) cannot declare parameters	This method declares one or more parameters but is not annotated with any [Arguments] attributes. To ensure correct argument binding, methods with parameters must explicitly be annotated with one or more [Arguments] attributes. Either add the [Arguments] attribute(s) or remove the parameters.
BDN1026	Usage	Error	Number of values passed to an [Arguments] attribute must match the number of parameters declared in the targeted benchmark method	The number of values passed to an [Arguments] attribute must match the number of parameters declared in the targeted benchmark method
BDN1027	Usage	Error	Values passed to an [Arguments] attribute must match exactly the parameters declared in the targeted benchmark method in both type and order	The values passed to an [Arguments] attribute must match the parameters declared in the targeted benchmark method in both type (or be convertible to) and order

TODO

Check that the argument to [ArgumentsSource] points to a valid method
Check that the argument to [ParamsSource] points to a valid method
Check that each corresponding parameter of a [GenericTypeArguments] attribute matches the type of the type parameter

See #2666 for discussion as well as #389.

silkfire · 2025-10-10T16:16:07Z

@dotnet-policy-service agree

...markDotNet.Analyzers.Tests/AnalyzerTests/Attributes/ParamsAllValuesAttributeAnalyzerTests.cs

...hmarkDotNet.Analyzers/BenchmarkDotNet.Analyzers.Tests/BenchmarkDotNet.Analyzers.Tests.csproj

src/BenchmarkDotNet.Disassembler.x64/BenchmarkDotNet.Disassembler.x64.csproj

silkfire · 2025-10-10T19:18:09Z

I'm not sure whether the analyzers should be automatically enabled with the base BenchmarkDotNet package or be opt-in via its own NuGet package, what do you think?

timcassell · 2025-10-10T19:20:21Z

They should be enabled by default.

silkfire · 2025-10-10T19:26:26Z

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

timcassell · 2025-10-10T19:53:12Z

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

I actually think the analyzer should be included directly into the annotations package. Otherwise, it was found that a separate analyzer package pulls in too many unnecessary dependencies. It's a bit complicated to set up the build to do it, though, so I can do it separately after this is merged if you want. [Edit] Or I can push to your branch after your changes are complete.

silkfire · 2025-10-10T22:26:51Z

I'm getting Referenced assembly 'BenchmarkDotNet.Analyzers, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have a strong name. when trying to compile the test project.

timcassell · 2025-10-10T22:58:04Z

You need to import common.props in the analyzer project, too.

silkfire · 2025-10-10T23:19:57Z

Solved it. I also needed to add the public key of the assembly to the InternalsVisibleTo attribute.

silkfire · 2025-10-10T23:25:54Z

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

I actually think the analyzer should be included directly into the annotations package. Otherwise, it was found that a separate analyzer package pulls in too many unnecessary dependencies. It's a bit complicated to set up the build to do it, though, so I can do it separately after this is merged if you want. [Edit] Or I can push to your branch after your changes are complete.

Do I just reference the analyzer project from the annotations project? Or do I need to do something special for the analyzers to activate for the user? Mind that we of course don't want the analyzers to activate for the annotations project, but transitively for the user.

timcassell · 2025-10-10T23:27:24Z

Do I just reference the analyzer project from the annotations project?

Yes that's sufficient for now.

timcassell · 2025-10-10T23:29:17Z

Also you should move the analyzers test project to under the tests/ directory (and move the analyzers project up 1 level).

…ectories

timcassell · 2025-10-11T00:06:13Z

src/BenchmarkDotNet.Analyzers/AnalyzerReleases.Unshipped.md

+BDN1003  |  Usage   | Error	   | BDN1003_General_BenchmarkClass_ClassMustBePublic
+BDN1004  |  Usage   | Error	   | BDN1004_General_BenchmarkClass_ClassMustBeNonStatic
+BDN1005  |  Usage   | Error	   | BDN1005_General_BenchmarkClass_ClassMustBeNonAbstract
+BDN1006  |  Usage   | Error	   | BDN1006_General_BenchmarkClass_ClassMustBeNonGeneric


This is an incorrect error (or a misleading description). Benchmark classes may be generic, but if they are, they must have [GenericTypeArguments].

Also the attribute might not be required if it's abstract and a derived class supplies the generic types.

If a non-generic class is annotated with [GenericTypeArguments], should I loop infinitely over its ancestor classes until I find one that is generic?

No, non-generic classes should not have [GenericTypeArguments].

timcassell · 2025-10-11T00:10:40Z

src/BenchmarkDotNet.Analyzers/AnalyzerReleases.Unshipped.md

+BDN1002  |  Usage   | Error	   | BDN1002_General_BenchmarkClass_MethodMustBeNonGeneric
+BDN1003  |  Usage   | Error	   | BDN1003_General_BenchmarkClass_ClassMustBePublic
+BDN1004  |  Usage   | Error	   | BDN1004_General_BenchmarkClass_ClassMustBeNonStatic
+BDN1005  |  Usage   | Error	   | BDN1005_General_BenchmarkClass_ClassMustBeNonAbstract


I'm not sure if this rule is entirely correct. I think users may create an abstract base class with benchmarks, then create derived concrete classes.

To be honest, I'm not entirely sure where I got this rule from. Maybe we should just remove it completely.

Perhaps it is only types with [GenericTypeArguments] that must be concrete.

Perhaps? :)

I think I can add a rule that the referenced benchmark class in BenchmarkRunner.Run<BenchmarkClass> cannot be abstract.

timcassell · 2025-10-11T00:45:47Z

src/BenchmarkDotNet.Analyzers/General/BenchmarkClassAnalyzer.cs

+                    context.ReportDiagnostic(Diagnostic.Create(ClassMustBeNonGenericRule, classDeclarationSyntax.TypeParameterList.GetLocation(), classDeclarationSyntax.Identifier.ToString()));
+                }
+            }
+            else if (genericTypeArgumentsAttributes.Length == 1)


Multiple GenericTypeArgumentsAttributes are allowed. You should iterate over them all.

timcassell · 2025-10-11T00:51:01Z

src/BenchmarkDotNet.Analyzers/General/BenchmarkClassAnalyzer.cs

+            }
+
+            var benchmarkMethods = benchmarkMethodsBuilder.ToImmutable();
+            if (benchmarkMethods.Length == 0)


Base/derived classes without benchmarks may still apply other attributes like [GlobalSetup], [Params], etc. These types should be analyzed as well.

I would split each attribute analysis to a separate method and call them all.

src/BenchmarkDotNet.Annotations/BenchmarkDotNet.Annotations.csproj

… from BenchmarkDotNet.Annotations

silkfire · 2025-10-11T23:11:51Z

Is Baseline = true unique per benchmark category or per benchmark class?

timcassell · 2025-10-12T03:50:28Z

Is Baseline = true unique per benchmark category or per benchmark class?

It's per category.

Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet

e0bd1d6

silkfire force-pushed the roslyn-analyzers branch from 08ffcd6 to e0bd1d6 Compare October 10, 2025 16:16

timcassell linked an issue Oct 10, 2025 that may be closed by this pull request

Analyzers for Framework #389

Open

timcassell reviewed Oct 10, 2025

View reviewed changes

...markDotNet.Analyzers.Tests/AnalyzerTests/Attributes/ParamsAllValuesAttributeAnalyzerTests.cs Outdated Show resolved Hide resolved

timcassell reviewed Oct 10, 2025

View reviewed changes

...markDotNet.Analyzers.Tests/AnalyzerTests/Attributes/ParamsAllValuesAttributeAnalyzerTests.cs Outdated Show resolved Hide resolved

timcassell reviewed Oct 10, 2025

View reviewed changes

...hmarkDotNet.Analyzers/BenchmarkDotNet.Analyzers.Tests/BenchmarkDotNet.Analyzers.Tests.csproj Outdated Show resolved Hide resolved

timcassell reviewed Oct 10, 2025

View reviewed changes

src/BenchmarkDotNet.Disassembler.x64/BenchmarkDotNet.Disassembler.x64.csproj Outdated Show resolved Hide resolved

silkfire force-pushed the roslyn-analyzers branch from 4cec602 to de94c5b Compare October 10, 2025 22:34

Unify C# language version

c18a417

silkfire force-pushed the roslyn-analyzers branch from de94c5b to c18a417 Compare October 10, 2025 23:19

silkfire added 2 commits October 11, 2025 01:21

Remove Analyzers package projects

19e6ff9

Revert BenchmarkDotNet.Disassembler changes

58ce544

Reference Analyzers project from Annotations

86d3a9c

Move Benchmark.Analyzers and Benchmark.Analyzers.Tests to correct dir…

770e190

…ectories

timcassell reviewed Oct 11, 2025

View reviewed changes

src/BenchmarkDotNet.Annotations/BenchmarkDotNet.Annotations.csproj Outdated Show resolved Hide resolved

Remove accidentally added package Microsoft.CodeAnalysis.NetAnalyzers…

9d2423c

… from BenchmarkDotNet.Annotations

Uh oh!

Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet #2837

Are you sure you want to change the base?

Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet #2837

Conversation

silkfire commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

silkfire commented Oct 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

silkfire commented Oct 10, 2025

Uh oh!

timcassell commented Oct 10, 2025

Uh oh!

silkfire commented Oct 10, 2025

Uh oh!

timcassell commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

silkfire commented Oct 10, 2025

Uh oh!

timcassell commented Oct 10, 2025

Uh oh!

silkfire commented Oct 10, 2025

Uh oh!

silkfire commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timcassell commented Oct 10, 2025

Uh oh!

timcassell commented Oct 10, 2025

Uh oh!

timcassell Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timcassell Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

silkfire Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timcassell Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

timcassell Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

silkfire Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timcassell Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

silkfire Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

timcassell Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

timcassell Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timcassell Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

silkfire commented Oct 11, 2025

Uh oh!

timcassell commented Oct 12, 2025

Uh oh!

Reviewers

silkfire commented Oct 10, 2025 •

edited

Loading

timcassell commented Oct 10, 2025 •

edited

Loading

silkfire commented Oct 10, 2025 •

edited

Loading

timcassell Oct 11, 2025 •

edited

Loading

silkfire Oct 11, 2025 •

edited

Loading

silkfire Oct 11, 2025 •

edited

Loading

timcassell Oct 11, 2025 •

edited

Loading