[WIP] [Infra] Onboarding eval to msbench

[JanKrivanek]

Approach

• explicit opt-in flag in eval.yaml indicates the scenario is meant for msbench eval (msbench_ready: true)
• harbor format transformation is scripted out (sourcing the scenario files and eval.yaml)
• metrics are now sourced from the grading defined by scenario

TBDs

• review and improve the eval/grading
• onboard to msbench acr and push images
• e2e testing; automated pipeline

[github-actions]

Skill Validation Results

Skill	Scenario	Baseline	With Skill	Δ	Skills Loaded	Overfit	Verdict
dotnet-trace-collect	High CPU in Kubernetes on Linux (.NET 8)	4.0/5	4.0/5	0.0	✅ dotnet-trace-collect; tools: skill, report_intent, view, glob, bash	✅ 0.13	✅
dotnet-trace-collect	.NET Framework on Windows without admin privileges	1.0/5 ⏰ timeout	5.0/5	+4.0	✅ dotnet-trace-collect; tools: skill	✅ 0.13	✅
dotnet-trace-collect	.NET 10 on Linux with root access and native call stacks	1.0/5	3.5/5	+2.5	✅ dotnet-trace-collect; tools: skill	✅ 0.13	✅
dotnet-trace-collect	Memory leak on Linux (.NET 8)	2.5/5 ⏰ timeout	3.0/5	+0.5	✅ dotnet-trace-collect; tools: skill, report_intent, view, glob, bash	✅ 0.13	✅
dotnet-trace-collect	Slow requests on Windows with PerfView	4.5/5	5.0/5	+0.5	✅ dotnet-trace-collect; tools: skill, report_intent, view	✅ 0.13	✅
dotnet-trace-collect	Excessive GC on Linux (.NET 8)	4.5/5	5.0/5	+0.5	✅ dotnet-trace-collect; tools: skill, bash	✅ 0.13	✅
dotnet-trace-collect	Hang or deadlock diagnosis on Linux	2.5/5	4.0/5	+1.5	✅ dotnet-trace-collect; tools: skill	✅ 0.13	✅
dotnet-trace-collect	Windows container high CPU with PerfView	1.5/5	4.5/5	+3.0	✅ dotnet-trace-collect; tools: skill, report_intent, view, bash	✅ 0.13	✅
dotnet-trace-collect	Long-running intermittent issue with PerfView triggers	2.0/5	5.0/5	+3.0	✅ dotnet-trace-collect; tools: skill, report_intent, view, glob	✅ 0.13	✅
dotnet-trace-collect	Linux pre-.NET 10 needing native call stacks	3.0/5	5.0/5	+2.0	✅ dotnet-trace-collect; tools: skill, bash	✅ 0.13	✅
dotnet-trace-collect	Windows modern .NET with admin high CPU	2.5/5	4.5/5	+2.0	✅ dotnet-trace-collect; tools: skill, report_intent, view, glob	✅ 0.13	✅
dotnet-trace-collect	Memory leak on .NET Framework Windows	3.5/5	4.5/5	+1.0	✅ dump-collect; dotnet-trace-collect; tools: report_intent, skill, view, glob, bash	✅ 0.13	✅
dotnet-trace-collect	Kubernetes with console access prefers console tools	4.5/5	5.0/5	+0.5	✅ dotnet-trace-collect; tools: skill, report_intent, view, glob, bash	✅ 0.13	❌
dotnet-trace-collect	Container installation without .NET SDK	4.0/5	4.5/5	+0.5	✅ dotnet-trace-collect; tools: skill, bash	✅ 0.13	✅
dotnet-trace-collect	HTTP 500s from downstream service on Linux (.NET 8)	4.0/5	5.0/5	+1.0	✅ dotnet-trace-collect; tools: skill, report_intent, view, glob, bash	✅ 0.13	✅
dotnet-trace-collect	Networking timeouts on Windows with admin (.NET 8)	2.0/5	4.5/5	+2.5	✅ dotnet-trace-collect; tools: skill, report_intent, view, glob	✅ 0.13	✅
microbenchmarking	Investigate runtime upgrade performance impact	2.5/5	4.0/5	+1.5	✅ microbenchmarking; tools: skill, glob, stop_bash	✅ 0.11	✅
csharp-scripts	Test a C# language feature with a script	3.0/5	4.0/5	+1.0	✅ csharp-scripts; tools: skill, create, edit, view	🟡 0.25	✅
clr-activation-debugging	Diagnose unexpected FOD dialog from native build tool	1.0/5	5.0/5	+4.0	✅ clr-activation-debugging; tools: skill	✅ 0.07	✅
clr-activation-debugging	Diagnose FOD suppressed but activation still failing	1.0/5	5.0/5	+4.0	✅ clr-activation-debugging; tools: skill	✅ 0.07	✅
clr-activation-debugging	Explain why same binary behaves differently under different launch methods	1.0/5	5.0/5	+4.0	✅ clr-activation-debugging; tools: skill	✅ 0.07	✅
clr-activation-debugging	Analyze healthy managed EXE activation	1.0/5	5.0/5	+4.0	✅ clr-activation-debugging; tools: skill	✅ 0.07	✅
clr-activation-debugging	Identify multiple activation sequences in a single log	1.0/5	5.0/5	+4.0	✅ clr-activation-debugging; tools: skill	✅ 0.07	✅
clr-activation-debugging	Explain useLegacyV2RuntimeActivationPolicy in activation log	2.0/5	3.5/5	+1.5	✅ clr-activation-debugging; tools: skill	✅ 0.07	✅
clr-activation-debugging	Decline non-CLR-activation issue	1.0/5	5.0/5	+4.0	ℹ️ not activated (expected)	✅ 0.07	✅
thread-abort-migration	Worker thread with abort-based cancellation	5.0/5	5.0/5	0.0	✅ thread-abort-migration; tools: skill, report_intent, bash	✅ 0.13	✅
thread-abort-migration	Timeout enforcement via Thread.Abort	4.0/5	4.5/5	+0.5	✅ thread-abort-migration; tools: skill	✅ 0.13	✅
thread-abort-migration	Blocking WaitHandle with Thread.Interrupt	4.0/5	4.5/5	+0.5	✅ thread-abort-migration; tools: skill	✅ 0.13	✅
thread-abort-migration	ASP.NET Response.End and Response.Redirect with Thread.Abort	4.0/5	5.0/5	+1.0	✅ thread-abort-migration; tools: skill	✅ 0.13	✅
thread-abort-migration	Thread.Join and Thread.Sleep only — should not migrate	4.0/5	5.0/5	+1.0	✅ thread-abort-migration; tools: skill	✅ 0.13	✅
migrate-nullable-references	Enable NRT in a small library with mixed nullability	5.0/5	5.0/5	0.0	✅ migrate-nullable-references; tools: skill, glob	✅ 0.06	❌
migrate-nullable-references	File-by-file migration: only modify the targeted file	5.0/5	5.0/5	0.0	⚠️ NOT ACTIVATED	✅ 0.06	❌
migrate-nullable-references	Enable NRT in ASP.NET Core Web API with EF Core	3.0/5	3.0/5	0.0	✅ migrate-nullable-references; tools: skill	✅ 0.06	❌
nuget-trusted-publishing	Set up trusted publishing for a new NuGet library	3.0/5	4.0/5	+1.0	✅ nuget-trusted-publishing; tools: skill, glob	✅ 0.11	✅
nuget-trusted-publishing	Set up NuGet publishing without mentioning trusted publishing	2.5/5	4.5/5	+2.0	✅ nuget-trusted-publishing; tools: skill, glob, view, report_intent	✅ 0.11	✅
nuget-trusted-publishing	Migrate existing workflow from API key to trusted publishing	3.0/5	4.0/5	+1.0	✅ nuget-trusted-publishing; tools: skill, bash	✅ 0.11	✅
analyzing-dotnet-performance	Detects compiled regex startup budget and regex chain allocations	1.0/5	3.5/5 ⏰ timeout	+2.5	✅ analyzing-dotnet-performance; tools: skill	✅ 0.18	✅
analyzing-dotnet-performance	Detects CurrentCulture comparer and compiled regex budget in inflection rules	1.0/5	3.0/5 ⏰ timeout	+2.0	✅ analyzing-dotnet-performance; tools: skill, task, glob	✅ 0.18	✅
analyzing-dotnet-performance	Finds per-call Dictionary allocation not hoisted to static	1.0/5	5.0/5 ⏰ timeout	+4.0	✅ analyzing-dotnet-performance; tools: skill	✅ 0.18	✅
analyzing-dotnet-performance	Catches compound allocations in recursive number converter with ToLower	1.0/5	5.0/5	+4.0	✅ analyzing-dotnet-performance; tools: skill	✅ 0.18	✅
analyzing-dotnet-performance	Finds StringComparison.Ordinal missing and FrozenDictionary opportunities	1.0/5	5.0/5	+4.0	✅ analyzing-dotnet-performance; tools: skill	✅ 0.18	✅
analyzing-dotnet-performance	Detects Aggregate+Replace chain and struct missing IEquatable	1.0/5	3.5/5 ⏰ timeout	+2.5	✅ analyzing-dotnet-performance; tools: skill, read_bash	✅ 0.18	✅
analyzing-dotnet-performance	Finds branched Replace chain in format string manipulation	1.0/5	4.0/5	+3.0	✅ analyzing-dotnet-performance; tools: skill, stop_bash	✅ 0.18	✅
analyzing-dotnet-performance	Catches LINQ on hot-path string processing and All(char.IsUpper)	1.0/5	5.0/5	+4.0	✅ analyzing-dotnet-performance; tools: skill	✅ 0.18	✅
analyzing-dotnet-performance	Detects LINQ pipeline in TimeSpan formatting and collection processing	1.0/5	4.0/5	+3.0	✅ analyzing-dotnet-performance; tools: skill	✅ 0.18	✅
analyzing-dotnet-performance	Flags Span inconsistencies and compound method chains in truncation library	1.0/5	5.0/5	+4.0	✅ analyzing-dotnet-performance; tools: skill, stop_bash	✅ 0.18	✅
analyzing-dotnet-performance	Identifies unsealed leaf classes and locale hierarchy patterns	1.0/5	4.0/5 ⏰ timeout	+3.0	✅ analyzing-dotnet-performance; tools: skill, task, glob	✅ 0.18	✅
dotnet-aot-compat	Make Azure.ResourceManager AOT-compatible	2.0/5	2.0/5 ⏰ timeout	0.0	✅ dotnet-aot-compat; tools: skill, bash, read_agent	✅ 0.10	❌
optimizing-ef-core-queries	Optimize bulk operations with EF Core 7+ ExecuteUpdate and ExecuteDelete	5.0/5	4.5/5	-0.5	✅ optimizing-ef-core-queries; tools: skill	🟡 0.29	✅
android-tombstone-symbolication	Symbolicate .NET frames in an Android tombstone	4.0/5	5.0/5	+1.0	✅ android-tombstone-symbolication; tools: skill, stop_bash	✅ 0.17	✅
android-tombstone-symbolication	Recognize tombstone with no .NET frames	5.0/5	5.0/5	0.0	✅ android-tombstone-symbolication; tools: skill, bash	✅ 0.17	❌
android-tombstone-symbolication	Symbolicate CoreCLR frames in an Android tombstone	3.0/5	4.0/5	+1.0	✅ android-tombstone-symbolication; tools: skill, glob	✅ 0.17	❌
android-tombstone-symbolication	Recognize NativeAOT tombstone with app binary and libSystem.Native.so	3.5/5	4.5/5	+1.0	✅ android-tombstone-symbolication; tools: skill, bash, stop_bash	✅ 0.17	✅
android-tombstone-symbolication	Symbolicate multi-thread tombstone	4.5/5	5.0/5	+0.5	✅ android-tombstone-symbolication; tools: skill, stop_bash, glob	✅ 0.17	❌
android-tombstone-symbolication	Handle .NET frames with no BuildId metadata	5.0/5	4.5/5	-0.5	✅ android-tombstone-symbolication; tools: skill, bash	✅ 0.17	✅
android-tombstone-symbolication	Symbolicate tombstone with multiple .NET libraries and different BuildIds	4.0/5	4.0/5	0.0	✅ android-tombstone-symbolication; tools: skill, glob	✅ 0.17	✅
android-tombstone-symbolication	Reject iOS crash log as wrong format	5.0/5	5.0/5	0.0	ℹ️ not activated (expected)	✅ 0.17	✅
dotnet-pinvoke	Generate LibraryImport declaration from C header (.NET 8+)	4.5/5	5.0/5	+0.5	✅ dotnet-pinvoke; tools: skill	✅ 0.10	✅
dotnet-pinvoke	Generate LibraryImport declaration from C header (.NET Framework)	5.0/5	5.0/5	0.0	✅ dotnet-pinvoke; tools: report_intent, skill	✅ 0.10	❌
dump-collect	Configure automatic crash dumps for CoreCLR app on Linux	5.0/5	5.0/5	0.0	✅ dump-collect; tools: skill, report_intent, view, glob, bash	✅ 0.09	✅
dump-collect	Set up NativeAOT crash dumps with createdump in Kubernetes	2.0/5 ⏰ timeout	5.0/5	+3.0	✅ dump-collect; tools: skill	✅ 0.09	✅
dump-collect	Recover crash dump from macOS NativeAOT without createdump	3.0/5	3.0/5	0.0	✅ dump-collect; tools: skill, report_intent, view, glob, bash	✅ 0.09	✅
dump-collect	Configure CoreCLR dump collection in Alpine Docker as non-root	3.5/5	4.0/5	+0.5	✅ dump-collect; tools: skill, report_intent, view, glob, bash	✅ 0.09	✅
dump-collect	Advisory: macOS NativeAOT crash dump recovery steps	4.5/5	3.0/5	-1.5	✅ dump-collect; tools: skill, glob, bash	✅ 0.09	❌
dump-collect	Advisory: CoreCLR Alpine Docker non-root configuration	3.5/5	4.5/5	+1.0	✅ dump-collect; tools: skill, bash	✅ 0.09	✅
dump-collect	Advisory: NativeAOT Kubernetes dump collection setup	2.5/5	2.5/5 ⏰ timeout	0.0	✅ dump-collect; tools: skill	✅ 0.09	❌
dump-collect	Detect runtime and configure crash dumps for unknown .NET app on Linux	3.5/5	4.5/5	+1.0	✅ dump-collect; tools: skill, glob	✅ 0.09	✅
dump-collect	Decline dump analysis request	2.0/5	2.0/5	0.0	✅ tools: bash	✅ 0.09	✅
build-parallelism	Analyze build parallelism bottlenecks	1.0/5 ⏰ timeout	1.0/5 ⏰ timeout	0.0	✅ binlog-generation; build-parallelism; tools: skill, task, edit	✅ 0.16	✅
including-generated-files	Diagnose generated file inclusion failure	3.0/5	5.0/5	+2.0	✅ including-generated-files; tools: skill	✅ 0.13	✅
msbuild-antipatterns	Review MSBuild files for anti-patterns and style issues	4.5/5	5.0/5	+0.5	✅ msbuild-antipatterns; tools: skill, glob, edit	✅ 0.06	✅
build-perf-baseline	Establish build performance baseline and recommend optimizations	2.5/5 ⏰ timeout	4.5/5	+2.0	✅ build-perf-baseline; build-perf-diagnostics; tools: skill	🟡 0.22	✅
msbuild-modernization	Modernize legacy project to SDK-style	5.0/5	5.0/5	0.0	✅ msbuild-modernization; tools: skill, edit, bash	✅ 0.06	✅
directory-build-organization	Organize build infrastructure for a multi-project repo	4.0/5	5.0/5	+1.0	✅ directory-build-organization; msbuild-antipatterns; tools: skill	✅ 0.13	✅
check-bin-obj-clash	Diagnose bin/obj output path clashes	3.5/5	4.5/5 ⏰ timeout	+1.0	✅ check-bin-obj-clash; binlog-generation; tools: skill, glob, edit	✅ 0.15	❌
incremental-build	Analyze incremental build issues	3.0/5	4.5/5	+1.5	✅ incremental-build; tools: skill, bash, edit	✅ 0.14	✅
eval-performance	Analyze MSBuild evaluation performance issues	3.5/5	4.5/5	+1.0	✅ eval-performance; tools: skill	✅ 0.13	✅
build-perf-diagnostics	Analyze analyzer performance impact on builds	3.0/5 ⏰ timeout	4.0/5 ⏰ timeout	+1.0	✅ binlog-generation; build-perf-diagnostics; tools: skill, edit	🟡 0.22	✅
binlog-generation	Build project with /bl flag	1.0/5	5.0/5	+4.0	✅ binlog-generation; tools: skill	🟡 0.48	✅
binlog-generation	Build with /bl in PowerShell	3.0/5	5.0/5	+2.0	✅ binlog-generation; tools: skill	🟡 0.48	✅
binlog-generation	Build multiple configurations with unique binlogs	3.0/5	5.0/5	+2.0	✅ binlog-generation; tools: skill	🟡 0.48	✅
binlog-failure-analysis	Diagnose build failures from binlog only (no source files)	4.0/5	5.0/5	+1.0	✅ binlog-failure-analysis; tools: skill	✅ 0.04	✅

⏰ timeout — run hit the scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output

Model: claude-opus-4.6 | Judge: claude-opus-4.6

Full results