diff --git a/plugins/dotnet-test/.claude-plugin/plugin.json b/plugins/dotnet-test/.claude-plugin/plugin.json index f98445a0c4..a61a7eaece 100644 --- a/plugins/dotnet-test/.claude-plugin/plugin.json +++ b/plugins/dotnet-test/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "dotnet-test", - "version": "0.2.19", + "version": "0.2.21", "description": "Skills for running, generating, analyzing, and improving .NET tests: test execution, filtering, platform detection, coverage, testability, and MSTest workflows.", "skills": ["./skills/"], "agents": [ diff --git a/plugins/dotnet-test/.codex-plugin/plugin.json b/plugins/dotnet-test/.codex-plugin/plugin.json index febfa26a49..debaeeec2e 100644 --- a/plugins/dotnet-test/.codex-plugin/plugin.json +++ b/plugins/dotnet-test/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "dotnet-test", - "version": "0.2.19", + "version": "0.2.21", "description": "Skills for running, generating, analyzing, and improving .NET tests: test execution, filtering, platform detection, coverage, testability, and MSTest workflows.", "skills": ["./skills/"], "agents": [ diff --git a/plugins/dotnet-test/plugin.json b/plugins/dotnet-test/plugin.json index f98445a0c4..a61a7eaece 100644 --- a/plugins/dotnet-test/plugin.json +++ b/plugins/dotnet-test/plugin.json @@ -1,6 +1,6 @@ { "name": "dotnet-test", - "version": "0.2.19", + "version": "0.2.21", "description": "Skills for running, generating, analyzing, and improving .NET tests: test execution, filtering, platform detection, coverage, testability, and MSTest workflows.", "skills": ["./skills/"], "agents": [ diff --git a/plugins/dotnet-test/skills/coverage-analysis/SKILL.md b/plugins/dotnet-test/skills/coverage-analysis/SKILL.md index ce5b4807bc..d5d706468a 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/SKILL.md +++ b/plugins/dotnet-test/skills/coverage-analysis/SKILL.md @@ -1,20 +1,17 @@ --- name: coverage-analysis description: > - Interprets .NET Cobertura line, branch, and condition evidence and, when - explicitly requested, computes project-wide CRAP/refactoring-risk hotspots. - MUST USE for "why is branch coverage lower than line coverage?", - condition-coverage="50% (1/2)", a supplied coverage excerpt, partially covered - conditions, coverage plateaus, members blocking a target, project-wide CRAP, - refactoring safety, or coverage-backed risk priorities. A supplied report is - analyzed directly without rerunning tests, installing tools, generating a - report, or calculating CRAP unless the request asks for risk/CRAP/refactoring - safety. DO NOT USE FOR: CRAP or refactoring-safety analysis of only one named - method, class, or file (crap-score); test trait distributions (test-tagging); - static source-to-test pairing (find-untested-sources); - behavioral/pseudo-mutation gaps (test-gap-analysis); test-code audits - (test-anti-patterns); raw collection/percentage-only requests or just running - tests (run-tests); non-.NET coverage; or writing tests. + Activation requires either supplied .NET coverage reports/percentages/line, + branch, or condition metrics, or an explicit request to collect .NET coverage + for analysis. USE FOR: interpreting that evidence, including Cobertura data, + partial conditions, plateaus, target arithmetic, + project-wide coverage-backed CRAP, and coverage-backed refactoring safety. + Analyze supplied reports directly without rerunning tests or installing + tools. DO NOT USE FOR: requests with neither coverage evidence nor explicit + coverage-collection intent, including hypothetical change-survival questions + (use test-gap-analysis); CRAP or refactoring safety for one named target (use + crap-score); or requests owned by test-tagging, find-untested-sources, + test-anti-patterns, run-tests, or code-testing-agent. license: MIT --- @@ -64,7 +61,7 @@ the current directory or a supplied report is sufficient. | Explain a supplied excerpt, condition, or summary | Answer directly from the supplied evidence | Tools, CRAP, discovery, report files | | Interpret a supplied Cobertura path or diagnose a plateau | Read that report, reconcile totals, name all material gaps, answer directly | Rerun tests, install tools, compute CRAP, or generate files unless explicitly requested | | Rank risk hotspots, compute project-wide CRAP, or assess refactoring safety | Use the supplied/existing report, read `references/guidelines.md`, and compute CRAP before ranking | Coverage-only ranking or a full report template unless requested | -| Analyze coverage when no report exists | Read `references/setup-discovery.md`; collect once using `references/test-execution.md` if safe | CRAP unless risk was requested | +| Analyze coverage when no report exists | Invoke `run-tests` to collect coverage with the repository-compatible runner, then analyze the generated report | Choose or execute a test command independently; CRAP unless risk was requested | | Produce a full markdown/HTML/CSV report | First deliver the direct answer; then read `references/output-format.md` or `references/report-generation.md` | Report generation before the answer | Words such as **analyze coverage**, **what is blocking coverage**, or **why is @@ -79,19 +76,20 @@ When the user supplies a coverage excerpt, summary, or valid Cobertura path: - Treat it as authoritative input and start there. - Do not discover the solution or test projects unless source mapping is necessary. - Do not run `dotnet test`, install ReportGenerator, add a coverage package, or - read `references/setup-discovery.md`, `references/test-execution.md`, or - `references/report-generation.md`. + read `references/setup-discovery.md` or `references/report-generation.md`. - Do not write `coverage-analysis.md` or create a report directory unless the user requested a saved/full report. - For interpretation and plateau questions, parse only the evidence needed to answer. For explicit project-wide risk requests, use the bundled scripts as described in `references/guidelines.md`. -A failed read/view operation is not proof that a named path does not exist. After -one fails, make one allowed targeted existence probe, such as a workspace-relative -glob, and retry the same artifact with a normalized path or alternate reader. -Report the exact missing-path problem only when that independent check also fails. -Do not broaden the search to unrelated coverage files or present a substitute +A failed read/view operation is not proof that a named path does not exist. +Classify the failure, then make one targeted existence probe and use a normalized +path or alternate reader only for confirmed tool availability, transport, or +path-normalization failures and only after verifying the canonical path remains +inside the workspace. Stop on content-exclusion, permission/policy, +workspace-boundary, or unknown failures. Report a missing path only when the +independent probe also fails; do not broaden the search or substitute another artifact. ## Collection path @@ -101,15 +99,19 @@ analysis that requires it. 1. Read `references/setup-discovery.md`. 2. Prefer existing Cobertura discovered under the requested root. -3. If none exists, read `references/test-execution.md` and run the selected - coverage command once per entry point. +3. If none exists, invoke `run-tests` for repository overlay, platform, runner, + and command selection, and have it collect Cobertura once per entry point. 4. Analyze the resulting Cobertura. Compute CRAP only if risk analysis was explicitly requested. -Do not modify production code. The only permitted incidental project change is -adding one missing coverage provider to an SDK-style test project as described in -`references/test-execution.md`; never add a second provider, and report the -change plus its revert command. +When the workspace contains the project/test manifests needed by this path, +perform discovery directly and delegate collection to `run-tests`. Do not ask +the user to attach files or pre-run coverage that the current workspace lets +you inspect and generate. + +Do not modify production code. Any incidental test-project change needed for +collection is owned by `run-tests`; include that change and its revert command +in the final analysis. The automatic collection path is for SDK-style projects. For classic non-SDK or `packages.config` projects, use only a repository-owned coverage command. If none @@ -132,6 +134,8 @@ another assembly. - When asked whether one member can reach a target, show its maximum projected total and at least one concrete sufficient combination of supplied members or line gains. If no supplied combination is sufficient, say so. +- For a multi-member target, name the exact combination and resulting covered / + valid total; saying only "combine it with another member" is incomplete. - Reconcile member gaps against project totals. Method line ranges can overlap or omit class-level lines, so do not sum method counts as project truth. - Never call one member the **sole**, **entire**, or **all** remaining gap unless diff --git a/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md b/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md index 0db1b92cd9..4e380adfdf 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md +++ b/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md @@ -178,14 +178,15 @@ $coberturaFiles | ForEach-Object { Write-Host "EXISTING_COBERTURA:$($_.FullName) ``` - If `EXISTING_COBERTURA_COUNT` > 0 → skip fresh collection and analyze these paths. -- If `EXISTING_COBERTURA_COUNT` == 0 and all test projects are SDK-style → run - the collection workflow in `test-execution.md`. +- If `EXISTING_COBERTURA_COUNT` == 0 and all test projects are SDK-style → + invoke `run-tests` to select the repository-compatible platform/provider + command and collect Cobertura. - If `EXISTING_COBERTURA_COUNT` == 0 and only classic/packages.config projects exist → use a repository-owned coverage command that emits Cobertura; otherwise stop with the message above. - If `EXISTING_COBERTURA_COUNT` == 0 and both classic and SDK-style projects - exist → collect only for `SDK_TEST_PROJECTS` and mark the result partial until - classic-project Cobertura is available. + exist → ask `run-tests` to collect only for `SDK_TEST_PROJECTS` and mark the + result partial until classic-project Cobertura is available. ## Step 2c: Recommend ignoring `TestResults/` diff --git a/plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md b/plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md deleted file mode 100644 index 9f724de6a0..0000000000 --- a/plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md +++ /dev/null @@ -1,219 +0,0 @@ -# Coverage Analysis — fresh test execution - -Read this file **only** when discovery found no existing Cobertura XML -(`EXISTING_COBERTURA_COUNT:0`) and fresh coverage must be produced. When a report -already exists, return to the existing-data path in `SKILL.md`. - -This automatic provider workflow is for `SDK_TEST_PROJECTS` only. Exclude every -`CLASSIC_TEST_PROJECT` before provider detection, package addition, restore, or -test execution. In a mixed solution, run each SDK-style test project separately -instead of running a solution entry point that would include classic projects. -Use a checked-in repository coverage script for the classic subset when one -exists; otherwise label the analysis partial and request its Cobertura report. -Never run `dotnet add package` against a `packages.config` project or introduce -`PackageReference` / an SDK-style conversion implicitly. - -```powershell -$testProjects = @($sdkTestProjects) -if ($testProjects.Count -eq 0) { - throw "No SDK-style test projects are eligible for automatic coverage collection." -} -``` - -## Step 3: Detect coverage provider and run `dotnet test` with coverage collection - -Before running tests, detect which coverage provider the test projects use. Projects may reference -`Microsoft.Testing.Extensions.CodeCoverage` (Microsoft's built-in provider, common on .NET 9+) or -`coverlet.collector` (open-source, the default in xUnit templates). The provider determines which -`dotnet test` arguments to use — both produce Cobertura XML. - -```powershell -# Detect coverage provider per test project -$coverageProvider = "unknown" # will be set to "ms-codecoverage" or "coverlet" -$msCodeCovProjects = @() -$coverletProjects = @() -$neitherProjects = @() - -foreach ($tp in $testProjects) { - $hasMsCodeCov = Select-String -Path $tp.FullName -Pattern 'Microsoft\.Testing\.Extensions\.CodeCoverage' -Quiet - $hasCoverlet = Select-String -Path $tp.FullName -Pattern 'coverlet\.collector' -Quiet - if ($hasMsCodeCov) { $msCodeCovProjects += $tp } - elseif ($hasCoverlet) { $coverletProjects += $tp } - else { $neitherProjects += $tp } -} - -# Determine the provider strategy -if ($msCodeCovProjects.Count -gt 0 -and $coverletProjects.Count -eq 0) { - $coverageProvider = "ms-codecoverage" - Write-Host "COVERAGE_PROVIDER:ms-codecoverage (ms:$($msCodeCovProjects.Count), none:$($neitherProjects.Count))" -} elseif ($coverletProjects.Count -gt 0 -and $msCodeCovProjects.Count -eq 0) { - $coverageProvider = "coverlet" - Write-Host "COVERAGE_PROVIDER:coverlet (coverlet:$($coverletProjects.Count), none:$($neitherProjects.Count))" -} elseif ($msCodeCovProjects.Count -gt 0 -and $coverletProjects.Count -gt 0) { - $coverageProvider = "mixed-project" - Write-Host "COVERAGE_PROVIDER:mixed-project (ms:$($msCodeCovProjects.Count), coverlet:$($coverletProjects.Count), none:$($neitherProjects.Count))" -} else { - $coverageProvider = "coverlet" - Write-Host "COVERAGE_PROVIDER:none-detected — defaulting to coverlet" -} -``` - -If any discovered test projects have no provider, add one based on the selected strategy: - -```powershell -if ($coverageProvider -eq "ms-codecoverage" -and $neitherProjects.Count -gt 0) { - Write-Host "ADDING_MS_CODECOVERAGE:$($neitherProjects.Count) project(s)" - foreach ($tp in $neitherProjects) { - dotnet add $tp.FullName package Microsoft.Testing.Extensions.CodeCoverage --no-restore - Write-Host " ADDED_MS_CODECOVERAGE:$($tp.FullName)" - } - foreach ($tp in $neitherProjects) { - dotnet restore $tp.FullName --quiet - } -} - -if (($coverageProvider -eq "coverlet" -or $coverageProvider -eq "mixed-project") -and $neitherProjects.Count -gt 0) { - Write-Host "ADDING_COVERLET:$($neitherProjects.Count) project(s)" - foreach ($tp in $neitherProjects) { - dotnet add $tp.FullName package coverlet.collector --no-restore - Write-Host " ADDED:$($tp.FullName)" - } - foreach ($tp in $neitherProjects) { - dotnet restore $tp.FullName --quiet - } -} -``` - -Log each addition to the console so the developer sees what changed. Document the additions in the final report (see Output Format). - -Run one `dotnet test` per eligible entry point for the selected strategy: - -- In an all-SDK solution, run a single command for the solution entry. -- In a mixed classic/SDK solution, run once per `SDK_TEST_PROJECT`; never run the solution entry. -- In `mixed-project` mode: run one command per test project, using that project's existing provider to avoid dual-provider conflicts. - -```powershell -$sdkVersion = (dotnet --version 2>$null) -$major = if ($sdkVersion -match '^(\d+)\.') { [int]$Matches[1] } else { 9 } -$searchDir = (Get-Location).Path -$globalJson = $null -while ($searchDir -and -not $globalJson) { - $candidate = Join-Path $searchDir "global.json" - if (Test-Path -LiteralPath $candidate) { - $globalJson = Get-Item -LiteralPath $candidate - break - } - $parent = [System.IO.Directory]::GetParent($searchDir) - $searchDir = if ($parent) { $parent.FullName } else { $null } -} -$configuredRunner = if ($globalJson) { - (Get-Content $globalJson.FullName -Raw | ConvertFrom-Json).test.runner -} else { - $null -} -$dotnetTestMode = if ( - $major -ge 10 -and - $configuredRunner -eq "Microsoft.Testing.Platform" -) { - "native-MTP" -} else { - "VSTest" -} -$coverageEntries = if ($classicTestProjects.Count -gt 0) { - @($sdkTestProjects | ForEach-Object { - [pscustomobject]@{ Path = $_.FullName; Type = "Project" } - }) -} else { - @([pscustomobject]@{ Path = ""; Type = "" }) -} -``` - -**Coverlet** (`coverlet.collector`): - -```powershell -$rawDir = Join-Path "" "raw" -foreach ($entry in $coverageEntries) { - dotnet test $entry.Path ` - --collect:"XPlat Code Coverage" ` - --results-directory $rawDir ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Include="[*]*" ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Exclude="[*.Tests]*,[*.Test]*,[*Tests]*,[*Test]*,[*.Specs]*,[*.Testing]*" ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.SkipAutoProps=true -} -``` - -**Microsoft CodeCoverage** (`Microsoft.Testing.Extensions.CodeCoverage`): - -The command syntax depends on the `dotnet test` runner mode, not the SDK major -version alone. Native MTP mode on .NET 10+ accepts selectors and top-level -coverage options. VSTest mode — including .NET 10 VSTest mode bridging to an -MTP application — keeps the positional project/solution path and passes MTP -coverage arguments after `--`. - -```powershell -$rawDir = Join-Path "" "raw" - -foreach ($entry in $coverageEntries) { - if ($dotnetTestMode -eq "native-MTP") { - # Native MTP mode: selectors and coverage are top-level dotnet test options. - $selector = if ($entry.Type -eq "Solution") { "--solution" } else { "--project" } - dotnet test $selector $entry.Path ` - --results-directory $rawDir ` - --coverage ` - --coverage-output-format cobertura ` - --coverage-output $rawDir - } else { - # VSTest mode (including an MTP bridge): keep the positional path and - # pass Microsoft.Testing.Platform arguments after the separator. - dotnet test $entry.Path ` - --results-directory $rawDir ` - -- --coverage --coverage-output-format cobertura --coverage-output $rawDir - } -} -``` - -**Mixed-project mode** (`Microsoft.Testing.Extensions.CodeCoverage` + `coverlet.collector` in the same solution): - -```powershell -$rawDir = Join-Path "" "raw" -foreach ($tp in $testProjects) { - $hasMsCodeCov = Select-String -Path $tp.FullName -Pattern 'Microsoft\.Testing\.Extensions\.CodeCoverage' -Quiet - if ($hasMsCodeCov) { - if ($dotnetTestMode -eq "native-MTP") { - dotnet test --project $tp.FullName --results-directory $rawDir --coverage --coverage-output-format cobertura --coverage-output $rawDir - } else { - dotnet test $tp.FullName --results-directory $rawDir -- --coverage --coverage-output-format cobertura --coverage-output $rawDir - } - } else { - dotnet test $tp.FullName ` - --collect:"XPlat Code Coverage" ` - --results-directory $rawDir ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Include="[*]*" ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Exclude="[*.Tests]*,[*.Test]*,[*Tests]*,[*Test]*,[*.Specs]*,[*.Testing]*" ` - -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.SkipAutoProps=true - } -} -``` - -Exit code handling: - -- **0** — all tests passed, coverage collected -- **1** — some tests failed (coverage still collected — proceed with a warning) -- **Other** — build failure; stop and report the error - -After the run, locate coverage files: - -```powershell -$coberturaFiles = Get-ChildItem -Path (Join-Path "" "raw") -Filter "coverage.cobertura.xml" -Recurse -Write-Host "COBERTURA_COUNT:$($coberturaFiles.Count)" -$coberturaFiles | ForEach-Object { Write-Host "COBERTURA:$($_.FullName)" } -$vsCovFiles = Get-ChildItem -Path (Join-Path "" "raw") -Filter "*.coverage" -Recurse -ErrorAction SilentlyContinue -if ($vsCovFiles) { Write-Host "VS_BINARY_COVERAGE:$($vsCovFiles.Count)" } -``` - -If `COBERTURA_COUNT` is 0: - -- If `VS_BINARY_COVERAGE` > 0: warn the user — *"Found .coverage files (VS binary format) but no Cobertura XML. These were likely produced by Visual Studio's built-in collector, which outputs a binary format by default. This skill needs Cobertura XML. Re-running with the detected provider configured for Cobertura output."* Then re-run the appropriate `dotnet test` command above (Coverlet or Microsoft CodeCoverage) with Cobertura format. -- If no `.coverage` files either: stop and report — *"Coverage files not generated. Ensure `dotnet test` completed successfully and check the build output for errors."* diff --git a/plugins/dotnet-test/skills/detect-static-dependencies/SKILL.md b/plugins/dotnet-test/skills/detect-static-dependencies/SKILL.md index 7d15517a9c..048fc1a21d 100644 --- a/plugins/dotnet-test/skills/detect-static-dependencies/SKILL.md +++ b/plugins/dotnet-test/skills/detect-static-dependencies/SKILL.md @@ -1,15 +1,15 @@ --- name: detect-static-dependencies description: > - Scan C# source files for hard-to-test static dependencies — DateTime.Now/UtcNow, - File.*, Directory.*, Environment.*, HttpClient, Console.*, Process.*, and other - untestable statics. Produces a ranked report of static call sites by frequency. - USE FOR: find untestable statics, scan for static dependencies, testability audit, - identify hard-to-mock code, find DateTime.Now usage, detect static coupling, - testability report, static analysis for testability. - DO NOT USE FOR: generating wrappers (use generate-testability-wrappers), - migrating code (use migrate-static-to-wrapper), general code review, - or finding statics that are already behind abstractions. + ACTIVATION PREREQUISITE: the request or discovered target must explicitly + identify C#, .NET, `.cs`, or `.csproj`; otherwise stay dormant without + invoking this skill. USE FOR: locating + System.DateTime.Now/UtcNow, System.IO.File/Directory, System.Environment, + HttpClient, Console, or Process usage in C#; auditing C# code for hard-to-test + framework dependencies; or verifying those C# calls are already abstracted. + DO NOT USE FOR: any target lacking the activation prerequisite; generating + wrappers (use generate-testability-wrappers); migrating code (use + migrate-static-to-wrapper); or general code review. license: MIT --- @@ -30,6 +30,28 @@ Scan a C# codebase for calls to hard-to-test static APIs and produce a ranked re - When the user provides a specific file or directory path, scan only that scope — do not expand to the entire solution unless asked. - The full structured report format in Step 4 is for comprehensive audit requests. For focused questions, return only the relevant subset (e.g., category summary + affected files for the requested category). +## Execution Contract + +- A relative path named in the prompt is enough to start. Discover it with the + available file-listing tools and scan it immediately; do not ask the user to + provide or re-upload files before both discovery and a content search fail. +- Start with a recursive, line-numbered content search over eligible `.cs` + files. Do not search only for the `static` keyword: ambient calls inside + LINQ expressions, lambdas, callbacks, and interpolated strings usually have + no `static` modifier. +- If a file-reading tool fails on a path that listing or search proved exists, + classify the failure before retrying. Fall back to another available + mechanism such as `rg -n`, grep, or a shell file reader only for confirmed + tool availability, transport, or path-normalization failures and only after + verifying the canonical path remains inside the workspace. Stop on + content-exclusion, permission/policy, workspace-boundary, or unknown failures. + Search output can seed the occurrence ledger; open only the surrounding code + needed to verify receiver provenance. +- Never stop after loading this skill or announcing a scan plan. Return the + completed audit in the same response. If every fallback genuinely fails, + report the verified partial findings and the exact limitation; do not invent + findings or replace the audit with a request to rerun. + ## When Not to Use - The user wants wrappers generated (hand off to `generate-testability-wrappers`) @@ -41,7 +63,7 @@ Scan a C# codebase for calls to hard-to-test static APIs and produce a ranked re | Input | Required | Description | |-------|----------|-------------| -| Target path | Yes | A file, directory, project (.csproj), or solution (.sln) to scan | +| Target path | No | A file, directory, project (.csproj), or solution (.sln) to scan. Defaults to the current workspace. | | Exclusion patterns | No | Glob patterns to skip (e.g., `**/obj/**`, `**/Migrations/**`) | | Category filter | No | Limit to specific categories: `time`, `filesystem`, `environment`, `network`, `console`, `process` | @@ -50,6 +72,10 @@ Scan a C# codebase for calls to hard-to-test static APIs and produce a ranked re ### Step 1: Determine scan scope Resolve the target to a set of `.cs` files: +- Treat a prompt-named workspace-relative path as the target; locate it rather + than asking the user for an absolute path. +- If omitted, scan every eligible `.cs` file under the current workspace; do not + pick one project and silently omit its siblings. - If a `.cs` file, scan that single file. - If a directory, scan all `.cs` files recursively (excluding `obj/`, `bin/`). - If a `.csproj`, find its directory and scan `.cs` files within. diff --git a/plugins/dotnet-test/skills/migrate-static-to-wrapper/SKILL.md b/plugins/dotnet-test/skills/migrate-static-to-wrapper/SKILL.md index af28d13583..cd96f86f73 100644 --- a/plugins/dotnet-test/skills/migrate-static-to-wrapper/SKILL.md +++ b/plugins/dotnet-test/skills/migrate-static-to-wrapper/SKILL.md @@ -4,10 +4,11 @@ description: > ALWAYS USE when asked to migrate, replace, or make testable existing C# static calls with a named wrapper or built-in abstraction: DateTime.UtcNow/Now or DateTimeOffset.UtcNow to TimeProvider/IClock, File.* to IFileSystem or an - existing store, and Environment.* to an existing reader. Covers scoped - files/projects, constructor injection, updating tests with fakes, "already - registered" abstractions, and static classes whose callers/signatures must stay - unchanged. Preserves DateTimeKind and call count. DO NOT USE for finding + existing store such as ITextFileStore, and Environment.* to an existing reader + such as IEnvironmentReader. Covers scoped files/projects, constructor + injection, replacing temp-file or process-environment tests with fakes, + "already registered" abstractions, and static classes whose callers/signatures + must stay unchanged. Preserves DateTimeKind and call count. DO NOT USE for finding statics (detect-static-dependencies), choosing/designing a new wrapper (generate-testability-wrappers), behavior tests with no chosen seam (testability-obstacle), or test-framework migration. @@ -46,9 +47,9 @@ Perform mechanical, codemod-style replacement of static dependency call sites wi | Input | Required | Description | |-------|----------|-------------| -| Static pattern | Yes | What to replace (e.g., `DateTime.UtcNow`, `File.ReadAllText`) | -| Replacement abstraction | Yes | What to use instead (e.g., `TimeProvider`, `IFileSystem`) | -| Scope | Yes | File path, project (.csproj), namespace, or directory to migrate | +| Static pattern | No | Infer from the request and discovered call sites (e.g., `DateTime.UtcNow`, `File.ReadAllText`) | +| Replacement abstraction | No | Infer from the request and existing project abstractions; stop only when no named/existing abstraction is available | +| Scope | No | Infer from the requested file/project/namespace, otherwise discover the narrowest relevant workspace scope | | Injection strategy | No | `constructor` (default), `primary-constructor`, or `ambient` | ## Workflow @@ -63,6 +64,23 @@ Perform mechanical, codemod-style replacement of static dependency call sites wi even when sharing a captured timestamp looks cleaner. - **The requested scope is exhaustive and exclusive.** Replace every named call in scope and no adjacent member or file. +- **Repository-backed requests require repository work.** Start by discovering + files from the current workspace. Do not claim the repository is unavailable + or ask the user for a path or file contents until workspace-relative discovery + found no target. Do not say work was implemented unless the diff proves it. +- **Discovered workspace files must be completed in this turn when permitted.** + Use a host-native shell reader (`sed`/`cat` or `Get-Content`) only after a + confirmed reader availability, transport, or path-normalization failure and + only after verifying the canonical path remains inside the current workspace. + Stop on content-exclusion, permission/policy, workspace-boundary, or unknown + read failures. Use a shell edit fallback only for a confirmed editor + availability, transport, or path-normalization failure, never for a stale + context, concurrent change, permission/policy denial, or path-boundary error. + Before fallback, resolve the canonical path inside the current workspace, + freshly read the file, and require an anchored replacement with the expected + old text and exact match count; abort if either changed. Then re-open the file, + inspect the diff, and validate. Do not ask the user to paste a readable + discovered file or report a proposed patch as completed work. ### Step 1: Verify prerequisites @@ -74,13 +92,15 @@ Before modifying any code: 3. **Identify all files in scope**: List the `.cs` files that will be modified. Exclude test projects, `obj/`, `bin/`, and generated code. -4. **Count every in-scope occurrence before editing**: Search the exact member - named by the user and record its file/line inventory. Do not infer the count - from a partial read or from how many methods were initially noticed. +4. **Lock and count the member set before editing**: Use the exact member named + by the user, or infer the smallest unambiguous set from the request and + discovered call sites. Record that set, then search every member and capture + the file/line inventory. Do not change the set mid-edit or infer counts from + a partial read. ### Step 2: Plan the migration for each file -**Migrate exactly what was asked — nothing adjacent.** If the user named a member (`DateTime.UtcNow`), migrate only that member and leave siblings such as `DateTime.Now` untouched. If the user named files, do not touch other files. Never migrate a call site whose comment or name marks it as deliberate (e.g. `// intentional local time`). List everything you deliberately left alone under "Remaining (out of scope)" so the user can ask for it in a follow-up; suggesting is fine, silently widening the scope is not. +**Migrate exactly what was asked — nothing adjacent.** If the user named a member (`DateTime.UtcNow`), migrate only that member and leave siblings such as `DateTime.Now` untouched. If the user named files, do not touch other files. Preserve a call site whose comment or name marks it as deliberate (e.g. `// intentional local time`) unless the user explicitly names that site and requests a semantics-preserving migration. List everything you deliberately left alone under "Remaining (out of scope)" so the user can ask for it in a follow-up; suggesting is fine, silently widening the scope is not. For each file containing the static pattern, determine: @@ -286,7 +306,8 @@ verified unchanged, and the targeted build/test result: - [ ] A before/after exact-member search proves the in-scope occurrence count reached zero - [ ] No call site outside the requested member/file scope was modified -- [ ] Call sites documented as intentional (e.g. local time) were left untouched and reported +- [ ] Call sites documented as intentional were left untouched and reported unless + the user explicitly named them for semantics-preserving migration - [ ] Constructor injection added to all affected classes - [ ] Field naming follows existing class conventions - [ ] Required `using` directives added diff --git a/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md b/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md index 25946e50c4..1cb0594b14 100644 --- a/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md +++ b/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md @@ -20,7 +20,9 @@ license: MIT Quick, pragmatic analysis of test code in any supported language for anti-patterns and quality issues that undermine test reliability, maintainability, and diagnostic value. -> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase (e.g., `extensions/dotnet.md`, `extensions/python.md`, `extensions/typescript.md`, `extensions/go.md`). The extension file tells you which sleep / time / random / skip / setup-teardown / mystery-guest APIs to look for in that language. +> **Language-specific guidance**: Try `test-analysis-extensions` once. If it is +> unavailable, continue immediately with this skill's built-in framework rules; +> never block the audit on the helper. ## When to Use @@ -47,7 +49,7 @@ Quick, pragmatic analysis of test code in any supported language for anti-patter | Input | Required | Description | |-------|----------|-------------| -| Test code | Yes | One or more test files or classes to analyze | +| Test scope | No | Test files, classes, directory, or project to analyze. Discover from the current workspace when omitted. | | Production code | No | The code under test, for context on what tests should verify | | Specific concern | No | A focused area like "flakiness" or "naming" to narrow the review | @@ -55,17 +57,39 @@ Quick, pragmatic analysis of test code in any supported language for anti-patter ### Step 1: Detect language and load extension -Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The extension file documents framework-specific anti-pattern markers — what counts as a sleep/wait, a test marker, a skip, a setup/teardown, a shared-state hot spot, and an integration boundary — so this skill stays language-neutral. +Resolve the named test path from the current workspace before asking for input. +When no path is supplied, discover test files under the current directory using +the repository manifests and conventional test markers. The skill context's +`Base directory` is documentation storage, not the user's workspace; never +resolve target files relative to it. + +If one reader says a path is missing but a workspace glob/search finds it, +normalize that exact path and retry. Use a shell text reader (`sed`/`cat` on +Unix, `Get-Content` on PowerShell) only for a confirmed reader availability, +transport, or path-normalization failure and only after verifying the canonical +path remains inside the current workspace. Stop on content-exclusion, +permission/policy, workspace-boundary, or unknown failures. Audit any discovered +file that a permitted reader can access; never ask the user to paste it. If +every permitted reader fails, report the exact blocker without bypassing +security boundaries. + +Identify the language and framework. Try the matching +`test-analysis-extensions` guidance once; if unavailable, use the catalog below. ### Step 2: Gather the test code -Read the test files the user wants reviewed. If the user points to a directory or project, scan for all test files using the discovery markers in the loaded language extension file (e.g., `[TestClass]`/`[Fact]`/`[Test]` for .NET, `test_*.py` / `def test_*` for pytest, `*.test.ts` / `it()` for Jest, `*Test.java` / `@Test` for JUnit, `*_test.go` / `func TestXxx` for Go, `*_spec.rb` for RSpec, `#[test]` for Rust, `*.Tests.ps1` / `Describe` for Pester, `TEST(...)` for GoogleTest, `TEST_CASE(...)` for Catch2/doctest). +Read every test file in the resolved scope. Use extension discovery markers +when loaded; otherwise use the built-in markers in this skill (attributes such +as `[TestClass]`/`[Fact]`/`[Test]`, `test_*.py`, `*.test.*`, `*_test.go`, +`*_spec.rb`, `#[test]`, `*.Tests.ps1`, `TEST(...)`, and `TEST_CASE(...)`). If production code is available, read it too -- this is critical for detecting tests that are coupled to implementation details rather than behavior. ### Step 3: Scan for anti-patterns -Check each test file against the anti-pattern catalog below. Report findings grouped by severity. The examples are .NET-centric but the patterns generalize — use the loaded language extension file to map each pattern to the framework you are auditing. +Check each test file against the anti-pattern catalog below. Report findings +grouped by severity. Use extension mappings when loaded; otherwise use the +cross-framework examples in the catalog. Before drafting the report, make a private completeness ledger with one row for every test method and every class-level fixture/resource. Record its oracle (or @@ -109,7 +133,7 @@ sound. In particular: | Anti-Pattern | What to Look For | |---|---| -| **Poor naming** | Test names like `Test1`, `TestMethod`, `test`, names that don't describe the scenario or expected outcome. Good naming differs by language convention — see the loaded language extension file (e.g., `Add_NegativeNumber_ThrowsArgumentException` for .NET, `test_add_negative_number_raises_value_error` for pytest, `addNegativeNumber_throwsArgumentException` for Java, `'adds negative number throws'` for Jest descriptions, `TestAdd_NegativeNumber_ReturnsError` for Go). | +| **Poor naming** | Test names like `Test1`, `TestMethod`, or `test` that don't describe the scenario or outcome. Use the loaded extension when available; otherwise follow the existing naming convention in the same suite. | | **Magic values** | Unexplained numbers or strings in arrange/assert: `Assert.AreEqual(42, result)` / `assert result == 42` / `expect(result).toBe(42)` -- what does 42 mean? | | **Duplicate tests** | Three or more test methods with near-identical bodies that differ only in a single input value. Should be parametrized: `[DataRow]`/`[Theory]`/`[TestCase]` (.NET), `@pytest.mark.parametrize` (pytest), `test.each` / `it.each` (Jest/Vitest), `@ParameterizedTest` + `@ValueSource` (JUnit 5), `@DataProvider` (TestNG), Go table-driven tests, `where` / shared examples (RSpec), `#[rstest]` (Rust), `@ParameterizedTest` + `@MethodSource` (Kotlin), `-ForEach` / `-TestCases` (Pester), `INSTANTIATE_TEST_SUITE_P` (GoogleTest), `SECTION` / `GENERATE` (Catch2), `TEST_CASE_TEMPLATE` (doctest). For a detailed duplication analysis in .NET, use `exp-test-maintainability`. Note: Two tests covering distinct boundary conditions (e.g., zero vs. negative) are NOT duplicates -- separate tests for different edge cases provide clearer failure diagnostics and are a valid practice. | | **Giant tests** | Test methods exceeding ~30 lines or testing multiple behaviors at once. Hard to diagnose when they fail. | @@ -157,8 +181,12 @@ Before reporting, re-check each finding against these severity rules: - Explicit per-test setup instead of `[TestInitialize]` / `beforeEach` (this *improves* isolation). - Tests that are short and clear but could theoretically be consolidated. - Round-trip or serialization equality with non-trivial input. It is valid - metamorphic evidence; suggest an independent representation assertion when - two implementations could share the same bug. + metamorphic evidence, not a self-comparison; still recommend one independent + representation when producer and consumer could share a defect. + - A transformation tested only with an already-transformed input. Keep it out + of the tautology count, but report the weak oracle when removing the + transformation would still pass. Use an input that must change and pin its + independently expected output. - Clone value equality. Keep it, and add distinct-reference or mutation- independence evidence when the contract promises a deep copy. - A validator or accessor returning the original value when pass-through is the diff --git a/plugins/dotnet-test/skills/test-tagging/SKILL.md b/plugins/dotnet-test/skills/test-tagging/SKILL.md index 7919922ebf..56518eeca5 100644 --- a/plugins/dotnet-test/skills/test-tagging/SKILL.md +++ b/plugins/dotnet-test/skills/test-tagging/SKILL.md @@ -2,12 +2,13 @@ name: test-tagging description: > Classifies existing tests by standard traits and reports their distribution. - MUST USE to tag all tests with category attributes, categorize/tag/label each + USE FOR: tagging all tests with category attributes, categorizing/tagging/ + labeling each test, compare happy vs error paths, audit the test mix, describe coverage shape by test type, or tag then verify the project builds. Read bodies when names - mislead. Apply canonical attributes; otherwise report only. DO NOT USE for - test-quality audits, executed coverage or CRAP, behavioral gaps, writing - tests, or migration. + mislead. Apply canonical attributes; otherwise report only. DO NOT USE FOR: + requests owned by test-anti-patterns, coverage-analysis, crap-score, + test-gap-analysis, code-testing-agent, or migration skills. license: MIT --- @@ -15,7 +16,9 @@ license: MIT Analyze an existing test suite in any supported language and apply a standardized set of trait tags to each test method, giving teams visibility into their test distribution (positive vs. negative, critical-path coverage, smoke tests, etc.). -> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase. The extension file documents framework-specific tag attributes and a "tag-support capability" (auto-edit, report-only, or convention-based) that drives whether this skill modifies source files or only emits a report. +> **Language-specific guidance**: Try `test-analysis-extensions` once. If it is +> unavailable, continue immediately with the built-in framework table below; +> never block tagging on the helper. ## When to Use @@ -38,7 +41,7 @@ Analyze an existing test suite in any supported language and apply a standardize | Input | Required | Description | |-------|----------|-------------| -| Test project or files | Yes | Path to the test project, folder, or specific test files to analyze | +| Test project or files | No | Path to the test project, folder, or specific test files. Discover from the current workspace when omitted. | | Scope | No | `tag` (apply canonical attributes, or a confirmed project convention), `audit` (report only), or `both` (default: `both`). Frameworks declared `report-only` always emit a report; `convention-based` frameworks edit only after the user confirms the convention. | | Framework | No | Auto-detected. Override when detection fails. | @@ -70,7 +73,35 @@ A single test may have **multiple traits** (e.g., both `negative` and `boundary` ### Step 1: Detect the language, framework, and tagging capability -Identify the codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The extension file declares a **tag-support capability** for each framework: +Resolve the requested test scope from the current workspace before asking for a +path. The skill context's `Base directory` contains these instructions, not the +user's repository. Always inspect the current working directory before claiming +that repository files are unavailable. If the prompt's relative path is absent, +search the workspace for the named project/file and retry the exact result. A +successful search proves that the target is present; if the normal reader then +reports that same path missing, treat the contradiction as a reader +path-normalization or transport failure rather than asking the user for files. +Use a shell text reader (`sed`/`cat` on Unix, +`Get-Content` on PowerShell) only for a confirmed reader availability, +transport, or path-normalization failure and only after verifying the canonical +path remains inside the current workspace. Stop on content-exclusion, +permission/policy, workspace-boundary, or unknown read failures. Never ask the +user for a path or file contents after a workspace search found a readable +target. + +For an `auto-edit` framework, a failed patch/editor call is not a stopping +condition only when the failure is confirmed tool availability, transport, or +path normalization. Do not bypass stale-context, concurrent-change, +permission/policy, or path-boundary errors. Before a shell fallback, resolve +the canonical path inside the current workspace, freshly read the file, and use +an anchored transformation that aborts unless the expected old text and exact +match count are unchanged. Then re-open the complete file, inspect the diff, +and run Step 6 validation. Do not report proposed attributes as completion when +the user asked to apply them. + +Identify the language and framework. Try the matching +`test-analysis-extensions` guidance once. If unavailable, classify capability +from the built-in rules below: - **`auto-edit`** — framework has canonical tag syntax this skill can safely insert (.NET `[TestCategory]` / `[Trait]` / `[Category]` / `[Property]`, pytest `@pytest.mark.`, JUnit 5 `@Tag("...")`, TestNG `groups = {"..."}`, RSpec metadata `it "..." , :tag => true`, Pester `-Tag '...'`, Kotest `@Tags(...)`, Swift Testing `@Tag(.tagName)`, Catch2 `[tag]`, doctest `* doctest::test_suite("tag")` decorator). - **`report-only`** — framework has no canonical, agreed-upon tag attribute; report tags in a Markdown table only and do not edit source (Go standard `testing` without build-tag conventions, Jest/Vitest without consistent describe-prefix convention, Rust without project-specific cfg conventions, XCTest without a test plan, GoogleTest without test-name prefix conventions, Mocha without describe-prefix conventions). @@ -80,7 +111,8 @@ Capture the capability before Step 4. ### Step 2: Scan existing traits -Check which tests already have trait attributes. Use the loaded language extension as the source of truth — examples: +Check which tests already have trait attributes. Use the extension when loaded; +otherwise use this built-in table as the source of truth: | Framework | Existing Attribute | Example | |-----------|--------------------|---------| @@ -139,7 +171,9 @@ expand into the behavioral-gap audit owned by `test-gap-analysis`. ### Step 4: Apply trait attributes (or report only) -**If the loaded language extension declares `auto-edit` for the framework**, add the appropriate attribute to each test method. Place trait attributes adjacent to the existing test attribute. Examples: +**If the resolved capability is `auto-edit`**, add the appropriate attribute to +each test method. Place trait attributes adjacent to the existing test +attribute. Examples: Apply traits at the individual test-method/case level. Do not substitute one class-level category for method-level classification: different methods usually @@ -225,9 +259,17 @@ func parseNullInputThrows() throws { ... } TEST_CASE("Parse null input throws", "[negative][boundary]") { ... } ``` -**If the loaded language extension declares `report-only` for the framework** (Go standard `testing`, plain Jest/Vitest without convention, Rust without project-specific cfg, plain XCTest, plain GoogleTest, plain Mocha), do NOT modify source files. Instead emit a concise mapping from each test to its suggested tags. Recommend a project-wide convention only when the user asks how to persist or filter those tags; an analysis-only request should report and stop. +**If the resolved capability is `report-only`** (Go standard `testing`, plain +Jest/Vitest without convention, Rust without project-specific cfg, plain +XCTest, plain GoogleTest, plain Mocha), do NOT modify source files. Instead emit +a concise mapping from each test to its suggested tags. Recommend a project-wide +convention only when the user asks how to persist or filter those tags; an +analysis-only request should report and stop. -**If the loaded language extension declares `convention-based`** (e.g., Go `//go:build integration`, `*_integration_test.go`, GoogleTest `INTEGRATION_*` prefix), only emit canonical edits when the user has confirmed the project's convention. Otherwise treat as `report-only`. +**If the resolved capability is `convention-based`** (e.g., Go +`//go:build integration`, `*_integration_test.go`, GoogleTest `INTEGRATION_*` +prefix), only emit canonical edits when the user has confirmed the project's +convention. Otherwise treat as `report-only`. ### Step 5: Generate trait summary @@ -286,7 +328,8 @@ never publish a successful distribution handoff for uncompiled edits. - [ ] No invented trait values outside the taxonomy table - [ ] Existing trait attributes were preserved, not duplicated - [ ] The trait summary table was generated -- [ ] For `auto-edit` frameworks, the project still builds / tests still discover after changes (`dotnet build` / `pytest --collect-only` / `mvn test-compile` / `go vet ./...` / `cargo check --tests` / `npm run test:list` / `Invoke-Pester -PassThru -Skip` / equivalent) +- [ ] For `auto-edit` frameworks, the project still builds / tests still discover without executing unrequested tests (`dotnet build` plus list mode / `pytest --collect-only` / `mvn test-compile` / `go vet ./...` / `cargo check --tests` / `npm run test:list` / equivalent) +- [ ] The final summary cites successful validation commands and the discovered test count when a discovery command is available - [ ] For `report-only` frameworks, no source files were modified - [ ] For `convention-based` frameworks, edits were applied ONLY when a project convention was confirmed @@ -296,9 +339,9 @@ never publish a successful distribution handoff for uncompiled edits. |---------|----------| | Guessing traits without reading the test body | Always read assertions and setup to classify accurately | | Tagging a test only as `boundary` without `positive`/`negative` | Every test should also be `positive` or `negative` -- `boundary` is additive | -| Using the wrong attribute syntax for the detected framework | Match the attribute style to the loaded language extension (don't put `[TestCategory]` in an xUnit project or `@pytest.mark.x` in a unittest test) | +| Using the wrong attribute syntax for the detected framework | Match the loaded extension or built-in table (don't put `[TestCategory]` in xUnit or `@pytest.mark.x` in unittest) | | Duplicating an existing category attribute | Check for pre-existing traits in Step 2 before adding | | Over-tagging as `critical-path` | Reserve for tests on primary public entry points, not every helper | | Editing Go / plain Jest / plain Rust / plain XCTest / plain GoogleTest source | These are `report-only` by default — emit a Markdown table instead. Only edit if the user confirms a project-wide convention (build tag, file suffix, describe-prefix, test-plan grouping). | | Inventing tag prefixes for convention-based frameworks | Confirm the project's existing convention before adopting one — don't guess between `_integration_test.go`, `//go:build integration`, or `IntegrationTest` prefix | -| Missing language-specific concurrency / async primitives | Each language has its own primitives — read the loaded language extension and the Trait Taxonomy concurrency row before classifying as `concurrency` | +| Missing language-specific concurrency / async primitives | Use the loaded extension when available; otherwise use the Trait Taxonomy concurrency row | diff --git a/tests/dotnet-test/coverage-analysis/eval.yaml b/tests/dotnet-test/coverage-analysis/eval.yaml index 3bf479b0d4..4ac242d834 100644 --- a/tests/dotnet-test/coverage-analysis/eval.yaml +++ b/tests/dotnet-test/coverage-analysis/eval.yaml @@ -52,37 +52,6 @@ stimuli: - Uses a repository-owned coverage workflow if one exists; otherwise requests Cobertura XML and reports that CRAP cannot be calculated without real coverage - Does not create a temporary SDK-style wrapper/project or report substitute coverage from another assembly - Does not claim project-wide coverage or CRAP results when no report was produced - - name: Keep MTP coverage arguments behind the separator in SDK 10 VSTest mode - prompt: | - Give me the exact `dotnet test` command to collect Microsoft CodeCoverage - as Cobertura from tests/Contoso.Tests.csproj. The repository uses SDK - 10.0.100, but global.json explicitly sets test.runner to VSTest. The test - project is an MTP application with the VSTest-to-MTP bridge enabled and - references Microsoft.Testing.Extensions.CodeCoverage. Put results under - TestResults/coverage/raw. Do not run the command or modify files. - expect_activation: false - graders: - - type: output-matches - config: - pattern: (?i)VSTest mode - - type: output-matches - config: - pattern: (?ims)dotnet\s+test\s+"?tests[\\/]Contoso\.Tests\.csproj"?.*?\s--\s+--coverage\b - - type: output-not-matches - config: - pattern: (?im)^\s*(?:[$>]\s*)?dotnet\s+test\s+--project\b - - type: prompt - rubric: - - Selected syntax from the configured VSTest runner mode rather than the SDK major version alone - - Used the project path positionally instead of using the native-MTP --project selector - - Kept dotnet test options such as --results-directory before the separator - - Passed --coverage, --coverage-output-format cobertura, and --coverage-output after the separator - - Did not attempt project-wide coverage or CRAP analysis - constraints: - reject_tools: - - bash - - edit - - create - name: Project-wide coverage analysis with existing Cobertura data prompt: Run coverage analysis on this project. Coverage data is already in TestResults/coverage.cobertura.xml. environment: @@ -331,8 +300,8 @@ stimuli: - name: Stay dormant for behavioral gap analysis prompt: | My existing tests execute the null-check in CheckoutService.Submit. - Determine whether they would still pass if that guard were removed, and - add a focused edge-case test if the bug would survive. + Would they still pass if that guard were removed? Explain what production + and test evidence you would need to decide. Do not modify files. expect_activation: false graders: - type: prompt diff --git a/tests/dotnet-test/detect-static-dependencies/eval.yaml b/tests/dotnet-test/detect-static-dependencies/eval.yaml index 54be7cd630..0a296b181a 100644 --- a/tests/dotnet-test/detect-static-dependencies/eval.yaml +++ b/tests/dotnet-test/detect-static-dependencies/eval.yaml @@ -75,10 +75,12 @@ stimuli: - Recommended TimeProvider or a similar abstraction as the replacement for .NET 8+ - Provided a migration plan or prioritized steps for replacing the time statics - - name: Decline scan for non-C# project + - name: Stay dormant for Python timezone review prompt: > - I have a Python Django project at my_app/ that has lots of - datetime.now() calls. Can you scan it for untestable statics? + Review my Python Django view at my_app/views.py. Is its timestamp + timezone-aware, and what Django API should it use instead? Do not modify + files. + expect_activation: false environment: files: - src: ./fixtures/decline-non-csharp/views.py @@ -86,12 +88,15 @@ stimuli: graders: - type: output-matches config: - pattern: (Python|not.*(C#|\.NET)|different|unsupported) + pattern: datetime\.now + - type: output-matches + config: + pattern: (?i)(timezone\.now|django\.utils\.timezone) - type: prompt rubric: - - Recognized that the project is Python, not C#/.NET - - Did not attempt to scan for .NET static patterns - - Suggested an alternative approach or clarified it only supports .NET + - Identified that the direct datetime.now call produces a naive timestamp + - Recommended Django's timezone.now API for timezone-aware behavior + - Did not attempt a C#/.NET static-dependency audit - name: Verify structured report includes file count, categories, and top patterns prompt: > diff --git a/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/Services/OrderProcessor.cs b/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/Services/OrderProcessor.cs index 72dcd52b3a..b40468307a 100644 --- a/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/Services/OrderProcessor.cs +++ b/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/Services/OrderProcessor.cs @@ -1,4 +1,5 @@ using System.Diagnostics; +using Microsoft.Extensions.Logging; namespace StaticHeavy.Services; diff --git a/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/StaticHeavy.csproj b/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/StaticHeavy.csproj index bad583f080..986a1bf20a 100644 --- a/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/StaticHeavy.csproj +++ b/tests/dotnet-test/detect-static-dependencies/fixtures/static-heavy-project/StaticHeavy.csproj @@ -5,4 +5,7 @@ enable enable + + + diff --git a/tests/dotnet-test/migrate-static-to-wrapper/eval.yaml b/tests/dotnet-test/migrate-static-to-wrapper/eval.yaml index eda35b6d62..c91ed62f4f 100644 --- a/tests/dotnet-test/migrate-static-to-wrapper/eval.yaml +++ b/tests/dotnet-test/migrate-static-to-wrapper/eval.yaml @@ -2,8 +2,11 @@ name: migrate-static-to-wrapper executionShard: c description: Evaluates the dotnet-test/migrate-static-to-wrapper skill type: capability +# The prior GPT-family run produced 4W/5T/0L: positive but one discordant +# stimulus short of the sign-test floor. This predeclared scenario adds distinct +# local-calendar semantics breadth instead of repeating a UTC migration. config: - timeout: 3m + timeout: 10m stimuli: - name: Migrate DateTime.UtcNow to TimeProvider in a service class prompt: > @@ -110,6 +113,58 @@ stimuli: - Restricted the change to the DateTime.UtcNow call sites the user named, leaving the intentional local-time call untouched + - name: Preserve local calendar semantics when migrating DateTime.Now + prompt: > + CouponService.IsRedeemableToday intentionally compares against the local + calendar day with DateTime.Now. TimeProvider is already registered in DI. + Migrate only that DateTime.Now read to TimeProvider without changing its + DateTime type or local-calendar date semantics. Leave every + DateTime.UtcNow call in the file untouched. + environment: + files: + - src: ./fixtures/ready-to-migrate + dest: ReadyToMigrate + graders: + - type: file-contains + config: + path: "**/CouponService.cs" + value: GetLocalNow() + - type: file-contains + config: + path: "**/CouponService.cs" + value: LocalDateTime.Date + - type: file-contains + config: + path: "**/CouponService.cs" + value: DateTime.UtcNow + - type: run-command + config: + command: >- + python3 -c 'import pathlib,re; + source=pathlib.Path("ReadyToMigrate/Services/CouponService.cs").read_text(); + code=re.sub(r"@\"(?:\"\"|[^\"])*\"|\"(?:\\.|[^\"\\])*\"|//[^\r\n]*|/\*.*?\*/", + "", source, flags=re.MULTILINE|re.DOTALL); + assert "DateTime.Now" not in code; + assert code.count("DateTime.UtcNow") == 4' + expected_exit_code: 0 + timeout: 1m + - type: run-command + config: + command: dotnet build ReadyToMigrate/ReadyToMigrate.csproj + expected_exit_code: 0 + timeout: 3m + - type: exit-success + - type: output-matches + config: + pattern: (?i)(build succeeded|built successfully|0 errors?) + - type: prompt + rubric: + - Replaced only the DateTime.Now read in IsRedeemableToday with TimeProvider + - Used GetLocalNow().LocalDateTime.Date so the replacement preserves the local calendar date rather than switching to UTC + - Left all four DateTime.UtcNow reads untouched because they were outside the requested scope + - Injected the already-registered TimeProvider without changing Coupon date property types or unrelated files + - Built the affected project successfully and reported the observed result + - name: Preserve DateTimeOffset values during TimeProvider migration prompt: > SubscriptionWindow.cs uses DateTimeOffset.UtcNow and stores diff --git a/tests/dotnet-test/test-tagging/eval.yaml b/tests/dotnet-test/test-tagging/eval.yaml index 7be16af769..79aa3f66d0 100644 --- a/tests/dotnet-test/test-tagging/eval.yaml +++ b/tests/dotnet-test/test-tagging/eval.yaml @@ -215,6 +215,26 @@ stimuli: config: path: TaskManager.Tests/TaskServiceTests.cs value: "[Category(" + - type: run-command + config: + command: dotnet build TaskManager.Tests/TaskManager.Tests.csproj + expected_exit_code: 0 + timeout: 3m + - type: run-command + config: + command: dotnet test TaskManager.Tests/TaskManager.Tests.csproj --no-build --list-tests + expected_exit_code: 0 + stdout_matches: DeleteTask_Success + timeout: 3m + - type: output-matches + config: + pattern: (?i)dotnet build + - type: output-matches + config: + pattern: (?i)(--list-tests|list tests|test discovery) + - type: output-matches + config: + pattern: (?i)(build succeeded|0 errors?|[1-9]\d* tests? (?:listed|discovered)) - type: prompt rubric: - Classified DeleteTask_Success as negative (and/or security) because the body asserts UnauthorizedAccessException, @@ -228,6 +248,9 @@ stimuli: - Classified DeleteTask_Validation as positive and security because the body verifies admin authorization succeeds - Used NUnit [Category("...")] attribute syntax, not MSTest or xUnit syntax - Every test method received at least one trait attribute + - Built the NUnit test project after editing and, when supported, used non-executing discovery rather than + running arbitrary tests + - Reported successful validation and any discovered test count without claiming success after a failed command - name: Tag MSTest tests and verify the project still builds prompt: I need you to tag all the tests in my OrderService.Tests/ MSTest project with appropriate category attributes. After adding the tags, please verify the project still compiles by running dotnet build.