Skip to content

Add VSTest to MTP migration skill - #346

Closed
Evangelink wants to merge 6 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/migrate-mtp
Closed

Add VSTest to MTP migration skill#346
Evangelink wants to merge 6 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/migrate-mtp

Conversation

@Evangelink

Copy link
Copy Markdown
Member

New migrate-vstest-to-mtp skill in the dotnet-test plugin covering:

  • Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net)
  • Directory.Build.props centralization of OutputType, runner properties
  • MSTest.Sdk UseVSTest removal
  • NUnit3TestAdapter 5.0.0+ upgrade requirement
  • xunit v2 to xunit.v3 upgrade requirement
  • dotnet test CLI argument translation (--logger to --report-trx, --collect to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump)
  • xUnit.net --filter syntax migration to --filter-class/--filter-trait/ --filter-query
  • global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport VSTest-bridge mode (.NET 9 and earlier)
  • Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration
  • MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage)
  • Exit code 8 behavioral difference (--ignore-exit-code 8, TESTINGPLATFORM_EXITCODE_IGNORE)

Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture directories, plugin.json, marketplace.json entries, CODEOWNERS, and README.md updates.

New migrate-vstest-to-mtp skill in the dotnet-test plugin covering:

- Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net)
- Directory.Build.props centralization of OutputType, runner properties
- MSTest.Sdk UseVSTest removal
- NUnit3TestAdapter 5.0.0+ upgrade requirement
- xunit v2 to xunit.v3 upgrade requirement
- dotnet test CLI argument translation (--logger to --report-trx, --collect
  to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump)
- xUnit.net --filter syntax migration to --filter-class/--filter-trait/
  --filter-query
- global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport
  VSTest-bridge mode (.NET 9 and earlier)
- Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration
- MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage)
- Exit code 8 behavioral difference (--ignore-exit-code 8,
  TESTINGPLATFORM_EXITCODE_IGNORE)

Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture
directories, plugin.json, marketplace.json entries, CODEOWNERS, and
README.md updates.
@JanKrivanek

Copy link
Copy Markdown
Member

/evaluate

@github-actions

Copy link
Copy Markdown
Contributor

❌ Evaluation did not complete (upstream job failed or was skipped). View workflow run

@JanKrivanek

Copy link
Copy Markdown
Member

/evaluate

@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
migrate-vstest-to-mtp Migrate MSTest project from VSTest to Microsoft.Testing.Platform 5.0/5 → 5.0/5 ✅ migrate-vstest-to-mtp; tools: report_intent, skill, view ✅ 0.07
migrate-vstest-to-mtp Migrate NUnit project from VSTest to Microsoft.Testing.Platform 1.7/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07
migrate-vstest-to-mtp Migrate xUnit.net v2 project from VSTest to Microsoft.Testing.Platform 1.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill, report_intent, view ✅ 0.07
migrate-vstest-to-mtp Update Azure DevOps pipeline from VSTest task to MTP 3.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07
migrate-vstest-to-mtp Migrate MSTest.Sdk project that explicitly uses VSTest 3.3/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07
migrate-vstest-to-mtp Translate dotnet test VSTest arguments to MTP equivalents 4.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07
migrate-vstest-to-mtp Handle exit code 8 when migrating from VSTest to MTP 3.0/5 → 4.3/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07
migrate-vstest-to-mtp Configure dotnet test MTP mode on .NET 10 SDK 2.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07
migrate-vstest-to-mtp Migrate xUnit.net VSTest filter syntax to MTP 1.0/5 → 2.3/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07
migrate-vstest-to-mtp Full VSTest to MTP migration plan for MSTest solution 3.3/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.07

Model: claude-opus-4.6 | Judge: claude-opus-4.6

Full results

@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality (Isolated) Quality (Plugin) Skills Loaded Overfit Verdict
migrate-vstest-to-mtp Migrate MSTest project from VSTest to Microsoft.Testing.Platform 4.0/5 → 5.0/5 🟢 4.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill, report_intent / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Migrate NUnit project from VSTest to Microsoft.Testing.Platform 2.0/5 → 5.0/5 🟢 2.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Migrate xUnit.net v2 project from VSTest to Microsoft.Testing.Platform 1.0/5 → 5.0/5 🟢 1.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Update Azure DevOps pipeline from VSTest task to MTP 3.0/5 → 5.0/5 🟢 3.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Migrate MSTest.Sdk project that explicitly uses VSTest 3.7/5 → 5.0/5 🟢 3.7/5 → 4.7/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Translate dotnet test VSTest arguments to MTP equivalents 5.0/5 → 5.0/5 5.0/5 → 5.0/5 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill, report_intent ✅ 0.06
migrate-vstest-to-mtp Handle exit code 8 when migrating from VSTest to MTP 3.0/5 → 4.7/5 🟢 3.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Configure dotnet test MTP mode on .NET 10 SDK 2.0/5 → 5.0/5 🟢 2.0/5 → 4.7/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Migrate xUnit.net VSTest filter syntax to MTP 1.0/5 → 2.7/5 🟢 1.0/5 → 3.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06
migrate-vstest-to-mtp Full VSTest to MTP migration plan for MSTest solution 3.3/5 → 5.0/5 🟢 3.3/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.06

Model: claude-opus-4.6 | Judge: claude-opus-4.6

Full results

@Evangelink
Evangelink marked this pull request as ready for review March 14, 2026 13:20
Copilot AI review requested due to automatic review settings March 14, 2026 13:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new migrate-vstest-to-mtp skill to the dotnet-test plugin, along with evaluation coverage and fixtures, to guide migrating .NET test projects from VSTest to Microsoft.Testing.Platform (MTP) and updating CI usage accordingly.

Changes:

  • Introduces migrate-vstest-to-mtp skill documentation covering runner enablement, config centralization, CLI arg translation, and CI guidance.
  • Adds a new eval suite (eval.yaml) plus multiple fixture projects/pipeline samples for MSTest, NUnit, and xUnit scenarios.
  • Updates plugin discovery/metadata (plugin description + marketplace + repo README listing).

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/xunit-vstest/TestProject.csproj New xUnit v2 VSTest fixture project for migration scenarios.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/xunit-vstest/CalculatorTests.cs New xUnit fixture tests used by eval scenarios.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/nunit-vstest/TestProject.csproj New NUnit VSTest fixture project for migration scenarios.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/nunit-vstest/CalculatorTests.cs New NUnit fixture tests used by eval scenarios.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/mstest-vstest/TestProject.csproj New MSTest VSTest fixture project for migration scenarios.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/mstest-vstest/CalculatorTests.cs New MSTest fixture tests used by eval scenarios.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/mstest-sdk-vstest/TestProject.csproj New MSTest.Sdk fixture demonstrating <UseVSTest>true</UseVSTest> opt-out.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/mstest-sdk-vstest/AppTests.cs New MSTest.Sdk fixture tests used by eval scenarios.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/mstest-ci-pipeline/azure-pipelines.yml New Azure DevOps pipeline fixture demonstrating VSTest@3 usage to be migrated.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/mstest-ci-pipeline/TestProject.csproj New MSTest project fixture paired with the pipeline migration scenario.
tests/dotnet-test/migrate-vstest-to-mtp/fixtures/mstest-ci-pipeline/ServiceTests.cs New MSTest fixture tests used by the pipeline scenario.
tests/dotnet-test/migrate-vstest-to-mtp/eval.yaml New eval suite defining 10 migration scenarios and rubrics.
plugins/dotnet-test/skills/migrate-vstest-to-mtp/SKILL.md New skill playbook for VSTest→MTP migrations (projects, CLI args, CI).
plugins/dotnet-test/plugin.json Updates plugin description to include migration capabilities.
README.md Adds dotnet-test to the root plugin list.
.github/plugin/marketplace.json Registers dotnet-test in the GitHub marketplace list.
.claude-plugin/marketplace.json Mirrors the marketplace registration for Claude marketplace list.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +104 to +106
```

Ensure the project references MSTest 3.2.0 or later. Recommend updating to the latest version.
Comment on lines +195 to +200
| `--blame` | Not applicable | |
| `--blame-crash` | `--crashdump` | Requires `Microsoft.Testing.Extensions.CrashDump` NuGet package |
| `--blame-crash-dump-type <TYPE>` | `--crashdump-type <TYPE>` | Requires CrashDump extension |
| `--blame-hang` | `--hangdump` | Requires `Microsoft.Testing.Extensions.HangDump` NuGet package |
| `--blame-hang-dump-type <TYPE>` | `--hangdump-type <TYPE>` | Requires HangDump extension |
| `--blame-hang-timeout <TIMESPAN>` | `--hangdump-timeout <TIMESPAN>` | Requires HangDump extension |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants