Test: Customizations modal redesign

[hawkticehurst]

Refs: #322043

• anyOS Logan Ramos (@lramos15)
• anyOS Dirk Bäumer (@dbaeumer)

Complexity: 5

Authors: Hawk Ticehurst (@hawkticehurst)

Create Issue

---

Context

The Customizations modal has been redesigned with new UI and affordances. The redesign applies to both the regular VS Code workbench and the Agents window.

Also see the end of this test plan for a suggested prompt you can give an agent to interactively guide you through this plan.

Setup

• Install the latest VS Code Insiders
• Use a workspace with a mix of user and workspace customizations, including agents, skills, instructions, hooks, and prompts
• Configure at least one MCP server, one plugin, and an extension that contributes language model tools
• In the Agents window, create sessions for Copilot, Claude, and Codex harnesses
• Create or use a workspace with both kinds of migration candidates:
  • Deprecated user or workspace *.prompt.md files that can be converted to skills
  • Legacy User Data customizations that can be moved to a location supported by the active harness

Overview and navigation

1. Open Customizations from the regular workbench and from the Agents window.
2. Verify the overview heading, introduction, cards, sidebar sections, item counts, and editor title describe the active customization target.
3. Navigate between the overview and each available section. Verify the selected section is retained, the correct search field receives focus, and back navigation from embedded detail pages works.
4. Switch between Copilot, Claude, and Codex harnesses in Agents window. Verify the modal title, project context, supported sections, overview cards, migration entries, and counts update together and do not show items from the previous harness.

Agents, skills, instructions, hooks, and prompts

1. Inspect each available customization page and verify items are grouped under the correct source (such as workspace, user, remote, or built-in).
2. Search by name and description. Verify result counts, empty states, and source groups update correctly. Clearing search restores the full inventory.
3. Click on cards, rows, inline actions, and overflow menus. Verify actions apply only to the intended item and editable items open the correct source.

Migration

1. Verify migration cards and sidebar counts appear only when the active workspace has eligible candidates.
2. Confirm per-item and per-group selection, including the checked, unchecked, and mixed group states.
3. Preview candidates, delete an unwanted candidate, and verify the destination and confirmation text are accurate.
4. Migrate a partial selection and verify selected files reach the intended destination without overwriting an existing target. Check that content formatting is preserved and unsupported metadata is called out for review.
5. Repeat while choosing to keep the original files.
6. Start a migration, switch the active Agents-window session before confirming, and verify the migration remains bound to the session and workspace where it began.

MCP servers

1. Verify installed and marketplace sections load, search correctly, and distinguish installed, available, disabled, and plugin-owned servers.
2. Install an available server and open its embedded detail page. Verify source, scope, status, configuration or definition information, and back navigation.
3. Start, stop, and restart a server, then open its output and configuration. Verify status and available actions update without requiring the editor to be reopened.
4. Where available, verify setting-disabled and policy-disabled states explain why management is unavailable and do not offer actions that cannot succeed.

Plugins

1. Verify installed, remote, recommended, and marketplace content appears in the appropriate sections.
2. Browse and search marketplace, then install a plugin and confirm it moves to the installed state.
3. Open installed and marketplace plugin details. Verify provenance, installation path and copy action, enablement or policy state, and the available install, update, enable, disable, and uninstall actions.
4. Inspect the plugin's contributed agents, skills, instructions, hooks, MCP servers, and commands. Open several contributions and verify navigation reaches the correct customization section or source file.
5. Install a plugin from source (such as the Slack plugin) and create a plugin through the provided entry points.

Tools and remaining sections

1. Verify tools are grouped by built-in, MCP or user-connected, and extension-contributed sources.
2. Expand tool sets, search for sets and individual tools, and exercise per-set and per-tool enablement.
3. Verify provider-owned or otherwise read-only sets cannot be changed and clearly communicate that state.
4. Browse extensions that contribute language model tools and verify extension management actions work in both the regular workbench and the Agents window.

Keyboard, accessibility, layout, and persistence

1. Complete the main flows using only the keyboard. Verify logical focus order and visible focus for the overview, sidebar, search fields, cards, rows, inline actions, overflow menus, checkboxes, and detail-page back buttons.
2. Verify cards and controls activate with the expected Enter or Space key and that focus returns predictably after navigation or an action.
3. With a screen reader enabled, verify controls have descriptive labels and search result counts and state changes are announced without duplicate announcements.
4. Resize the editor between narrow and wide layouts. Verify cards, badges, actions, text, and scrollable regions reflow without clipping, overlap, or inaccessible controls.
5. Check representative pages in light, dark, and high-contrast themes.
6. Reload the window and verify the selected section, active target, inventories, counts, enablement states, and detail navigation recover correctly.

Coding agent test guide

Give the following prompt to a coding agent to have it guide you interactively through this test plan:

Please help me complete the test plan described in https://github.com/microsoft/vscode/issues/333535. Act as an interactive test facilitator, not as the tester: do not assume that a check passed, and do not implement or fix the feature while guiding me.

First, read the full test plan and organize its checks into a checklist. Then guide me through it in small, logical batches. For each batch:

1. Tell me any prerequisites and the exact actions I should take.
2. Clearly state what I should expect to observe.
3. Wait for my verification before moving on.

You must use the Ask User Question tool after every batch so I can respond by clicking one of these options:

- It worked
- I found a problem
- I am blocked

Do not ask me to report the result in plain chat text. If I select "I found a problem," use the Ask User Question tool again to collect what happened, the expected behavior, and any useful reproduction details before continuing. If I am blocked, help diagnose the blocker or adapt the remaining plan without marking the checks as passed. When a prerequisite or feature is unavailable, record the affected checks as skipped or blocked rather than silently omitting them.

Keep a running record of passed, failed, and blocked checks. Do not mark anything as passed without my explicit confirmation. At the end, use the Ask User Question tool to confirm that I am ready to finish, then provide GitHub-ready issue descriptions I can paste into each issue I create against the test plan, that include:

- A concise summary of the issue
- Reproduction steps and observed versus expected behavior for each failure
- Relevant environment details