diff --git a/plugin/skills/azure-deploy/SKILL.md b/plugin/skills/azure-deploy/SKILL.md index 015cd3836..f85134db9 100644 --- a/plugin/skills/azure-deploy/SKILL.md +++ b/plugin/skills/azure-deploy/SKILL.md @@ -4,7 +4,7 @@ description: "Execute Azure deployments for ALREADY-PREPARED applications that h license: MIT metadata: author: Microsoft - version: "1.0.2" + version: "1.0.3" --- # Azure Deploy @@ -86,4 +86,4 @@ Activate this skill when user wants to: ## References - [Troubleshooting](references/troubleshooting.md) - Common issues and solutions -- [Post-Deployment Steps](references/recipes/azd/post-deployment.md) - SQL + EF Core setup \ No newline at end of file +- [Post-Deployment Steps](references/recipes/azd/post-deployment.md) - SQL + EF Core setup diff --git a/plugin/skills/azure-deploy/references/recipes/bicep/README.md b/plugin/skills/azure-deploy/references/recipes/bicep/README.md index 10aa06b7f..e83a56c15 100644 --- a/plugin/skills/azure-deploy/references/recipes/bicep/README.md +++ b/plugin/skills/azure-deploy/references/recipes/bicep/README.md @@ -77,6 +77,16 @@ az deployment sub show \ | `mcp_bicep_get_az_resource_type_schema` | Resource schemas | | `mcp_bicep_list_avm_metadata` | Azure Verified Modules | +## AVM Verification Before Deploy + +Before running deployment commands, verify generated templates followed AVM-first module selection: + +1. AVM Bicep Pattern Modules (prefer AVM+AZD patterns) +2. AVM Bicep Resource Modules +3. AVM Bicep Utility Modules + +If no AVM+AZD pattern module is available, fallback must remain within AVM modules (resource -> utility). + ## Cleanup (DESTRUCTIVE) ```bash diff --git a/plugin/skills/azure-prepare/SKILL.md b/plugin/skills/azure-prepare/SKILL.md index 55885842a..e2b6012cb 100644 --- a/plugin/skills/azure-prepare/SKILL.md +++ b/plugin/skills/azure-prepare/SKILL.md @@ -4,7 +4,7 @@ description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.ya license: MIT metadata: author: Microsoft - version: "1.0.1" + version: "1.0.2" --- # Azure Prepare diff --git a/plugin/skills/azure-prepare/references/recipes/azcli/README.md b/plugin/skills/azure-prepare/references/recipes/azcli/README.md index e8cd40ec2..2e57c32d5 100644 --- a/plugin/skills/azure-prepare/references/recipes/azcli/README.md +++ b/plugin/skills/azure-prepare/references/recipes/azcli/README.md @@ -17,7 +17,7 @@ Azure CLI workflow for imperative Azure deployments. | Artifact | Research Action | |----------|-----------------| | Bicep files | Call `mcp_bicep_get_bicep_best_practices` | -| Bicep modules | Call `mcp_bicep_list_avm_metadata` for Azure Verified Modules | +| Bicep modules | Call `mcp_bicep_list_avm_metadata` and follow [AVM module order](../azd/iac-rules.md#avm-module-selection-order-mandatory) | | Azure CLI commands | Call `activate_azure_cli_management_tools` | | Azure best practices | Call `mcp_azure_mcp_get_bestpractices` | diff --git a/plugin/skills/azure-prepare/references/recipes/azd/README.md b/plugin/skills/azure-prepare/references/recipes/azd/README.md index 57c118b74..d5b9b536e 100644 --- a/plugin/skills/azure-prepare/references/recipes/azd/README.md +++ b/plugin/skills/azure-prepare/references/recipes/azd/README.md @@ -46,7 +46,7 @@ Azure Developer CLI workflow for preparing Azure deployments. | Azure Functions templates | [Templates](../../services/functions/templates/README.md) | | Bicep best practices | `mcp_bicep_get_bicep_best_practices` | | Bicep resource schema | `mcp_bicep_get_az_resource_type_schema` | -| Azure Verified Modules | `mcp_bicep_list_avm_metadata` | +| Azure Verified Modules | `mcp_bicep_list_avm_metadata` + [AVM module order](iac-rules.md#avm-module-selection-order-mandatory) | | Terraform best practices | `mcp_azure_mcp_azureterraformbestpractices` | | Dockerfiles | [Docker Guide](docker.md) | diff --git a/plugin/skills/azure-prepare/references/recipes/azd/iac-rules.md b/plugin/skills/azure-prepare/references/recipes/azd/iac-rules.md index 5daae7de1..0497e57f8 100644 --- a/plugin/skills/azure-prepare/references/recipes/azd/iac-rules.md +++ b/plugin/skills/azure-prepare/references/recipes/azd/iac-rules.md @@ -1,6 +1,37 @@ # AZD IAC Rules -Bicep rules for AZD projects. **Additive** — apply `mcp_bicep_get_bicep_best_practices`, `mcp_bicep_list_avm_metadata`, and `mcp_bicep_get_az_resource_type_schema` first, then these azd-specific rules. +IaC rules for AZD projects. **Additive** — for Bicep, apply `mcp_bicep_get_bicep_best_practices`, `mcp_bicep_list_avm_metadata`, and `mcp_bicep_get_az_resource_type_schema` first; for Terraform, apply `mcp_azure_mcp_azureterraformbestpractices` first; then apply these azd-specific rules. + +## AVM Module Selection Order (MANDATORY) + +Always prefer modules in provider-specific order: + +For **Bicep**: +1. AVM Bicep Pattern Modules (AVM+AZD first when available) +2. AVM Bicep Resource Modules +3. AVM Bicep Utility Modules + +For **Terraform**: +1. AVM Terraform Pattern Modules +2. AVM Terraform Resource Modules +3. AVM Terraform Utility Modules + +If no pattern module exists for the active provider, default immediately to AVM modules in the same provider order (resource, then utility) instead of using non-AVM modules. + +## Retrieval Strategy (Hybrid: azure-documentation MCP + Context7) + +- **Primary (authoritative):** Use `mcp_azure_mcp_documentation` (`azure-documentation`) for current Azure guidance and AVM integration documentation. +- **Primary (module catalog):** Use `mcp_bicep_list_avm_metadata` plus official AVM indexes to select concrete modules. +- **Secondary (supplemental):** Use Context7 only for implementation examples when `mcp_azure_mcp_documentation` does not provide enough detail. + +## Validation Plan + +Before finalizing generated guidance: + +1. Verify the selected module path uses the required AVM order above. +2. Verify AVM+AZD pattern modules were checked first, and fallback moved to AVM resource/utility modules when no pattern module exists. +3. Verify Terraform guidance follows pattern -> resource -> utility ordering. +4. Include selected module names and source links in the plan/output for traceability. ## File Structure diff --git a/plugin/skills/azure-prepare/references/recipes/azd/terraform.md b/plugin/skills/azure-prepare/references/recipes/azd/terraform.md index 1c078ada7..f1ffd35e5 100644 --- a/plugin/skills/azure-prepare/references/recipes/azd/terraform.md +++ b/plugin/skills/azure-prepare/references/recipes/azd/terraform.md @@ -274,6 +274,16 @@ When preparing a new azd+Terraform project: - Hosting resources: `azd-service-name` (matches azure.yaml services) 4. **Research best practices** - Call `mcp_azure_mcp_azureterraformbestpractices` +## AVM Terraform Module Priority + +For Terraform module selection, enforce this order: + +1. AVM Terraform Pattern Modules +2. AVM Terraform Resource Modules +3. AVM Terraform Utility Modules + +Use `mcp_azure_mcp_documentation` (`azure-documentation`) for current guidance and AVM context first, then use Context7 only as supplemental examples if required. + ## Migration from Pure Terraform Converting existing Terraform project to use azd: diff --git a/plugin/skills/azure-prepare/references/recipes/bicep/README.md b/plugin/skills/azure-prepare/references/recipes/bicep/README.md index 93bcce4f9..b06d6bc80 100644 --- a/plugin/skills/azure-prepare/references/recipes/bicep/README.md +++ b/plugin/skills/azure-prepare/references/recipes/bicep/README.md @@ -17,7 +17,7 @@ Standalone Bicep workflow (without AZD). | Artifact | Research Action | |----------|-----------------| | Bicep files | Call `mcp_bicep_get_bicep_best_practices` | -| Bicep modules | Call `mcp_bicep_list_avm_metadata` for Azure Verified Modules | +| Bicep modules | Call `mcp_bicep_list_avm_metadata` and follow [AVM module order](../azd/iac-rules.md#avm-module-selection-order-mandatory) | | Resource schemas | Use `activate_azure_resource_schema_tools` if needed | ## Generation Steps diff --git a/tests/README.md b/tests/README.md index 2d49bd7b0..b5b20f621 100644 --- a/tests/README.md +++ b/tests/README.md @@ -166,6 +166,7 @@ go install github.com/microsoft/waza/cmd/waza@latest **Committed eval suites** (⬢ customized graders, fixtures, and assertions): - `azure-prepare` — template selection, recipe composition, plan-first workflow +- `azure-deploy` — deploy routing and AVM+AZD module-priority/fallback guidance **Auto-generated** (⬡ from SKILL.md frontmatter): all other skills diff --git a/tests/azure-deploy/eval/eval.yaml b/tests/azure-deploy/eval/eval.yaml new file mode 100644 index 000000000..bf29b5f15 --- /dev/null +++ b/tests/azure-deploy/eval/eval.yaml @@ -0,0 +1,51 @@ +# Waza eval specification for azure-deploy skill +# Focuses on deploy-time AVM guidance and fallback behavior from issue #1085. +name: azure-deploy-eval +description: | + Evaluation suite for the azure-deploy skill. + Tests deployment guidance quality with emphasis on: + - AVM+AZD pattern-module preference + - AVM fallback when no pattern module exists + - deploy-only routing (not prepare/validate) + +skill: azure-deploy +version: "1.0" + +config: + trials_per_task: 3 + timeout_seconds: 420 + parallel: false + executor: mock + model: claude-sonnet-4-20250514 + +metrics: + - name: task_completion + weight: 0.4 + threshold: 0.8 + description: Did the skill complete a deploy-focused request? + + - name: trigger_accuracy + weight: 0.3 + threshold: 0.85 + description: Does the skill trigger on deploy prompts and avoid prepare-only prompts? + + - name: behavior_quality + weight: 0.3 + threshold: 0.75 + description: Does output preserve AVM+AZD preference and AVM fallback order? + +graders: + - type: code + name: has_output + config: + assertions: + - "len(output) > 10" + + - type: regex + name: no_runtime_failure + config: + must_not_match: + - "(?i)fatal error|exception occurred|crashed" + +tasks: + - "tasks/*.yaml" diff --git a/tests/azure-deploy/eval/tasks/avm-fallback-no-pattern.yaml b/tests/azure-deploy/eval/tasks/avm-fallback-no-pattern.yaml new file mode 100644 index 000000000..59b04dd7d --- /dev/null +++ b/tests/azure-deploy/eval/tasks/avm-fallback-no-pattern.yaml @@ -0,0 +1,42 @@ +# Task: If no AVM+AZD pattern module exists, stay in AVM modules +id: avm-fallback-no-pattern-001 +name: AVM Fallback When No AZD Pattern +description: | + Validate deploy guidance states that if no AVM+AZD pattern module exists, + the fallback remains AVM resource then AVM utility modules. + +tags: + - deploy + - bicep + - avm + - fallback + +inputs: + prompt: | + I'm deploying with Bicep and there is no AVM+AZD pattern module for my scenario. + What module order should I follow if no pattern module exists and fallback must stay AVM resource modules then AVM utility modules? + context: + phase: deploy + iac: bicep + no_pattern_module: true + +expected: + outcomes: + - type: task_completed + output_contains: + - "AVM" + - "resource" + - "utility" + +graders: + - name: explicit_no_pattern_fallback + type: regex + config: + must_match: + - "(?is)(no .*pattern module|if no .*pattern).*AVM.*resource.*AVM.*utility" + + - name: avoids_non_avm_fallback + type: regex + config: + must_not_match: + - "(?i)fallback to non-AVM|use non-AVM modules" diff --git a/tests/azure-deploy/eval/tasks/avm-order-bicep.yaml b/tests/azure-deploy/eval/tasks/avm-order-bicep.yaml new file mode 100644 index 000000000..f248d45a5 --- /dev/null +++ b/tests/azure-deploy/eval/tasks/avm-order-bicep.yaml @@ -0,0 +1,42 @@ +# Task: Deploy guidance prefers AVM+AZD pattern modules +id: avm-order-bicep-001 +name: AVM+AZD Priority - Bicep Deploy +description: | + Validate deploy guidance prefers AVM+AZD pattern modules first, + followed by AVM resource and utility modules for Bicep paths. + +tags: + - deploy + - bicep + - avm + - azd + +inputs: + prompt: | + My app is already prepared and validated. + Give me deploy guidance and module preference order for Bicep. + Prefer AVM+AZD patterns where available, with fallback to AVM resource modules and AVM utility modules. + context: + phase: deploy + iac: bicep + +expected: + outcomes: + - type: task_completed + output_contains: + - "AVM" + - "deploy" + - "pattern" + +graders: + - name: avm_pattern_first + type: regex + config: + must_match: + - "(?i)AVM\\+AZD|AZD pattern|pattern modules" + + - name: includes_resource_and_utility_fallback + type: regex + config: + must_match: + - "(?is)(AVM\\+AZD|AZD pattern|pattern modules).*resource modules.*utility modules" diff --git a/tests/azure-deploy/eval/trigger_tests.yaml b/tests/azure-deploy/eval/trigger_tests.yaml new file mode 100644 index 000000000..6b001c536 --- /dev/null +++ b/tests/azure-deploy/eval/trigger_tests.yaml @@ -0,0 +1,22 @@ +# Trigger accuracy tests for azure-deploy skill +skill: azure-deploy + +should_trigger_prompts: + - prompt: "Run azd up for my already prepared app" + reason: "Explicit deploy execution request" + + - prompt: "Deploy my validated Azure app using Bicep templates" + reason: "Deploy intent with validated state" + + - prompt: "My app is ready. Push it to Azure and keep AVM+AZD module preference." + reason: "Deploy action + AVM deploy guidance preference" + +should_not_trigger_prompts: + - prompt: "Prepare a new app for Azure from scratch" + reason: "Preparation belongs to azure-prepare" + + - prompt: "Validate my infra plan before deploying" + reason: "Validation belongs to azure-validate" + + - prompt: "Create a new Azure Functions app and scaffold code" + reason: "Creation/preparation request, not deploy-only"