diff --git a/plugin/skills/azure-deploy/SKILL.md b/plugin/skills/azure-deploy/SKILL.md index f85134db9..1d6934ae7 100644 --- a/plugin/skills/azure-deploy/SKILL.md +++ b/plugin/skills/azure-deploy/SKILL.md @@ -4,7 +4,7 @@ description: "Execute Azure deployments for ALREADY-PREPARED applications that h license: MIT metadata: author: Microsoft - version: "1.0.3" + version: "1.0.4" --- # Azure Deploy diff --git a/plugin/skills/azure-deploy/references/pre-deploy-checklist.md b/plugin/skills/azure-deploy/references/pre-deploy-checklist.md index 1b10725fc..ae153eb96 100644 --- a/plugin/skills/azure-deploy/references/pre-deploy-checklist.md +++ b/plugin/skills/azure-deploy/references/pre-deploy-checklist.md @@ -155,6 +155,22 @@ azd up --no-prompt --- +## Service-Specific Checks + +### Durable Functions — Verify DTS Backend + +> **⛔ MANDATORY**: If the plan includes Durable Functions, verify infrastructure uses **Durable Task Scheduler** (DTS), NOT Azure Storage. + +Check that `infra/` Bicep files contain: +- `Microsoft.DurableTask/schedulers` resource +- `Microsoft.DurableTask/schedulers/taskHubs` child resource +- `Durable Task Data Contributor` RBAC role assignment +- `DURABLE_TASK_SCHEDULER_CONNECTION_STRING` app setting + +If any are missing, **STOP** and invoke **azure-prepare** to regenerate with the durable recipe. + +--- + ## Non-AZD Deployments **For Azure CLI / Bicep:** diff --git a/plugin/skills/azure-prepare/SKILL.md b/plugin/skills/azure-prepare/SKILL.md index e2b6012cb..7253caacc 100644 --- a/plugin/skills/azure-prepare/SKILL.md +++ b/plugin/skills/azure-prepare/SKILL.md @@ -4,7 +4,7 @@ description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.ya license: MIT metadata: author: Microsoft - version: "1.0.2" + version: "1.0.3" --- # Azure Prepare @@ -63,6 +63,7 @@ Activate this skill when user wants to: | Azure Functions, function app, serverless function, timer trigger, HTTP trigger, func new | Stay in **azure-prepare** — prefer Azure Functions templates in Step 4 | | APIM, API Management, API gateway, deploy APIM | Stay in **azure-prepare** — see [APIM Deployment Guide](references/apim.md) | | AI gateway, AI gateway policy, AI gateway backend, AI gateway configuration | **azure-aigateway** | +| workflow, orchestration, multi-step, pipeline, fan-out/fan-in, saga, long-running process, durable | Stay in **azure-prepare** — select **durable** recipe in Step 4. **MUST** load [durable.md](references/services/functions/durable.md) and [DTS reference](references/services/durable-task-scheduler/README.md). Generate `Microsoft.DurableTask/schedulers` + `taskHubs` Bicep resources. | > ⚠️ Check the user's **prompt text** — not just existing code. Critical for greenfield projects with no codebase to scan. See [full routing table](references/specialized-routing.md). diff --git a/plugin/skills/azure-prepare/references/architecture.md b/plugin/skills/azure-prepare/references/architecture.md index e340c7a19..dba46e7fb 100644 --- a/plugin/skills/azure-prepare/references/architecture.md +++ b/plugin/skills/azure-prepare/references/architecture.md @@ -18,7 +18,8 @@ Select hosting stack and map components to Azure services. | Event-driven | ✓ | ✓✓ | | | Variable traffic | | ✓✓ | ✓ | | Complex dependencies | ✓✓ | | ✓ | -| Long-running processes | ✓✓ | | ✓ | +| Long-running processes | ✓✓ | ✓ (Durable Functions) | ✓ | +| Workflow / orchestration | | ✓✓ (Durable Functions + DTS) | | | Minimal ops overhead | | ✓✓ | ✓ | ## Service Mapping @@ -51,7 +52,13 @@ Select hosting stack and map components to Azure services. | Message Queue | Service Bus | | Pub/Sub | Event Grid | | Streaming | Event Hubs | -| Workflow | Logic Apps, Durable Functions | + +### Workflow & Orchestration + +| Need | Service | Notes | +|------|---------|-------| +| Multi-step workflow / orchestration | **Durable Functions + Durable Task Scheduler** | DTS is the **required** managed backend for Durable Functions. Do NOT use Azure Storage or MSSQL backends. See [durable.md](services/functions/durable.md). | +| Low-code / visual workflow | Logic Apps | For integration-heavy, low-code scenarios | ### Supporting (Always Include) diff --git a/plugin/skills/azure-prepare/references/research.md b/plugin/skills/azure-prepare/references/research.md index e8d2838fc..23ee8c172 100644 --- a/plugin/skills/azure-prepare/references/research.md +++ b/plugin/skills/azure-prepare/references/research.md @@ -35,6 +35,9 @@ After architecture planning, research each selected component to gather best pra | **Integration** | | | | API Management | [APIM](apim.md) | `azure-aigateway` (invoke for AI Gateway policies) | | Logic Apps | [Logic Apps](services/logic-apps/README.md) | — | +| **Workflow & Orchestration** | | | +| Durable Functions | [Durable Functions](services/functions/durable.md), [Durable Task Scheduler](services/durable-task-scheduler/README.md) | — | +| Durable Task Scheduler | [Durable Task Scheduler](services/durable-task-scheduler/README.md) | — | | **Security & Identity** | | | | Key Vault | [Key Vault](services/key-vault/README.md) | `azure-security`, `azure-keyvault-expiration-audit` | | Managed Identity | — | `azure-security`, `entra-app-registration` | diff --git a/plugin/skills/azure-prepare/references/services/durable-task-scheduler/README.md b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/README.md new file mode 100644 index 000000000..11fc4f6ad --- /dev/null +++ b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/README.md @@ -0,0 +1,77 @@ +# Durable Task Scheduler + +Build reliable, fault-tolerant workflows using durable execution with Azure Durable Task Scheduler. + +## When to Use + +- Long-running workflows requiring state persistence +- Distributed transactions with compensating actions (saga pattern) +- Multi-step orchestrations with checkpointing +- Fan-out/fan-in parallel processing +- Workflows requiring human interaction or external events +- Stateful entities (aggregators, counters, state machines) +- Multi-agent AI orchestration +- Data processing pipelines + +## Framework Selection + +| Framework | Best For | Hosting | +|-----------|----------|---------| +| **Durable Functions** | Serverless event-driven apps | Azure Functions | +| **Durable Task SDKs** | Any compute (containers, VMs) | Azure Container Apps, Azure Kubernetes Service, App Service, VMs | + +> **💡 TIP**: Use Durable Functions for serverless with built-in triggers. Use Durable Task SDKs for hosting flexibility. + +## Quick Start - Local Emulator + +```bash +# Start the emulator (see https://mcr.microsoft.com/v2/dts/dts-emulator/tags/list for available versions) +docker pull mcr.microsoft.com/dts/dts-emulator:latest +docker run -d -p 8080:8080 -p 8082:8082 --name dts-emulator mcr.microsoft.com/dts/dts-emulator:latest + +# Dashboard available at http://localhost:8082 +``` + +## Workflow Patterns + +| Pattern | Use When | +|---------|----------| +| **Function Chaining** | Sequential steps, each depends on previous | +| **Fan-Out/Fan-In** | Parallel processing with aggregated results | +| **Async HTTP APIs** | Long-running operations with HTTP polling | +| **Monitor** | Periodic polling with configurable timeouts | +| **Human Interaction** | Workflow pauses for external input/approval | +| **Saga** | Distributed transactions with compensation | +| **Durable Entities** | Stateful objects (counters, accounts) | + +## Connection & Authentication + +| Environment | Connection String | +|-------------|-------------------| +| Local Development (Emulator) | `Endpoint=http://localhost:8080;Authentication=None;TaskHub=default` | +| Azure (System-Assigned MI) | `Endpoint=https://.durabletask.io;Authentication=ManagedIdentity;TaskHub=default` | +| Azure (User-Assigned MI) | `Endpoint=https://.durabletask.io;Authentication=ManagedIdentity;ClientID=;TaskHub=default` | + +> **⚠️ NOTE**: Durable Task Scheduler uses identity-based authentication only — no connection strings with keys. When using a User-Assigned Managed Identity (UAMI), you must include the `ClientID` in the connection string. + +## Troubleshooting + +| Error | Cause | Fix | +|-------|-------|-----| +| **403 PermissionDenied** on gRPC call (e.g., `client.start_new()`) | Function App managed identity lacks RBAC on the Durable Task Scheduler resource, or IP allowlist blocks traffic | 1. Assign `Durable Task Data Contributor` role (`0ad04412-c4d5-4796-b79c-f76d14c8d402`) to the identity (SAMI or UAMI) scoped to the Durable Task Scheduler resource. For UAMI, also ensure the connection string includes `ClientID=`. 2. Ensure the scheduler's `ipAllowlist` includes `0.0.0.0/0` (an empty list denies all traffic). 3. RBAC propagation can take up to 10 minutes — restart the Function App after assigning roles. | +| **Connection refused** to emulator | Emulator container not running or wrong port | Verify container is running: `docker ps` and confirm port 8080 is mapped | +| **403 despite correct RBAC** | Scheduler IP allowlist is empty (denies all) | Set `ipAllowlist: ['0.0.0.0/0']` in Bicep or update via CLI: `az durabletask scheduler update --ip-allowlist '0.0.0.0/0'` | +| **TaskHub not found** | Task hub not provisioned or name mismatch | Ensure the `TaskHub` parameter in the `DURABLE_TASK_SCHEDULER_CONNECTION_STRING` matches the provisioned task hub name | +| **403 Forbidden** on DTS dashboard | Deploying user lacks RBAC on the scheduler | Assign `Durable Task Data Contributor` role to your own user identity (not just the Function App MI) scoped to the scheduler resource — see [Bicep Patterns](bicep.md) for the dashboard role assignment snippet | + +## References + +- [.NET](dotnet.md) — packages, setup, examples, determinism, retry, SDK +- [Python](python.md) — packages, setup, examples, determinism, retry, SDK +- [Java](java.md) — dependencies, setup, examples, determinism, retry, SDK +- [JavaScript](javascript.md) — packages, setup, examples, determinism, retry, SDK +- [Bicep Patterns](bicep.md) — scheduler, task hub, RBAC, CLI provisioning +- [Official Documentation](https://learn.microsoft.com/azure/azure-functions/durable/durable-task-scheduler/durable-task-scheduler) +- [Durable Functions Overview](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-overview) +- [Sample Repository](https://github.com/Azure-Samples/Durable-Task-Scheduler) +- [Choosing an Orchestration Framework](https://learn.microsoft.com/azure/azure-functions/durable/durable-task-scheduler/choose-orchestration-framework) diff --git a/plugin/skills/azure-prepare/references/services/durable-task-scheduler/bicep.md b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/bicep.md new file mode 100644 index 000000000..b595dc528 --- /dev/null +++ b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/bicep.md @@ -0,0 +1,116 @@ +# Durable Task Scheduler — Bicep Patterns + +Bicep templates for provisioning the Durable Task Scheduler, task hubs, and RBAC role assignments. + +## Scheduler + Task Hub + +```bicep +// Parameters — define these at file level or pass from a parent module +param schedulerName string +param location string = resourceGroup().location + +@allowed(['Consumption', 'Dedicated']) +@description('Use Consumption for quickstarts/variable workloads, Dedicated for high-demand/predictable throughput') +param skuName string = 'Consumption' + +resource scheduler 'Microsoft.DurableTask/schedulers@2025-11-01' = { + name: schedulerName + location: location + properties: { + sku: { name: skuName } + ipAllowlist: ['0.0.0.0/0'] // Required: empty list denies all traffic + } +} + +resource taskHub 'Microsoft.DurableTask/schedulers/taskHubs@2025-11-01' = { + parent: scheduler + name: 'default' +} +``` + +## SKU Selection + +| SKU | Best For | +|-----|----------| +| **Consumption** | quickstarts, variable or bursty workloads, pay-per-use | +| **Dedicated** | High-demand workloads, predictable throughput requirements | + +> **💡 TIP**: Start with `Consumption` for development and variable workloads. Switch to `Dedicated` when you need consistent, high-throughput performance. + +> **⚠️ WARNING**: The scheduler's `ipAllowlist` **must** include at least one entry (e.g., `['0.0.0.0/0']` for allow-all). An empty array `[]` denies **all** traffic, causing 403 errors on gRPC calls even with correct RBAC. + +## RBAC — Durable Task Data Contributor + +The Function App's managed identity **must** have the `Durable Task Data Contributor` role on the scheduler resource. Without it, the app receives **403 PermissionDenied** on gRPC calls. + +```bicep +// Assumes the UAMI principal ID is passed from the base template's identity module +param functionAppPrincipalId string + +var durableTaskDataContributorRoleId = '0ad04412-c4d5-4796-b79c-f76d14c8d402' + +resource durableTaskRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(scheduler.id, functionAppPrincipalId, durableTaskDataContributorRoleId) + scope: scheduler + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', durableTaskDataContributorRoleId) + principalId: functionAppPrincipalId + principalType: 'ServicePrincipal' + } +} +``` + +## RBAC — Dashboard Access for Developers + +To allow developers to view orchestration status and history in the [DTS dashboard](https://portal.azure.com), assign the same `Durable Task Data Contributor` role to the deploying user's identity. Without this, the dashboard returns **403 Forbidden**. + +```bicep +// Accept the deploying user's principal ID (azd auto-populates this from AZURE_PRINCIPAL_ID) +param principalId string = '' + +resource dashboardRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (!empty(principalId)) { + name: guid(scheduler.id, principalId, durableTaskDataContributorRoleId) + scope: scheduler + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', durableTaskDataContributorRoleId) + principalId: principalId + principalType: 'User' + } +} +``` + +> **💡 TIP**: This is the same role used for the Function App's managed identity, but assigned with `principalType: 'User'` to the developer. See the [sample repo](https://github.com/Azure-Samples/Durable-Task-Scheduler/blob/main/samples/infra/main.bicep) for a full example. + +## Connection String App Setting + +Include these entries in the Function App resource's `siteConfig.appSettings` array: + +```bicep +// UAMI client ID from base template identity module - REQUIRED for UAMI auth +param uamiClientId string + +{ + name: 'DURABLE_TASK_SCHEDULER_CONNECTION_STRING' + value: 'Endpoint=${scheduler.properties.endpoint};TaskHub=${taskHub.name};Authentication=ManagedIdentity;ClientID=${uamiClientId}' +} +``` + +> **⚠️ IMPORTANT**: The base templates use User Assigned Managed Identity (UAMI). You **must** include `ClientID=` in the connection string. Without it, the Durable Task SDK cannot resolve the correct identity. + +> **⚠️ WARNING**: Always use `scheduler.properties.endpoint` to get the scheduler URL. Do **not** construct it manually — the endpoint includes a hash suffix and region (e.g., `https://myscheduler-abc123.westus2.durabletask.io`). + +## Provision via CLI + +> **💡 TIP**: When hosting Durable Functions, use a **Flex Consumption** plan (`FC1` SKU) rather than the legacy Consumption plan (`Y1`). Flex Consumption supports identity-based storage connections natively and handles deployment artifacts correctly. + +```bash +# Install the durabletask CLI extension (if not already installed) +az extension add --name durabletask + +# Create scheduler (consumption SKU for getting started) +az durabletask scheduler create \ + --resource-group myResourceGroup \ + --name my-scheduler \ + --location eastus \ + --sku consumption +``` diff --git a/plugin/skills/azure-prepare/references/services/durable-task-scheduler/dotnet.md b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/dotnet.md new file mode 100644 index 000000000..09975dc33 --- /dev/null +++ b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/dotnet.md @@ -0,0 +1,190 @@ +# Durable Task Scheduler — .NET + +## Learn More + +- [Durable Task Scheduler documentation](https://learn.microsoft.com/azure/durable-task-scheduler/) +- [Durable Functions .NET isolated worker guide](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-dotnet-isolated-overview) + +## Durable Functions Setup + +### Required NuGet Packages + +```xml + + + + + +``` + +> **💡 Finding latest versions**: Search [nuget.org](https://www.nuget.org/) for each package name to find the current stable version. Look for the `Microsoft.Azure.Functions.Worker.Extensions.DurableTask` and `Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureManaged` packages. + +### host.json + +```json +{ + "version": "2.0", + "extensions": { + "durableTask": { + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + }, + "hubName": "default" + } + } +} +``` + +> **💡 NOTE**: .NET isolated uses the `DurableTask.AzureManaged` NuGet package, which registers the `azureManaged` storage provider type. Other runtimes (Python, Java, JavaScript) use extension bundles and require `durabletask-scheduler` instead — see the respective language files. All runtimes use the same `DURABLE_TASK_SCHEDULER_CONNECTION_STRING` environment variable. + +### local.settings.json + +```json +{ + "IsEncrypted": false, + "Values": { + "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated", + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None" + } +} +``` + +## Minimal Example + +```csharp +using Microsoft.Azure.Functions.Worker; +using Microsoft.DurableTask; +using Microsoft.DurableTask.Client; + +public static class DurableFunctionsApp +{ + [Function("HttpStart")] + public static async Task HttpStart( + [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, + [DurableClient] DurableTaskClient client) + { + string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(nameof(MyOrchestration)); + return await client.CreateCheckStatusResponseAsync(req, instanceId); + } + + [Function(nameof(MyOrchestration))] + public static async Task MyOrchestration([OrchestrationTrigger] TaskOrchestrationContext context) + { + var result1 = await context.CallActivityAsync(nameof(SayHello), "Tokyo"); + var result2 = await context.CallActivityAsync(nameof(SayHello), "Seattle"); + return $"{result1}, {result2}"; + } + + [Function(nameof(SayHello))] + public static string SayHello([ActivityTrigger] string name) => $"Hello {name}!"; +} +``` + +## Workflow Patterns + +### Fan-Out/Fan-In + +```csharp +[Function(nameof(FanOutFanIn))] +public static async Task FanOutFanIn([OrchestrationTrigger] TaskOrchestrationContext context) +{ + string[] cities = { "Tokyo", "Seattle", "London", "Paris", "Berlin" }; + + // Fan-out: schedule all in parallel + var tasks = cities.Select(city => context.CallActivityAsync(nameof(SayHello), city)); + + // Fan-in: wait for all + return await Task.WhenAll(tasks); +} +``` + +### Human Interaction + +```csharp +[Function(nameof(ApprovalWorkflow))] +public static async Task ApprovalWorkflow([OrchestrationTrigger] TaskOrchestrationContext context) +{ + await context.CallActivityAsync(nameof(SendApprovalRequest), context.GetInput()); + + // Wait for approval event with timeout + using var cts = new CancellationTokenSource(); + var approvalTask = context.WaitForExternalEvent("ApprovalEvent"); + var timeoutTask = context.CreateTimer(context.CurrentUtcDateTime.AddDays(3), cts.Token); + + var winner = await Task.WhenAny(approvalTask, timeoutTask); + + if (winner == approvalTask) + { + cts.Cancel(); + return await approvalTask ? "Approved" : "Rejected"; + } + return "Timed out"; +} +``` + +## Orchestration Determinism + +| ❌ NEVER | ✅ ALWAYS USE | +|----------|--------------| +| `DateTime.Now` | `context.CurrentUtcDateTime` | +| `Guid.NewGuid()` | `context.NewGuid()` | +| `Random` | Pass random values from activities | +| `Task.Delay()`, `Thread.Sleep()` | `context.CreateTimer()` | +| Direct I/O, HTTP, database | `context.CallActivityAsync()` | + +### Replay-Safe Logging + +```csharp +[Function(nameof(MyOrchestration))] +public static async Task MyOrchestration([OrchestrationTrigger] TaskOrchestrationContext context) +{ + ILogger logger = context.CreateReplaySafeLogger(nameof(MyOrchestration)); + logger.LogInformation("Started"); // Only logs once, not on replay + return await context.CallActivityAsync(nameof(MyActivity), "input"); +} +``` + +## Error Handling & Retry + +```csharp +var retryOptions = new TaskOptions +{ + Retry = new RetryPolicy( + maxNumberOfAttempts: 3, + firstRetryInterval: TimeSpan.FromSeconds(5), + backoffCoefficient: 2.0, + maxRetryInterval: TimeSpan.FromMinutes(1)) +}; + +var input = context.GetInput(); + +try +{ + await context.CallActivityAsync(nameof(UnreliableService), input, retryOptions); +} +catch (TaskFailedException ex) +{ + context.SetCustomStatus(new { Error = ex.Message }); + await context.CallActivityAsync(nameof(CompensationActivity), input); +} +``` + +## Durable Task SDK (Non-Functions) + +For applications running outside Azure Functions (containers, VMs, Azure Container Apps, Azure Kubernetes Service): + +```csharp +var connectionString = "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None"; + +// Worker +builder.Services.AddDurableTaskWorker() + .AddTasks(registry => registry.AddAllGeneratedTasks()) + .UseDurableTaskScheduler(connectionString); + +// Client +var client = DurableTaskClientBuilder.UseDurableTaskScheduler(connectionString).Build(); +string instanceId = await client.ScheduleNewOrchestrationInstanceAsync("MyOrchestration", input); +``` + diff --git a/plugin/skills/azure-prepare/references/services/durable-task-scheduler/java.md b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/java.md new file mode 100644 index 000000000..74c32b21f --- /dev/null +++ b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/java.md @@ -0,0 +1,243 @@ +# Durable Task Scheduler — Java + +## Learn More + +- [Durable Task Scheduler documentation](https://learn.microsoft.com/azure/durable-task-scheduler/) +- [Durable Functions Java guide](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-overview?tabs=java) + +## Durable Functions Setup + +### Required Maven Dependencies + +```xml + + + com.microsoft.azure.functions + azure-functions-java-library + 3.2.3 + + + com.microsoft + durabletask-azure-functions + 1.7.0 + + +``` + +> **💡 Finding latest versions**: Search [Maven Central](https://central.sonatype.com/) for `durabletask-azure-functions` (group: `com.microsoft`) to find the current stable version. + +### host.json + +```json +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.*, 5.0.0)" + } +} +``` + +### local.settings.json + +```json +{ + "IsEncrypted": false, + "Values": { + "FUNCTIONS_WORKER_RUNTIME": "java", + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None" + } +} +``` + +## Minimal Example + +```java +import com.microsoft.azure.functions.*; +import com.microsoft.azure.functions.annotation.*; +import com.microsoft.durabletask.*; +import com.microsoft.durabletask.azurefunctions.*; + +public class DurableFunctionsApp { + + @FunctionName("HttpStart") + public HttpResponseMessage httpStart( + @HttpTrigger(name = "req", methods = {HttpMethod.POST}, authLevel = AuthorizationLevel.FUNCTION) + HttpRequestMessage request, + @DurableClientInput(name = "durableContext") DurableClientContext durableContext) { + DurableTaskClient client = durableContext.getClient(); + String instanceId = client.scheduleNewOrchestrationInstance("MyOrchestration"); + return durableContext.createCheckStatusResponse(request, instanceId); + } + + @FunctionName("MyOrchestration") + public String myOrchestration( + @DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) { + String result1 = ctx.callActivity("SayHello", "Tokyo", String.class).await(); + String result2 = ctx.callActivity("SayHello", "Seattle", String.class).await(); + return result1 + ", " + result2; + } + + @FunctionName("SayHello") + public String sayHello(@DurableActivityTrigger(name = "name") String name) { + return "Hello " + name + "!"; + } +} +``` + +## Workflow Patterns + +### Fan-Out/Fan-In + +```java +@FunctionName("FanOutFanIn") +public List fanOutFanIn( + @DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) { + String[] cities = {"Tokyo", "Seattle", "London", "Paris", "Berlin"}; + List> parallelTasks = new ArrayList<>(); + + // Fan-out: schedule all activities in parallel + for (String city : cities) { + parallelTasks.add(ctx.callActivity("SayHello", city, String.class)); + } + + // Fan-in: wait for all to complete + List results = new ArrayList<>(); + for (Task task : parallelTasks) { + results.add(task.await()); + } + + return results; +} +``` + +### Human Interaction + +```java +@FunctionName("ApprovalWorkflow") +public String approvalWorkflow( + @DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) { + ctx.callActivity("SendApprovalRequest", ctx.getInput(String.class)).await(); + + // Wait for approval event with timeout + Task approvalTask = ctx.waitForExternalEvent("ApprovalEvent", Boolean.class); + Task timeoutTask = ctx.createTimer(Duration.ofDays(3)); + + Task winner = ctx.anyOf(approvalTask, timeoutTask).await(); + + if (winner == approvalTask) { + return approvalTask.await() ? "Approved" : "Rejected"; + } + return "Timed out"; +} +``` + +## Orchestration Determinism + +| ❌ NEVER | ✅ ALWAYS USE | +|----------|--------------| +| `System.currentTimeMillis()` | `ctx.getCurrentInstant()` | +| `UUID.randomUUID()` | Pass random values from activities | +| `Thread.sleep()` | `ctx.createTimer()` | +| Direct I/O, HTTP, database | `ctx.callActivity()` | + +### Replay-Safe Logging + +```java +private static final java.util.logging.Logger logger = + java.util.logging.Logger.getLogger("MyOrchestration"); + +@FunctionName("MyOrchestration") +public String myOrchestration( + @DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) { + // Use isReplaying to avoid duplicate logs + if (!ctx.getIsReplaying()) { + logger.info("Started"); // Only logs once, not on replay + } + return ctx.callActivity("MyActivity", "input", String.class).await(); +} +``` + +## Error Handling & Retry + +```java +@FunctionName("WorkflowWithRetry") +public String workflowWithRetry( + @DurableOrchestrationTrigger(name = "ctx") TaskOrchestrationContext ctx) { + TaskOptions retryOptions = new TaskOptions(new RetryPolicy( + 3, // maxNumberOfAttempts + Duration.ofSeconds(5) // firstRetryInterval + )); + + try { + return ctx.callActivity("UnreliableService", ctx.getInput(String.class), + retryOptions, String.class).await(); + } catch (TaskFailedException ex) { + ctx.setCustomStatus(Map.of("Error", ex.getMessage())); + ctx.callActivity("CompensationActivity", ctx.getInput(String.class)).await(); + return "Compensated"; + } +} +``` + +## Durable Task SDK (Non-Functions) + +For applications running outside Azure Functions (containers, VMs, Azure Container Apps, Azure Kubernetes Service): + +```java +import com.microsoft.durabletask.*; +import com.microsoft.durabletask.azuremanaged.DurableTaskSchedulerWorkerExtensions; +import com.microsoft.durabletask.azuremanaged.DurableTaskSchedulerClientExtensions; + +import java.time.Duration; + +public class App { + public static void main(String[] args) throws Exception { + String connectionString = "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None"; + + // Worker + DurableTaskGrpcWorker worker = DurableTaskSchedulerWorkerExtensions + .createWorkerBuilder(connectionString) + .addOrchestration(new TaskOrchestrationFactory() { + @Override public String getName() { return "MyOrchestration"; } + @Override public TaskOrchestration create() { + return ctx -> { + String result = ctx.callActivity("SayHello", + ctx.getInput(String.class), String.class).await(); + ctx.complete(result); + }; + } + }) + .addActivity(new TaskActivityFactory() { + @Override public String getName() { return "SayHello"; } + @Override public TaskActivity create() { + return ctx -> "Hello " + ctx.getInput(String.class) + "!"; + } + }) + .build(); + + worker.start(); + + // Client + DurableTaskClient client = DurableTaskSchedulerClientExtensions + .createClientBuilder(connectionString).build(); + String instanceId = client.scheduleNewOrchestrationInstance("MyOrchestration", "World"); + OrchestrationMetadata result = client.waitForInstanceCompletion( + instanceId, Duration.ofSeconds(30), true); + System.out.println("Output: " + result.readOutputAs(String.class)); + + worker.stop(); + } +} +``` + diff --git a/plugin/skills/azure-prepare/references/services/durable-task-scheduler/javascript.md b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/javascript.md new file mode 100644 index 000000000..32f5e9364 --- /dev/null +++ b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/javascript.md @@ -0,0 +1,211 @@ +# Durable Task Scheduler — JavaScript + +## Learn More + +- [Durable Task Scheduler documentation](https://learn.microsoft.com/azure/durable-task-scheduler/) +- [Durable Functions JavaScript guide](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-overview?tabs=javascript) + +## Durable Functions Setup + +### Required npm Packages + +```json +{ + "dependencies": { + "@azure/functions": "^4.0.0", + "durable-functions": "^3.0.0" + } +} +``` + +> **💡 Finding latest versions**: Run `npm view durable-functions version` or check [npmjs.com/package/durable-functions](https://www.npmjs.com/package/durable-functions) for the latest stable release. + +### host.json + +```json +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.*, 5.0.0)" + } +} +``` + +### local.settings.json + +```json +{ + "IsEncrypted": false, + "Values": { + "FUNCTIONS_WORKER_RUNTIME": "node", + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None" + } +} +``` + +## Minimal Example + +```javascript +const { app } = require("@azure/functions"); +const df = require("durable-functions"); + +// Activity +df.app.activity("sayHello", { + handler: (city) => `Hello ${city}!`, +}); + +// Orchestrator +df.app.orchestration("myOrchestration", function* (context) { + const result1 = yield context.df.callActivity("sayHello", "Tokyo"); + const result2 = yield context.df.callActivity("sayHello", "Seattle"); + return `${result1}, ${result2}`; +}); + +// HTTP Starter +app.http("HttpStart", { + route: "orchestrators/{orchestrationName}", + methods: ["POST"], + authLevel: "function", + extraInputs: [df.input.durableClient()], + handler: async (request, context) => { + const client = df.getClient(context); + const instanceId = await client.startNew(request.params.orchestrationName); + return client.createCheckStatusResponse(request, instanceId); + }, +}); +``` + +## Workflow Patterns + +### Fan-Out/Fan-In + +```javascript +df.app.orchestration("fanOutFanIn", function* (context) { + const cities = ["Tokyo", "Seattle", "London", "Paris", "Berlin"]; + + // Fan-out: schedule all activities in parallel + const tasks = cities.map((city) => context.df.callActivity("sayHello", city)); + + // Fan-in: wait for all to complete + const results = yield context.df.Task.all(tasks); + return results; +}); +``` + +### Human Interaction + +```javascript +df.app.orchestration("approvalWorkflow", function* (context) { + yield context.df.callActivity("sendApprovalRequest", context.df.getInput()); + + // Wait for approval event with timeout + const expiration = new Date(context.df.currentUtcDateTime); + expiration.setDate(expiration.getDate() + 3); + + const approvalTask = context.df.waitForExternalEvent("ApprovalEvent"); + const timeoutTask = context.df.createTimer(expiration); + + const winner = yield context.df.Task.any([approvalTask, timeoutTask]); + + if (winner === approvalTask) { + return approvalTask.result ? "Approved" : "Rejected"; + } + return "Timed out"; +}); +``` + +## Orchestration Determinism + +| ❌ NEVER | ✅ ALWAYS USE | +|----------|--------------| +| `new Date()` | `context.df.currentUtcDateTime` | +| `Math.random()` | Pass random values from activities | +| `setTimeout()` | `context.df.createTimer()` | +| Direct I/O, HTTP, database | `context.df.callActivity()` | + +### Replay-Safe Logging + +```javascript +df.app.orchestration("myOrchestration", function* (context) { + if (!context.df.isReplaying) { + console.log("Started"); // Only logs once, not on replay + } + const result = yield context.df.callActivity("myActivity", "input"); + return result; +}); +``` + +## Error Handling & Retry + +```javascript +df.app.orchestration("workflowWithRetry", function* (context) { + const retryOptions = new df.RetryOptions(5000, 3); // firstRetryInterval, maxAttempts + retryOptions.backoffCoefficient = 2.0; + retryOptions.maxRetryIntervalInMilliseconds = 60000; + + try { + const result = yield context.df.callActivityWithRetry( + "unreliableService", + retryOptions, + context.df.getInput() + ); + return result; + } catch (ex) { + context.df.setCustomStatus({ error: ex.message }); + yield context.df.callActivity("compensationActivity", context.df.getInput()); + return "Compensated"; + } +}); +``` + +## Durable Task SDK (Non-Functions) + +For applications running outside Azure Functions (containers, VMs, Azure Container Apps, Azure Kubernetes Service): + +```javascript +const { createAzureManagedWorkerBuilder, createAzureManagedClient } = require("@microsoft/durabletask-js-azuremanaged"); + +const connectionString = "Endpoint=http://localhost:8080;Authentication=None;TaskHub=default"; + +// Activity +const sayHello = async (_ctx, name) => `Hello ${name}!`; + +// Orchestrator +const myOrchestration = async function* (ctx, name) { + const result = yield ctx.callActivity(sayHello, name); + return result; +}; + +async function main() { + // Worker + const worker = createAzureManagedWorkerBuilder(connectionString) + .addOrchestrator(myOrchestration) + .addActivity(sayHello) + .build(); + + await worker.start(); + + // Client + const client = createAzureManagedClient(connectionString); + const instanceId = await client.scheduleNewOrchestration("myOrchestration", "World"); + const state = await client.waitForOrchestrationCompletion(instanceId, true, 30); + console.log("Output:", state.serializedOutput); + + await client.stop(); + await worker.stop(); +} + +main().catch(console.error); +``` + diff --git a/plugin/skills/azure-prepare/references/services/durable-task-scheduler/python.md b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/python.md new file mode 100644 index 000000000..883935a6b --- /dev/null +++ b/plugin/skills/azure-prepare/references/services/durable-task-scheduler/python.md @@ -0,0 +1,219 @@ +# Durable Task Scheduler — Python + +## Learn More + +- [Durable Task Scheduler documentation](https://learn.microsoft.com/azure/durable-task-scheduler/) +- [Durable Functions Python guide](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-overview?tabs=python) + +## Durable Functions Setup + +### Required Packages + +```txt +# requirements.txt +azure-functions +azure-functions-durable +azure-identity +``` + +> **💡 Finding latest versions**: Run `pip index versions azure-functions-durable` or check [pypi.org/project/azure-functions-durable](https://pypi.org/project/azure-functions-durable/) for the latest stable release. + +### host.json + +```json +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.*, 5.0.0)" + } +} +``` + +> **💡 NOTE**: Python uses extension bundles, so the storage provider type is `durabletask-scheduler`. .NET isolated uses the NuGet package directly and requires `azureManaged` instead — see [dotnet.md](dotnet.md). + +### local.settings.json + +```json +{ + "IsEncrypted": false, + "Values": { + "FUNCTIONS_WORKER_RUNTIME": "python", + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None" + } +} +``` + +## Minimal Example + +```python +import azure.functions as func +import azure.durable_functions as df + +my_app = df.DFApp(http_auth_level=func.AuthLevel.FUNCTION) + +# HTTP Starter +@my_app.route(route="orchestrators/{function_name}", methods=["POST"]) +@my_app.durable_client_input(client_name="client") +async def http_start(req: func.HttpRequest, client): + function_name = req.route_params.get('function_name') + instance_id = await client.start_new(function_name) + return client.create_check_status_response(req, instance_id) + +# Orchestrator +@my_app.orchestration_trigger(context_name="context") +def my_orchestration(context: df.DurableOrchestrationContext): + result1 = yield context.call_activity("say_hello", "Tokyo") + result2 = yield context.call_activity("say_hello", "Seattle") + return f"{result1}, {result2}" + +# Activity +@my_app.activity_trigger(input_name="name") +def say_hello(name: str) -> str: + return f"Hello {name}!" +``` + +## Workflow Patterns + +### Fan-Out/Fan-In + +```python +@my_app.orchestration_trigger(context_name="context") +def fan_out_fan_in(context: df.DurableOrchestrationContext): + cities = ["Tokyo", "Seattle", "London", "Paris", "Berlin"] + + # Fan-out: schedule all in parallel + parallel_tasks = [] + for city in cities: + task = context.call_activity("say_hello", city) + parallel_tasks.append(task) + + # Fan-in: wait for all + results = yield context.task_all(parallel_tasks) + return results +``` + +### Human Interaction + +```python +import datetime + +@my_app.orchestration_trigger(context_name="context") +def approval_workflow(context: df.DurableOrchestrationContext): + yield context.call_activity("send_approval_request", context.get_input()) + + # Wait for approval event with timeout + timeout = context.current_utc_datetime + datetime.timedelta(days=3) + approval_task = context.wait_for_external_event("ApprovalEvent") + timeout_task = context.create_timer(timeout) + + winner = yield context.task_any([approval_task, timeout_task]) + + if winner == approval_task: + approved = approval_task.result + return "Approved" if approved else "Rejected" + return "Timed out" +``` + +## Orchestration Determinism + +| ❌ NEVER | ✅ ALWAYS USE | +|----------|--------------| +| `datetime.now()` | `context.current_utc_datetime` | +| `uuid.uuid4()` | `context.new_uuid()` | +| `random.random()` | Pass random values from activities | +| `time.sleep()` | `context.create_timer()` | +| Direct I/O, HTTP, database | `context.call_activity()` | + +### Replay-Safe Logging + +```python +import logging + +@my_app.orchestration_trigger(context_name="context") +def my_orchestration(context: df.DurableOrchestrationContext): + # Check if replaying to avoid duplicate logs + if not context.is_replaying: + logging.info("Started") # Only logs once, not on replay + result = yield context.call_activity("my_activity", "input") + return result +``` + +## Error Handling & Retry + +```python +retry_options = df.RetryOptions( + first_retry_interval_in_milliseconds=5000, + max_number_of_attempts=3, + backoff_coefficient=2.0, + max_retry_interval_in_milliseconds=60000 +) + +@my_app.orchestration_trigger(context_name="context") +def workflow_with_retry(context: df.DurableOrchestrationContext): + try: + result = yield context.call_activity_with_retry( + "unreliable_service", + retry_options, + context.get_input() + ) + return result + except Exception as ex: + context.set_custom_status({"error": str(ex)}) + yield context.call_activity("compensation_activity", context.get_input()) + return "Compensated" +``` + +## Durable Task SDK (Non-Functions) + +For applications running outside Azure Functions (containers, VMs, Azure Container Apps, Azure Kubernetes Service): + +```python +import asyncio +from durabletask.azuremanaged.worker import DurableTaskSchedulerWorker + +# Activity function +def say_hello(ctx, name: str) -> str: + return f"Hello {name}!" + +# Orchestrator function +def my_orchestration(ctx, name: str) -> str: + result = yield ctx.call_activity('say_hello', input=name) + return result + +async def main(): + with DurableTaskSchedulerWorker( + host_address="http://localhost:8080", + secure_channel=False, + taskhub="default" + ) as worker: + worker.add_activity(say_hello) + worker.add_orchestrator(my_orchestration) + worker.start() + + # Client + from durabletask.azuremanaged.client import DurableTaskSchedulerClient + client = DurableTaskSchedulerClient( + host_address="http://localhost:8080", + taskhub="default", + token_credential=None, + secure_channel=False + ) + instance_id = client.schedule_new_orchestration("my_orchestration", input="World") + result = client.wait_for_orchestration_completion(instance_id, timeout=30) + print(f"Output: {result.serialized_output}") + +if __name__ == "__main__": + asyncio.run(main()) +``` + diff --git a/plugin/skills/azure-prepare/references/services/functions/durable.md b/plugin/skills/azure-prepare/references/services/functions/durable.md index 23a9d8ab9..7124ba829 100644 --- a/plugin/skills/azure-prepare/references/services/functions/durable.md +++ b/plugin/skills/azure-prepare/references/services/functions/durable.md @@ -2,6 +2,18 @@ Long-running orchestrations with checkpointing and state management. +## Storage Backend + +Durable Functions requires a storage backend to persist orchestration state, queues, and history. + +| Backend | Recommendation | Notes | +|---------|---------------|-------| +| **Durable Task Scheduler** | **✅ Required** | Fully managed Azure service. Best performance, lowest latency, built-in monitoring. Supports all languages (.NET, Python, Java, JavaScript). | +| Azure Storage (default) | ⛔ Do not use | Legacy backend. Higher latency, harder to monitor, no built-in RBAC. Not permitted for new projects. | +| MSSQL | Specialized | SQL Server backend. Only when SQL infrastructure already exists and user explicitly requests it. | + +> **⛔ MANDATORY**: You **MUST** use **Durable Task Scheduler** as the storage backend for ALL new Durable Functions projects. Do NOT use Azure Storage as the backend — it is legacy and not permitted. Generate the DTS Bicep module (`Microsoft.DurableTask/schedulers` + `taskHubs`), RBAC role assignment (Durable Task Data Contributor), and `DURABLE_TASK_SCHEDULER_CONNECTION_STRING` app setting. See the [Durable Task Scheduler reference](../durable-task-scheduler/README.md) for setup, language-specific guidance, and Bicep templates. + ## When to Use - Multi-step workflows diff --git a/plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md b/plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md index 6ead0700f..f24ebc4c7 100644 --- a/plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md +++ b/plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md @@ -44,13 +44,19 @@ IF integration IN [http]: IF integration IN [timer]: → Source-only recipe. Skip to Step 5. -IF integration IN [durable, mcp]: +IF integration IN [mcp]: → Source-only recipe with storage configuration: - - Set `enableQueue: true` in main.bicep (required for Durable task hub and MCP) - - Set `enableTable: true` in main.bicep (required for Durable only; NOT required for MCP) + - Set `enableQueue: true` in main.bicep (required for MCP) Note: These are minimal parameter toggles, not structural changes to IaC. → Then skip to Step 5. +IF integration IN [durable]: + → Full recipe with Durable Task Scheduler backend: + - Add DTS IaC module (scheduler + task hub + RBAC). Continue to Step 3. + - Reference: [Durable Task Scheduler](../../../durable-task-scheduler/README.md) and [Bicep patterns](../../../durable-task-scheduler/bicep.md). + - Do NOT use Azure Storage queues/tables as the durable backend — always use Durable Task Scheduler. + → Continue to Step 3. + IF integration IN [cosmosdb, sql, servicebus, eventhubs, blob]: → Full recipe. Continue to Step 3. ``` @@ -296,15 +302,15 @@ Some integrations require additional storage endpoints. Toggle these in `main.bi | HTTP | ✓ | - | - | Default | | Timer | ✓ | - | - | Checkpointing uses blob | | Cosmos DB | ✓ | - | - | Standard | -| **Durable** | ✓ | **✓** | **✓** | Queue=task hub, Table=history | +| **Durable** | ✓ | - | - | Uses Durable Task Scheduler (not Storage queues/tables) | | **MCP** | ✓ | **✓** | - | Queue=state mgmt + backplane | ## Recipe Classification | Category | Integrations | What Recipe Provides | |----------|-------------|---------------------| -| **Source-only** | timer, durable, mcp | Source code snippet; may require minimal parameter toggles (e.g., `enableQueue`) but no new IaC modules | -| **Full recipe** | cosmosdb, sql, servicebus, eventhubs, blob | IaC modules + RBAC + networking + source code | +| **Source-only** | timer, mcp | Source code snippet; may require minimal parameter toggles (e.g., `enableQueue`) but no new IaC modules | +| **Full recipe** | cosmosdb, sql, servicebus, eventhubs, blob, durable | IaC modules + RBAC + networking + source code | ## Critical Rules diff --git a/plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/README.md b/plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/README.md index 83f50fed5..41a936092 100644 --- a/plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/README.md +++ b/plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/README.md @@ -1,6 +1,6 @@ # Durable Functions Recipe -Adds Durable Functions orchestration patterns to an Azure Functions base template. +Adds Durable Functions orchestration patterns to an Azure Functions base template with **Durable Task Scheduler** as the backend. ## Overview @@ -8,8 +8,9 @@ This recipe composes with any HTTP base template to create a Durable Functions a - **Orchestrator** - Coordinates workflow execution - **Activity** - Individual task units - **HTTP Client** - Starts and queries orchestrations +- **Durable Task Scheduler** - Fully managed backend for state persistence, orchestration history, and task hub management -No additional Azure resources required — uses the existing Storage account for state management. +> **⚠️ IMPORTANT**: This recipe uses **Durable Task Scheduler** (DTS) as the storage backend — NOT Azure Storage queues/tables. DTS is the recommended, fully managed option with the best performance and developer experience. See [Durable Task Scheduler reference](../../../../durable-task-scheduler/README.md) for details. ## Integration Type @@ -17,66 +18,121 @@ No additional Azure resources required — uses the existing Storage account for |--------|-------| | **Trigger** | `OrchestrationTrigger` + `ActivityTrigger` | | **Client** | `DurableClient` / `DurableOrchestrationClient` | -| **Auth** | N/A — internal orchestration | -| **IaC** | ⚠️ Set `enableQueue: true` and `enableTable: true` in main.bicep | +| **Auth** | UAMI (Managed Identity) → Durable Task Scheduler | +| **IaC** | Bicep module: scheduler + task hub + RBAC | -## ⚠️ CRITICAL: Storage Endpoint Flags +## Composition Steps + +Apply these steps AFTER `azd init -t functions-quickstart-{lang}-azd`: + +| # | Step | Details | +|---|------|---------| +| 1 | **Add IaC module** | Copy `bicep/durable-task-scheduler.bicep` → `infra/app/durable-task-scheduler.bicep` | +| 2 | **Wire into main** | Add module reference in `infra/main.bicep` | +| 3 | **Add app settings** | Add `DURABLE_TASK_SCHEDULER_CONNECTION_STRING` to function app configuration | +| 4 | **Add extension packages** | Add Durable Functions + DTS extension packages for the runtime | +| 5 | **Replace source code** | Add Orchestrator + Activity + Client from `source/{lang}.md` | +| 6 | **Configure host.json** | Set DTS storage provider (see [DTS language references](../../../../durable-task-scheduler/README.md)) | -Durable Functions requires Queue and Table storage for the task hub and history. The base template supports this via flags: +## IaC Module -### Enable in main.bicep +### Bicep -Set these flags in the storage module parameters: +Copy `bicep/durable-task-scheduler.bicep` → `infra/app/durable-task-scheduler.bicep` and add to `main.bicep`: ```bicep -module storage './shared/storage.bicep' = { +module durableTaskScheduler './app/durable-task-scheduler.bicep' = { + name: 'durableTaskScheduler' + scope: rg params: { - enableBlob: true // Default - deployment packages - enableQueue: true // REQUIRED for Durable - task hub messages - enableTable: true // REQUIRED for Durable - orchestration history + name: name + location: location + tags: tags + functionAppPrincipalId: app.outputs.SERVICE_API_IDENTITY_PRINCIPAL_ID + principalId: principalId // For dashboard access + uamiClientId: apiUserAssignedIdentity.outputs.clientId // REQUIRED for UAMI auth } } ``` -When these flags are `true`, the base template automatically: -1. Adds `AzureWebJobsStorage__queueServiceUri` app setting -2. Adds `AzureWebJobsStorage__tableServiceUri` app setting -3. Assigns `Storage Queue Data Contributor` RBAC role to UAMI -4. Assigns `Storage Table Data Contributor` RBAC role to UAMI +### App Settings -### What the Flags Control +Add the DTS connection string to the function app's `appSettings`: -| Flag | App Setting Added | RBAC Role Added | -|------|-------------------|-----------------| -| `enableQueue: true` | `AzureWebJobsStorage__queueServiceUri` | Storage Queue Data Contributor | -| `enableTable: true` | `AzureWebJobsStorage__tableServiceUri` | Storage Table Data Contributor | +```bicep +appSettings: { + DURABLE_TASK_SCHEDULER_CONNECTION_STRING: durableTaskScheduler.outputs.connectionString +} +``` -> **Note:** If these flags are missing or `false`, Durable Functions will fail with 503 errors. +> **💡 TIP**: The module output already includes `ClientID=` when you pass `uamiClientId` — no manual connection string construction needed. -## Composition Steps +> **⚠️ NOTE**: Do NOT set `enableQueue: true` or `enableTable: true` in the storage module — DTS replaces Azure Storage queues/tables for orchestration state. -Apply these steps AFTER `azd init -t functions-quickstart-{lang}-azd`: +## RBAC Roles Required -| # | Step | Details | -|---|------|---------| -| 1 | **Add extension** | Add Durable Functions extension package | -| 2 | **Replace source code** | Add Orchestrator + Activity + Client from `source/{lang}.md` | -| 3 | **Configure host.json** | Optional: tune concurrency settings | +| Role | GUID | Scope | Purpose | +|------|------|-------|---------| +| **Durable Task Data Contributor** | `0ad04412-c4d5-4796-b79c-f76d14c8d402` | Durable Task Scheduler | Read/write orchestrations and entities | + +## host.json Configuration + +The `host.json` must configure DTS as the storage provider. The `type` value differs by language: + +| Language | `storageProvider.type` | Reference | +|----------|----------------------|-----------| +| C# (.NET) | `azureManaged` | [dotnet.md](../../../../durable-task-scheduler/dotnet.md) | +| Python | `durabletask-scheduler` | [python.md](../../../../durable-task-scheduler/python.md) | +| JavaScript/TypeScript | `durabletask-scheduler` | [javascript.md](../../../../durable-task-scheduler/javascript.md) | +| Java | `durabletask-scheduler` | [java.md](../../../../durable-task-scheduler/java.md) | + +**Example (Python / JavaScript / Java):** +```json +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +**Example (.NET isolated):** +```json +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` ## Extension Packages -| Language | Package | -|----------|---------| -| Python | `azure-functions-durable` | -| TypeScript/JavaScript | `durable-functions` | -| C# (.NET) | `Microsoft.Azure.Functions.Worker.Extensions.DurableTask` | -| Java | `com.microsoft:durabletask-azure-functions` | -| PowerShell | Built-in (v2 bundles) | +| Language | Durable Functions Package | DTS Extension Package | +|----------|--------------------------|----------------------| +| Python | `azure-functions-durable` | _(uses extension bundles)_ | +| TypeScript/JavaScript | `durable-functions` | _(uses extension bundles)_ | +| C# (.NET) | `Microsoft.Azure.Functions.Worker.Extensions.DurableTask` | `Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureManaged` | +| Java | `com.microsoft:durabletask-azure-functions` | _(uses extension bundles)_ | +| PowerShell | Built-in (v2 bundles) | _(uses extension bundles)_ | ## Files | Path | Description | |------|-------------| +| [bicep/durable-task-scheduler.bicep](bicep/durable-task-scheduler.bicep) | DTS Bicep module (scheduler + task hub + RBAC) | | [source/python.md](source/python.md) | Python Durable Functions source code | | [source/typescript.md](source/typescript.md) | TypeScript Durable Functions source code | | [source/javascript.md](source/javascript.md) | JavaScript Durable Functions source code | @@ -104,33 +160,27 @@ HTTP Start → Orchestrator → [Activity1, Activity2, Activity3] → Aggregate ## Common Issues -### Storage Connection Error (503 "Function host is not running") +### 403 PermissionDenied on gRPC call -**Symptoms:** 503 "Function host is not running", or "Storage Queue connection failed" +**Symptoms:** 403 on `client.start_new()` or orchestration calls. -**Cause:** `enableQueue` and `enableTable` flags are not set to `true` in main.bicep. +**Cause:** Function App managed identity lacks RBAC on the DTS scheduler, or IP allowlist blocks traffic. -**Solution:** Set both flags to `true` in the storage module and redeploy: -```bicep -enableQueue: true -enableTable: true -``` +**Solution:** +1. Assign `Durable Task Data Contributor` role (`0ad04412-c4d5-4796-b79c-f76d14c8d402`) to the UAMI scoped to the scheduler. +2. Ensure the connection string includes `ClientID=`. +3. Ensure the scheduler's `ipAllowlist` includes `0.0.0.0/0` (empty list denies all traffic). +4. RBAC propagation can take up to 10 minutes — restart the Function App after assigning roles. + +### TaskHub not found + +**Cause:** Task hub not provisioned or name mismatch. + +**Solution:** Ensure the `TaskHub` parameter in `DURABLE_TASK_SCHEDULER_CONNECTION_STRING` matches the provisioned task hub name (default: `default`). ### Orchestrator Replay Issues **Cause:** Non-deterministic code in orchestrator (e.g., `DateTime.Now`, random values). **Solution:** Use `context.current_utc_datetime` or `context.CurrentUtcDateTime` instead. - -## host.json Configuration (Optional) - -```json -{ - "extensions": { - "durableTask": { - "maxConcurrentActivityFunctions": 10, - "maxConcurrentOrchestratorFunctions": 5 - } - } -} ``` diff --git a/plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/bicep/durable-task-scheduler.bicep b/plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/bicep/durable-task-scheduler.bicep new file mode 100644 index 000000000..664505ce9 --- /dev/null +++ b/plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/bicep/durable-task-scheduler.bicep @@ -0,0 +1,116 @@ +// recipes/durable/bicep/durable-task-scheduler.bicep +// Durable Task Scheduler recipe module — adds DTS scheduler, task hub, and RBAC +// to an Azure Functions base template. +// +// USAGE: Add this as a module in your main.bicep: +// module durableTaskScheduler './app/durable-task-scheduler.bicep' = { +// name: 'durableTaskScheduler' +// scope: rg +// params: { +// name: name +// location: location +// tags: tags +// functionAppPrincipalId: app.outputs.SERVICE_API_IDENTITY_PRINCIPAL_ID +// principalId: principalId +// uamiClientId: apiUserAssignedIdentity.outputs.clientId +// } +// } +// +// Then add the connection string app setting to the function app: +// appSettings: { +// DURABLE_TASK_SCHEDULER_CONNECTION_STRING: durableTaskScheduler.outputs.connectionString +// } + +targetScope = 'resourceGroup' + +@description('Base name for resources') +param name string + +@description('Azure region') +param location string = resourceGroup().location + +@description('Resource tags') +param tags object = {} + +@description('Principal ID of the Function App managed identity (UAMI)') +param functionAppPrincipalId string + +@description('Principal ID of the deploying user (for dashboard access). Set via AZURE_PRINCIPAL_ID.') +param principalId string = '' + +@description('UAMI client ID from base template identity module - REQUIRED for UAMI auth') +param uamiClientId string + +@allowed(['Consumption', 'Dedicated']) +@description('Use Consumption for quickstarts/variable workloads, Dedicated for high-demand/predictable throughput') +param skuName string = 'Consumption' + +// ============================================================================ +// Naming +// ============================================================================ +var resourceSuffix = take(uniqueString(subscription().id, resourceGroup().name, name), 6) +var schedulerName = 'dts-${name}-${resourceSuffix}' + +// ============================================================================ +// Durable Task Scheduler +// ============================================================================ +resource scheduler 'Microsoft.DurableTask/schedulers@2025-11-01' = { + name: schedulerName + location: location + tags: tags + properties: { + sku: { name: skuName } + ipAllowlist: ['0.0.0.0/0'] // Required: empty list denies all traffic + } +} + +// ============================================================================ +// Task Hub +// ============================================================================ +resource taskHub 'Microsoft.DurableTask/schedulers/taskHubs@2025-11-01' = { + parent: scheduler + name: 'default' +} + +// ============================================================================ +// RBAC — Durable Task Data Contributor for Function App +// ============================================================================ +var durableTaskDataContributorRoleId = '0ad04412-c4d5-4796-b79c-f76d14c8d402' + +resource functionAppRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(scheduler.id, functionAppPrincipalId, durableTaskDataContributorRoleId) + scope: scheduler + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', durableTaskDataContributorRoleId) + principalId: functionAppPrincipalId + principalType: 'ServicePrincipal' + } +} + +// ============================================================================ +// RBAC — Dashboard Access for Deploying User +// ============================================================================ +resource dashboardRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (!empty(principalId)) { + name: guid(scheduler.id, principalId, durableTaskDataContributorRoleId) + scope: scheduler + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', durableTaskDataContributorRoleId) + principalId: principalId + principalType: 'User' + } +} + +// ============================================================================ +// Outputs +// ============================================================================ +output schedulerName string = scheduler.name +output schedulerEndpoint string = scheduler.properties.endpoint +output taskHubName string = taskHub.name +output connectionString string = 'Endpoint=${scheduler.properties.endpoint};Authentication=ManagedIdentity;ClientID=${uamiClientId};TaskHub=${taskHub.name}' + +// ============================================================================ +// APP SETTINGS OUTPUT - Use this to ensure correct UAMI configuration +// ============================================================================ +output appSettings object = { + DURABLE_TASK_SCHEDULER_CONNECTION_STRING: 'Endpoint=${scheduler.properties.endpoint};Authentication=ManagedIdentity;ClientID=${uamiClientId};TaskHub=${taskHub.name}' +} diff --git a/plugin/skills/azure-prepare/references/services/functions/templates/selection.md b/plugin/skills/azure-prepare/references/services/functions/templates/selection.md index b3195e538..a7421f3f5 100644 --- a/plugin/skills/azure-prepare/references/services/functions/templates/selection.md +++ b/plugin/skills/azure-prepare/references/services/functions/templates/selection.md @@ -39,9 +39,15 @@ Cross-reference with [top Azure Functions scenarios](https://learn.microsoft.com Recipe: recipes/servicebus/ ✅ Available 7. Is it for orchestration or workflows? - Indicators: DurableOrchestrationTrigger, orchestrator, durable_functions - └─► YES → HTTP base + durable source snippet (toggle enableQueue + enableTable in base) + Code indicators: DurableOrchestrationTrigger, orchestrator, durable_functions + Natural language indicators (NEW projects): workflow, multi-step, pipeline, + orchestration, fan-out, fan-in, long-running process, chaining, state machine, + saga, order processing, approval flow + └─► YES → HTTP base + durable recipe (IaC: Durable Task Scheduler + task hub + RBAC + source) + ⛔ REQUIRED: Generate Microsoft.DurableTask/schedulers + taskHubs Bicep resources Recipe: recipes/durable/ ✅ Available + References: [durable.md](../../functions/durable.md) for storage backend rules, + [Durable Task Scheduler](../../durable-task-scheduler/README.md) for Bicep patterns and connection string 8. Does it use Event Hubs? Indicators: EventHubTrigger, @app.event_hub, event_hub_output @@ -71,5 +77,6 @@ Cross-reference with [top Azure Functions scenarios](https://learn.microsoft.com | Type | IaC Delta? | Examples | |------|-----------|----------| | **Full recipe** | Yes — Bicep module + Terraform module + RBAC + networking | cosmosdb, servicebus, eventhubs | +| **Full recipe (Bicep only)** | Yes — Bicep module + RBAC | durable | | **AZD template** | Use dedicated AZD template from Awesome AZD | sql, blob-eventgrid | -| **Source-only** | No — only replace function source code (may toggle storage params) | timer, durable, mcp | +| **Source-only** | No — only replace function source code (may toggle storage params) | timer, mcp | diff --git a/plugin/skills/azure-prepare/references/specialized-routing.md b/plugin/skills/azure-prepare/references/specialized-routing.md index e0e23907a..5f2847818 100644 --- a/plugin/skills/azure-prepare/references/specialized-routing.md +++ b/plugin/skills/azure-prepare/references/specialized-routing.md @@ -10,7 +10,8 @@ |----------|---------------------|--------------------|-----------------------------| | **1 (highest)** | Lambda, AWS Lambda, migrate AWS, migrate GCP, Lambda to Functions, migrate from AWS, migrate from GCP | **azure-cloud-migrate** | Phase 1 Step 4 (Select Recipe) — azure-cloud-migrate does assessment + code conversion, then azure-prepare takes over for infrastructure, local testing, or deployment | | 2 | copilot SDK, copilot app, copilot-powered, @github/copilot-sdk, CopilotClient, sendAndWait, copilot-sdk-service | **azure-hosted-copilot-sdk** | Phase 1 Step 4 (Select Recipe) | -| 3 (lowest) | Azure Functions, function app, serverless function, timer trigger, HTTP trigger, queue trigger, func new, func start | Stay in **azure-prepare** | Phase 1 Step 4 (Select Recipe) — prefer Azure Functions templates | +| 3 | Azure Functions, function app, serverless function, timer trigger, HTTP trigger, queue trigger, func new, func start | Stay in **azure-prepare** | Phase 1 Step 4 (Select Recipe) — prefer Azure Functions templates | +| 4 (lowest) | workflow, orchestration, multi-step, pipeline, fan-out/fan-in, saga, long-running process, durable | Stay in **azure-prepare** | Phase 1 Step 4 — select **durable** recipe. **MUST** load [durable.md](services/functions/durable.md) and [DTS reference](services/durable-task-scheduler/README.md). Generate `Microsoft.DurableTask/schedulers` + `taskHubs` Bicep resources. | > ⚠️ This checks the user's **prompt text**, not just existing code. Essential for greenfield projects where there is no codebase to scan. diff --git a/tests/azure-deploy/integration.test.ts b/tests/azure-deploy/integration.test.ts index 9573f29f0..1a4ed66c4 100644 --- a/tests/azure-deploy/integration.test.ts +++ b/tests/azure-deploy/integration.test.ts @@ -12,11 +12,11 @@ import { shouldSkipIntegrationTests, getIntegrationSkipReason, - useAgentRunner + useAgentRunner, } from "../utils/agent-runner"; import { hasDeployLinks, softCheckDeploySkills, softCheckContainerDeployEnvVars } from "./utils"; import { cloneRepo } from "../utils/git-clone"; -import { expectFiles, softCheckSkill } from "../utils/evaluate"; +import { expectFiles, softCheckSkill, doesWorkspaceFileIncludePattern } from "../utils/evaluate"; const SKILL_NAME = "azure-deploy"; const RUNS_PER_PROMPT = 1; @@ -261,6 +261,36 @@ describeIntegration(`${SKILL_NAME}_ - Integration Tests`, () => { }, deployTestTimeoutMs); }); + // Durable Task Scheduler (Durable Functions with DTS) + describe("durable-task-scheduler-deploy", () => { + test("creates and deploys workflow app with Durable Task Scheduler", async () => { + let workspacePath: string | undefined; + + const agentMetadata = await agent.run({ + setup: async (workspace: string) => { + workspacePath = workspace; + }, + prompt: "Create a workflow app that orchestrates a multi-step order processing pipeline and deploy to Azure using my current subscription in eastus2 region.", + nonInteractive: true, + followUp: FOLLOW_UP_PROMPT, + preserveWorkspace: true + }); + + softCheckDeploySkills(agentMetadata); + expect(workspacePath).toBeDefined(); + expectFiles(workspacePath!, [/infra\/.*\.bicep$/], [/\.tf$/]); + + // Verify DTS-specific Bicep content on disk + const bicepPattern = /\.bicep$/; + expect(doesWorkspaceFileIncludePattern(workspacePath!, /Microsoft\.DurableTask\/schedulers/i, bicepPattern)).toBe(true); + expect(doesWorkspaceFileIncludePattern(workspacePath!, /Microsoft\.DurableTask\/schedulers\/taskHubs/i, bicepPattern)).toBe(true); + expect(doesWorkspaceFileIncludePattern(workspacePath!, /0ad04412-c4d5-4796-b79c-f76d14c8d402/i, bicepPattern)).toBe(true); + + const containsDeployLinks = hasDeployLinks(agentMetadata); + expect(containsDeployLinks).toBe(true); + }, deployTestTimeoutMs); + }); + // Azure Container Apps (ACA) describe("azure-container-apps-deploy", () => { test("creates containerized web application", async () => { diff --git a/tests/azure-prepare/integration.test.ts b/tests/azure-prepare/integration.test.ts index 79997766a..8e41ceff4 100644 --- a/tests/azure-prepare/integration.test.ts +++ b/tests/azure-prepare/integration.test.ts @@ -842,4 +842,73 @@ describeIntegration(`${SKILL_NAME}_ - Integration Tests`, () => { expect(/Authentication\s*=\s*Active\s+Directory\s+(Default|Managed\s+Identity)/i.test(allFileContents)).toBe(true); }); }); + + describe("durable-task-scheduler", () => { + test("generates Durable Task Scheduler infrastructure and workflow code for a workflow app", async () => { + let workspacePath: string | undefined; + + const agentMetadata = await agent.run({ + setup: async (workspace: string) => { + workspacePath = workspace; + }, + prompt: + "Prepare the Azure deployment infrastructure for a new workflow app " + + "that will orchestrate a multi-step order processing pipeline. " + + "Generate the Bicep templates, RBAC assignments, and azure.yaml. " + + "Use the eastus2 region and my current subscription.", + nonInteractive: true, + followUp: FOLLOW_UP_PROMPT, + preserveWorkspace: true, + shouldEarlyTerminate: (metadata) => + hasPlanReadyForValidation(metadata) || hasValidationCommand(metadata) || isSkillInvoked(metadata, "azure-validate"), + }); + + // Preconditions + expect(workspacePath).toBeDefined(); + expect(isSkillInvoked(agentMetadata, SKILL_NAME)).toBe(true); + + // Collect all file contents the agent wrote via create tool calls + const createCalls = getToolCalls(agentMetadata, "create"); + + // Gather all Bicep file contents + const bicepContents = createCalls + .filter(event => { + const args = (event.data as Record).arguments as { path?: string } | undefined; + return args?.path?.endsWith(".bicep"); + }) + .map(event => { + const args = (event.data as Record).arguments as { file_text?: string }; + return args?.file_text ?? ""; + }); + const bicepContent = bicepContents.join("\n"); + expect(bicepContent.length).toBeGreaterThan(0); + + // Must provision a Durable Task Scheduler resource + expect(/Microsoft\.DurableTask\/schedulers/i.test(bicepContent)).toBe(true); + + // Must provision a task hub child resource + expect(/Microsoft\.DurableTask\/schedulers\/taskHubs/i.test(bicepContent)).toBe(true); + + // Must assign the Durable Task Data Contributor RBAC role (role ID: 0ad04412-c4d5-4796-b79c-f76d14c8d402) + expect(/0ad04412-c4d5-4796-b79c-f76d14c8d402/i.test(bicepContent)).toBe(true); + + // Must include the scheduler connection string app setting + const allFileContents = createCalls + .map(event => { + const args = (event.data as Record).arguments as { file_text?: string }; + return args?.file_text ?? ""; + }) + .join("\n"); + expect(/DURABLE_TASK_SCHEDULER_CONNECTION_STRING/i.test(allFileContents)).toBe(true); + + // Must include ipAllowlist to avoid 403 errors (empty list denies all traffic) + expect(/ipAllowlist/i.test(bicepContent)).toBe(true); + + // Workspace should contain orchestration/workflow code files + expectFiles(workspacePath!, + [/plan\.md$/, /azure\.yaml$/, /infra\/.*\.bicep$/], + [/\.tf$/], + ); + }); + }); });