diff --git a/plugin/skills/microsoft-foundry/SKILL.md b/plugin/skills/microsoft-foundry/SKILL.md index 11684224a..f098d55b4 100644 --- a/plugin/skills/microsoft-foundry/SKILL.md +++ b/plugin/skills/microsoft-foundry/SKILL.md @@ -1,579 +1,152 @@ --- name: microsoft-foundry -description: 'Expert in Microsoft Foundry: use this skill to help discover and deploy models, build RAG applications with knowledge indexes, create and evaluate AI agents, and troubleshoot common issues in Microsoft Foundry platform.' +description: >- + Microsoft Foundry is the unified AI platform for building, deploying, and managing AI assets. + Use this skill for: (1) Deploying AI models from the model catalog (OpenAI, Meta, Microsoft), + (2) Creating AI agents with tools (web search, functions, file search), (3) Building RAG applications + with knowledge indexes, (4) Evaluating agent performance and safety, (5) Monitoring AI assets in production. + Trigger phrases: "create a model", "deploy a model", "create an agent", "build an AI agent", + "RAG application", "knowledge index", "evaluate my agent", "AI observability", "model catalog", + "deploy GPT-4", "create AI assistant", "agent evaluation", "AI monitoring". +metadata: + author: microsoft + version: "2.0" +compatibility: Requires Azure CLI (az) and optionally Azure Developer CLI (azd) --- -# Microsoft Foundry Skill +# Microsoft Foundry -This skill helps developers work with Microsoft Foundry resources, covering model discovery and deployment, RAG (Retrieval-Augmented Generation) applications, AI agent creation, evaluation workflows, and troubleshooting. +**The unified AI platform for models, agents, and AI applications.** -## When to Use This Skill - -Use this skill when the user wants to: - -- **Discover and deploy AI models** from the Microsoft Foundry catalog -- **Build RAG applications** using knowledge indexes and vector search -- **Create AI agents** with tools like Azure AI Search, web search, or custom functions -- **Evaluate agent performance** using built-in evaluators -- **Set up monitoring** and continuous evaluation for production agents -- **Troubleshoot issues** with deployments, agents, or evaluations - -## Prerequisites - -### Azure Resources -- An Azure subscription with an active account -- Appropriate permissions to create Microsoft Foundry resources (e.g., Azure AI Owner role) -- Resource group for organizing Foundry resources +Microsoft Foundry enables you to discover and deploy AI models, create intelligent agents with tools, build RAG applications, and evaluate AI performance—all in one platform. -### Tools -- **Azure CLI** installed and authenticated (`az login`) -- **Azure Developer CLI (azd)** for deployment workflows (optional but recommended) - -### Language-Specific Requirements - -For SDK examples and implementation details in specific programming languages, refer to: -- **Python**: See [language/python.md](language/python.md) for Python SDK setup, authentication, and examples - -## Core Workflows - -### 1. Getting Started - Model Discovery and Deployment +## When to Use This Skill -#### Use Case -A developer new to Microsoft Foundry wants to explore available models and deploy their first one. +| Category | Triggers | +|----------|----------| +| **Models** | "create a model", "deploy a model", "model catalog", "deploy GPT-4", "list models" | +| **Agents** | "create an agent", "build AI agent", "AI assistant", "agent with tools" | +| **RAG** | "RAG application", "knowledge index", "document search", "grounded AI" | +| **Evaluation** | "evaluate agent", "agent quality", "AI safety", "test my agent" | +| **Observability** | "AI monitoring", "agent metrics", "production monitoring" | -#### Step 1: List Available Resources +--- -First, help the user discover their Microsoft Foundry resources. +## Quick Start -**Using Azure CLI:** +### 1. Discover Models -##### Bash ```bash -# List all Microsoft Foundry resources in subscription -az resource list \ - --resource-type "Microsoft.CognitiveServices/accounts" \ - --query "[?kind=='AIServices'].{Name:name, ResourceGroup:resourceGroup, Location:location}" \ - --output table - -# List resources in a specific resource group -az resource list \ - --resource-group \ - --resource-type "Microsoft.CognitiveServices/accounts" \ - --output table +# List Foundry resources +az resource list --resource-type "Microsoft.CognitiveServices/accounts" \ + --query "[?kind=='AIServices'].{Name:name, RG:resourceGroup}" -o table ``` -**Using MCP Tools:** - -Use the `foundry_resource_get` MCP tool to get detailed information about a specific Foundry resource, or to list all resources if no name is provided. - -#### Step 2: Browse Model Catalog - -Help users discover available models, including information about free playground support. - -**Key Points to Explain:** -- Some models support **free playground** for prototyping without costs -- Models can be filtered by **publisher** (e.g., OpenAI, Meta, Microsoft) -- Models can be filtered by **license type** -- Model availability varies by region - -**Using MCP Tools:** - -Use the `foundry_models_list` MCP tool: -- List all models: `foundry_models_list()` -- List free playground models: `foundry_models_list(search-for-free-playground=true)` -- Filter by publisher: `foundry_models_list(publisher="OpenAI")` -- Filter by license: `foundry_models_list(license="MIT")` - -**Example Output Explanation:** -When listing models, explain to users: -- Models with free playground support can be used for prototyping at no cost -- Some models support GitHub token authentication for easy access -- Check model capabilities and pricing before production deployment - -#### Step 3: Deploy a Model - -Guide users through deploying a model to their Foundry resource. +**MCP Tools:** +- `foundry_models_list` - Browse model catalog +- `foundry_models_list(search-for-free-playground=true)` - Free playground models +- `foundry_models_list(publisher="OpenAI")` - Filter by publisher -**Using Azure CLI:** +### 2. Deploy a Model -##### Bash ```bash -# Deploy a model (e.g., gpt-4o) az cognitiveservices account deployment create \ - --name \ - --resource-group \ + --name --resource-group \ --deployment-name gpt-4o-deployment \ - --model-name gpt-4o \ - --model-version "2024-05-13" \ - --model-format OpenAI \ - --sku-capacity 10 \ - --sku-name Standard - -# Verify deployment status -az cognitiveservices account deployment show \ - --name \ - --resource-group \ - --deployment-name gpt-4o-deployment -``` - -**Using MCP Tools:** - -Use the `foundry_models_deploy` MCP tool with parameters: -- `resource-group`: Resource group name -- `deployment`: Deployment name -- `model-name`: Model to deploy (e.g., "gpt-4o") -- `model-format`: Format (e.g., "OpenAI") -- `azure-ai-services`: Foundry resource name -- `model-version`: Specific version -- `sku-capacity`: Capacity units -- `scale-type`: Scaling type - -**Deployment Verification:** -Explain that when deployment completes, `provisioningState` should be `Succeeded`. If it fails, common issues include: -- Insufficient quota -- Region capacity limitations -- Permission issues - -#### Step 4: Get Resource Endpoint - -Users need the project endpoint to connect their code to Foundry. - -**Using MCP Tools:** - -Use the `foundry_resource_get` MCP tool to retrieve resource details including the endpoint. - -**Expected Output:** -The endpoint will be in format: `https://.services.ai.azure.com/api/projects/` - -Save this endpoint as it's needed for subsequent API and SDK calls. - -### 2. Building RAG Applications with Knowledge Indexes - -#### Use Case -A developer wants to build a Retrieval-Augmented Generation (RAG) application using their own documents. - -#### Understanding RAG and Knowledge Indexes - -**Explain the Concept:** -RAG enhances AI responses by: -1. **Retrieving** relevant documents from a knowledge base -2. **Augmenting** the AI prompt with retrieved context -3. **Generating** responses grounded in factual information - -**Knowledge Index Benefits:** -- Supports keyword, semantic, vector, and hybrid search -- Enables efficient retrieval of relevant content -- Stores metadata for better citations (document titles, URLs, file names) -- Integrates with Azure AI Search for production scenarios - -#### Step 1: List Existing Knowledge Indexes - -**Using MCP Tools:** - -Use `foundry_knowledge_index_list` with your project endpoint to list knowledge indexes. - -#### Step 2: Inspect Index Schema - -Understanding the index structure helps optimize queries. - -**Using MCP Tools:** - -Use the `foundry_knowledge_index_schema` MCP tool with your project endpoint and index name to get detailed schema information. - -**Schema Information Includes:** -- Field definitions and data types -- Searchable attributes -- Vectorization configuration -- Retrieval mode support (keyword, semantic, vector, hybrid) - -#### Step 3: Create an Agent with Azure AI Search Tool - -**Implementation:** - -To create a RAG agent with Azure AI Search tool integration: - -1. **Initialize the AI Project Client** with your project endpoint and credentials -2. **Get the Azure AI Search connection** from your project -3. **Create the agent** with: - - Agent name - - Model deployment - - Clear instructions (see best practices below) - - Azure AI Search tool configuration with: - - Connection ID - - Index name - - Query type (HYBRID recommended) - -**For SDK Implementation:** See [language/python.md](language/python.md#rag-applications-with-python-sdk) - -**Key Best Practices:** -- **Always request citations** in agent instructions -- Use **hybrid search** (AzureAISearchQueryType.HYBRID) for best results -- Instruct the agent to say "I don't know" when information isn't in the index -- Format citations consistently for easy parsing - -#### Step 4: Test the RAG Agent - -**Testing Process:** - -1. **Query the agent** with a test question -2. **Stream the response** to get real-time output -3. **Capture citations** from the response annotations -4. **Validate** that citations are properly formatted and included - -**For SDK Implementation:** See [language/python.md](language/python.md#testing-the-rag-agent) - -**Troubleshooting RAG Issues:** - -| Issue | Possible Cause | Resolution | -|-------|---------------|------------| -| No citations in response | Agent instructions don't request citations | Update instructions to explicitly request citation format | -| "Index not found" error | Wrong index name or connection | Verify `AI_SEARCH_INDEX_NAME` matches index in Azure AI Search | -| 401/403 authentication error | Missing RBAC permissions | Assign project managed identity **Search Index Data Contributor** role | -| Poor retrieval quality | Query type not optimal | Try HYBRID query type for better results | - -### 3. Creating Your First AI Agent - -#### Use Case -A developer wants to create an AI agent with tools (web search, function calling, file search). - -#### Step 1: List Existing Agents - -**Using MCP Tools:** - -Use `foundry_agents_list` with your project endpoint to list existing agents. - -#### Step 2: Create a Basic Agent - -**Implementation:** - -Create an agent with: -- **Model deployment name**: The model to use -- **Agent name**: Unique identifier -- **Instructions**: Clear, specific guidance for the agent's behavior - -**For SDK Implementation:** See [language/python.md](language/python.md#basic-agent) - -#### Step 3: Create an Agent with Custom Function Tools - -Agents can call custom functions to perform actions like querying databases, calling APIs, or performing calculations. - -**Implementation Steps:** - -1. **Define custom functions** with clear docstrings describing their purpose and parameters -2. **Create a function toolset** with your custom functions -3. **Create the agent** with the toolset and instructions on when to use the tools - -**For SDK Implementation:** See [language/python.md](language/python.md#agent-with-custom-function-tools) - -#### Step 4: Create an Agent with Web Search - -**Implementation:** - -Create an agent with web search capabilities by adding a Web Search tool: -- Optionally specify user location for localized results -- Provide instructions to always cite web sources - -**For SDK Implementation:** See [language/python.md](language/python.md#agent-with-web-search) - -#### Step 5: Interact with the Agent - -**Interaction Process:** - -1. **Create a conversation thread** for the agent interaction -2. **Add user messages** to the thread -3. **Run the agent** to process the messages and generate responses -4. **Check run status** for success or failure -5. **Retrieve messages** to see the agent's responses -6. **Cleanup** by deleting the agent when done - -**For SDK Implementation:** See [language/python.md](language/python.md#interacting-with-agents) - -**Agent Best Practices:** - -1. **Clear Instructions**: Provide specific, actionable instructions -2. **Tool Selection**: Only include tools the agent needs -3. **Error Handling**: Always check `run.status` for failures -4. **Cleanup**: Delete agents/threads when done to manage costs -5. **Rate Limits**: Handle rate limit errors gracefully (status code 429) - - -### 4. Evaluating Agent Performance - -#### Use Case -A developer has built an agent and wants to evaluate its quality, safety, and performance. - -#### Understanding Agent Evaluators - -**Built-in Evaluators:** - -1. **IntentResolutionEvaluator**: Measures how well the agent identifies and understands user requests (score 1-5) -2. **TaskAdherenceEvaluator**: Evaluates whether responses adhere to assigned tasks and system instructions (score 1-5) -3. **ToolCallAccuracyEvaluator**: Assesses whether the agent makes correct function tool calls (score 1-5) - -**Evaluation Output:** -Each evaluator returns: -- `{metric_name}`: Numerical score (1-5, higher is better) -- `{metric_name}_result`: "pass" or "fail" based on threshold -- `{metric_name}_threshold`: Binarization threshold (default or user-set) -- `{metric_name}_reason`: Explanation of the score - -#### Step 1: Single Agent Run Evaluation - -**Using MCP Tools:** - -Use the `foundry_agents_query_and_evaluate` MCP tool to query an agent and evaluate the response in one call. Provide: -- Agent ID -- Query text -- Project endpoint -- Azure OpenAI endpoint and deployment for evaluation -- Comma-separated list of evaluators to use - -**Example Output:** -```json -{ - "response": "The weather in Seattle is currently sunny and 22°C.", - "evaluation": { - "intent_resolution": 5.0, - "intent_resolution_result": "pass", - "intent_resolution_threshold": 3, - "intent_resolution_reason": "The agent correctly identified the user's intent to get weather information and provided a relevant response.", - "task_adherence": 4.0, - "task_adherence_result": "pass", - "tool_call_accuracy": 5.0, - "tool_call_accuracy_result": "pass" - } -} -``` - -#### Step 2: Evaluate Existing Response - -If you already have the agent's response, you can evaluate it directly. - -**Using MCP Tools:** - -Use the `foundry_agents_evaluate` MCP tool to evaluate a specific query/response pair with a single evaluator. - -**For SDK Implementation:** See [language/python.md](language/python.md#single-response-evaluation-using-mcp) - -#### Step 3: Batch Evaluation - -For evaluating multiple agent runs across multiple conversation threads: - -1. **Convert agent thread data** to evaluation format -2. **Prepare evaluation data** from multiple thread IDs -3. **Set up evaluators** with appropriate configuration -4. **Run batch evaluation** and view results in the Foundry portal - -**For SDK Implementation:** See [language/python.md](language/python.md#batch-evaluation) - -#### Interpreting Evaluation Results - -**Score Ranges (1-5 scale):** -- **5**: Excellent - Agent perfectly understood and executed the task -- **4**: Good - Minor issues, but overall successful -- **3**: Acceptable - Threshold for passing (default) -- **2**: Poor - Significant issues with understanding or execution -- **1**: Failed - Agent completely misunderstood or failed the task - -**Common Evaluation Issues:** - -| Issue | Cause | Resolution | -|-------|-------|------------| -| Job stuck in "Running" | Insufficient model capacity | Increase model quota/capacity and rerun | -| All metrics zero | Wrong evaluator or unsupported model | Verify evaluator compatibility with your model | -| Groundedness unexpectedly low | Incomplete context/retrieval | Verify RAG retrieval includes sufficient context | -| Evaluation missing | Not selected during setup | Rerun evaluation with required metrics | - -### 5. Troubleshooting Common Issues - -#### Deployment Issues - -**Problem: Deployment Stays Pending or Fails** - -##### Bash -```bash -# Check deployment status and details -az cognitiveservices account deployment show \ - --name \ - --resource-group \ - --deployment-name \ - --output json - -# Check account quota -az cognitiveservices account show \ - --name \ - --resource-group \ - --query "properties.quotaLimit" -``` - -**Common Causes:** -- Insufficient quota in the region -- Region at capacity for the model -- Permission issues - -**Resolution:** -1. Check quota limits in Azure Portal -2. Request quota increase if needed -3. Try deploying to a different region -4. Verify you have appropriate RBAC permissions - -#### Agent Response Issues - -**Problem: Agent Doesn't Return Citations (RAG)** - -**Diagnostics:** -1. Check agent instructions explicitly request citations -2. Verify the tool choice is set to "required" or "auto" -3. Confirm the Azure AI Search connection is configured correctly - -**Resolution:** - -Update the agent's instructions to explicitly request citations in the format `[message_idx:search_idx†source]` and to only use the knowledge base, never the agent's own knowledge. - -**For SDK Implementation:** See [language/python.md](language/python.md#update-agent-instructions) - -**Problem: "Index Not Found" Error** - -**Using MCP Tools:** - -Use the `foundry_knowledge_index_list` MCP tool to verify the index exists and get the correct name. - -**Resolution:** -1. Verify `AI_SEARCH_INDEX_NAME` environment variable matches actual index name -2. Check the connection points to correct Azure AI Search resource -3. Ensure index has been created and populated - -**Problem: 401/403 Authentication Errors** - -**Common Cause:** Missing RBAC permissions - -**Resolution:** - -##### Bash -```bash -# Assign Search Index Data Contributor role to managed identity -az role assignment create \ - --assignee \ - --role "Search Index Data Contributor" \ - --scope /subscriptions//resourceGroups//providers/Microsoft.Search/searchServices/ - -# Verify role assignment -az role assignment list \ - --assignee \ - --output table + --model-name gpt-4o --model-version "2024-05-13" \ + --model-format OpenAI --sku-name Standard --sku-capacity 10 ``` -#### Evaluation Issues - -**Problem: Evaluation Dashboard Shows No Data** - -**Common Causes:** -- No recent agent traffic -- Time range excludes the data -- Ingestion delay +**MCP Tool:** `foundry_models_deploy` -**Resolution:** -1. Generate new agent traffic (test queries) -2. Expand the time range filter in the dashboard -3. Wait a few minutes for data ingestion -4. Refresh the dashboard +### 3. Create an Agent -**Problem: Continuous Evaluation Not Running** +**MCP Tools:** +- `foundry_agents_list` - List existing agents +- Create agents via SDK - See [language/python.md](language/python.md#basic-agent) -**Diagnostics:** +### 4. Evaluate Agent -Check evaluation run status to identify issues. For SDK implementation, see [language/python.md](language/python.md#checking-evaluation-status). +**MCP Tools:** +- `foundry_agents_query_and_evaluate` - Query and evaluate in one call +- `foundry_agents_evaluate` - Evaluate existing response -**Resolution:** -1. Verify the evaluation rule is enabled -2. Confirm agent traffic is flowing -3. Check project managed identity has **Azure AI User** role -4. Verify OpenAI endpoint and deployment are accessible - -#### Rate Limiting and Capacity Issues - -**Problem: Agent Run Fails with Rate Limit Error** - -**Error Message:** `Rate limit is exceeded` or HTTP 429 - -**Resolution:** - -##### Bash -```bash -# Check current quota usage -az cognitiveservices usage list \ - --name \ - --resource-group - -# Request quota increase (manual process in portal) -echo "Request quota increase in Azure Portal under Quotas section" -``` +--- -# Request quota increase (manual process in portal) -Write-Output "Request quota increase in Azure Portal under Quotas section" -``` +## Core Capabilities -**Best Practices:** -- Implement exponential backoff retry logic -- Use Dynamic Quota when available -- Monitor quota usage proactively -- Consider multiple deployments across regions +### Models +- **Model Catalog**: Discover models from OpenAI, Meta, Microsoft, and more +- **Free Playground**: Prototype without costs on supported models +- **Deployment**: Deploy models with configurable capacity and SKU +- **MCP Tools**: `foundry_models_list`, `foundry_models_deploy`, `foundry_models_deployments_list` -## Quick Reference +### Agents +- **Tools**: Web search, Azure AI Search, custom functions, file search +- **Conversations**: Thread-based interactions with context +- **Best Practices**: Clear instructions, proper tool selection, cleanup +- **MCP Tools**: `foundry_agents_list`, `foundry_agents_connect` -### Common Environment Variables +### RAG Applications +- **Knowledge Indexes**: Vector search, hybrid search, semantic search +- **Azure AI Search Integration**: Production-ready retrieval +- **Citations**: Grounded responses with source attribution +- **MCP Tools**: `foundry_knowledge_index_list`, `foundry_knowledge_index_schema` -```bash -# Foundry Project -PROJECT_ENDPOINT=https://.services.ai.azure.com/api/projects/ -MODEL_DEPLOYMENT_NAME=gpt-4o +### Evaluation & Observability +- **Built-in Evaluators**: Intent resolution, task adherence, tool call accuracy +- **Batch Evaluation**: Evaluate multiple conversations +- **Continuous Monitoring**: Production agent performance tracking +- **MCP Tools**: `foundry_agents_query_and_evaluate`, `foundry_agents_evaluate` -# Azure AI Search (for RAG) -AZURE_AI_SEARCH_CONNECTION_NAME=my-search-connection -AI_SEARCH_INDEX_NAME=my-index +--- -# Evaluation -AZURE_OPENAI_ENDPOINT=https://.openai.azure.com -AZURE_OPENAI_DEPLOYMENT=gpt-4o -``` +## MCP Tools Reference + +| Tool | Purpose | +|------|---------| +| `foundry_resource_get` | Get resource details and endpoint | +| `foundry_models_list` | Browse model catalog | +| `foundry_models_deploy` | Deploy a model | +| `foundry_models_deployments_list` | List deployed models | +| `foundry_knowledge_index_list` | List knowledge indexes | +| `foundry_knowledge_index_schema` | Get index schema | +| `foundry_agents_list` | List agents | +| `foundry_agents_connect` | Query an agent | +| `foundry_agents_query_and_evaluate` | Query and evaluate | +| `foundry_openai_chat_completions_create` | Chat completions | +| `foundry_openai_embeddings_create` | Create embeddings | -### Useful MCP Tools Quick Reference +--- -**Resource Management** -- `foundry_resource_get` - Get resource details and endpoint +## Detailed Guides -**Models** -- `foundry_models_list` - Browse model catalog -- `foundry_models_deploy` - Deploy a model -- `foundry_models_deployments_list` - List deployed models +For step-by-step workflows and SDK examples: -**Knowledge & RAG** -- `foundry_knowledge_index_list` - List knowledge indexes -- `foundry_knowledge_index_schema` - Get index schema +- **[Detailed Workflows](references/workflows.md)** - Model deployment, RAG, agents, evaluation +- **[Python SDK Guide](language/python.md)** - Authentication, agents, RAG, evaluation +- **[Model Deployment](references/workflows.md#model-discovery-and-deployment)** - Deploy and configure models +- **[Agent Creation](references/workflows.md#creating-ai-agents)** - Create agents with tools +- **[RAG Applications](references/workflows.md#building-rag-applications)** - Knowledge indexes and search +- **[Evaluation](references/workflows.md#evaluating-agent-performance)** - Agent quality assessment -**Agents** -- `foundry_agents_list` - List agents -- `foundry_agents_connect` - Query an agent -- `foundry_agents_query_and_evaluate` - Query and evaluate +--- -**OpenAI Operations** -- `foundry_openai_chat_completions_create` - Create chat completions -- `foundry_openai_embeddings_create` - Create embeddings +## Troubleshooting -### Language-Specific Quick References +| Issue | Solution | +|-------|----------| +| Deployment pending/fails | Check quota, try different region | +| No citations in RAG | Update agent instructions to request citations | +| 401/403 auth error | Assign RBAC roles (Search Index Data Contributor) | +| Rate limit (429) | Implement retry logic, request quota increase | +| Evaluation shows no data | Generate traffic, expand time range, wait for ingestion | -For SDK-specific details, authentication, and code examples: -- **Python**: See [language/python.md](language/python.md) +--- -## Additional Resources +## Resources -### Documentation Links -- [Microsoft Foundry Documentation](https://learn.microsoft.com/azure/ai-foundry/) -- [Microsoft Foundry Quickstart](https://learn.microsoft.com/azure/ai-foundry/quickstarts/get-started-code) -- [RAG and Knowledge Indexes](https://learn.microsoft.com/azure/ai-foundry/concepts/retrieval-augmented-generation) +- [Microsoft Foundry Docs](https://learn.microsoft.com/azure/ai-foundry/) +- [Foundry Quickstart](https://learn.microsoft.com/azure/ai-foundry/quickstarts/get-started-code) - [Agent Evaluation Guide](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/agent-evaluate-sdk) - -### GitHub Samples -- [Microsoft Foundry Samples](https://github.com/azure-ai-foundry/foundry-samples) -- [Azure Search OpenAI Demo](https://github.com/Azure-Samples/azure-search-openai-demo) -- [Azure Search Classic RAG](https://github.com/Azure-Samples/azure-search-classic-rag) +- [Foundry Samples](https://github.com/azure-ai-foundry/foundry-samples) diff --git a/plugin/skills/microsoft-foundry/references/workflows.md b/plugin/skills/microsoft-foundry/references/workflows.md new file mode 100644 index 000000000..7a2b4f36e --- /dev/null +++ b/plugin/skills/microsoft-foundry/references/workflows.md @@ -0,0 +1,348 @@ +# Microsoft Foundry Detailed Workflows + +This document contains step-by-step workflows for common Microsoft Foundry tasks. + +## Table of Contents + +- [Model Discovery and Deployment](#model-discovery-and-deployment) +- [Building RAG Applications](#building-rag-applications) +- [Creating AI Agents](#creating-ai-agents) +- [Evaluating Agent Performance](#evaluating-agent-performance) +- [Troubleshooting](#troubleshooting) + +--- + +## Model Discovery and Deployment + +### Step 1: List Available Resources + +```bash +# List all Microsoft Foundry resources in subscription +az resource list \ + --resource-type "Microsoft.CognitiveServices/accounts" \ + --query "[?kind=='AIServices'].{Name:name, ResourceGroup:resourceGroup, Location:location}" \ + --output table + +# List resources in a specific resource group +az resource list \ + --resource-group \ + --resource-type "Microsoft.CognitiveServices/accounts" \ + --output table +``` + +**MCP Tool:** `foundry_resource_get` + +### Step 2: Browse Model Catalog + +**Key Points:** +- Some models support **free playground** for prototyping without costs +- Models can be filtered by **publisher** (OpenAI, Meta, Microsoft) +- Models can be filtered by **license type** +- Model availability varies by region + +**MCP Tool Examples:** +- List all models: `foundry_models_list()` +- Free playground models: `foundry_models_list(search-for-free-playground=true)` +- Filter by publisher: `foundry_models_list(publisher="OpenAI")` +- Filter by license: `foundry_models_list(license="MIT")` + +### Step 3: Deploy a Model + +```bash +# Deploy a model (e.g., gpt-4o) +az cognitiveservices account deployment create \ + --name \ + --resource-group \ + --deployment-name gpt-4o-deployment \ + --model-name gpt-4o \ + --model-version "2024-05-13" \ + --model-format OpenAI \ + --sku-capacity 10 \ + --sku-name Standard + +# Verify deployment status +az cognitiveservices account deployment show \ + --name \ + --resource-group \ + --deployment-name gpt-4o-deployment +``` + +**MCP Tool:** `foundry_models_deploy` with parameters: +- `resource-group`: Resource group name +- `deployment`: Deployment name +- `model-name`: Model to deploy (e.g., "gpt-4o") +- `model-format`: Format (e.g., "OpenAI") +- `azure-ai-services`: Foundry resource name +- `model-version`: Specific version +- `sku-capacity`: Capacity units +- `scale-type`: Scaling type + +**Deployment Verification:** +When deployment completes, `provisioningState` should be `Succeeded`. Common failure causes: +- Insufficient quota +- Region capacity limitations +- Permission issues + +### Step 4: Get Resource Endpoint + +**MCP Tool:** `foundry_resource_get` + +**Expected Output:** +Endpoint format: `https://.services.ai.azure.com/api/projects/` + +--- + +## Building RAG Applications + +### Understanding RAG + +RAG (Retrieval-Augmented Generation) enhances AI responses by: +1. **Retrieving** relevant documents from a knowledge base +2. **Augmenting** the AI prompt with retrieved context +3. **Generating** responses grounded in factual information + +### Step 1: List Knowledge Indexes + +**MCP Tool:** `foundry_knowledge_index_list` with your project endpoint + +### Step 2: Inspect Index Schema + +**MCP Tool:** `foundry_knowledge_index_schema` with project endpoint and index name + +**Schema Information:** +- Field definitions and data types +- Searchable attributes +- Vectorization configuration +- Retrieval mode support (keyword, semantic, vector, hybrid) + +### Step 3: Create RAG Agent + +To create a RAG agent with Azure AI Search: + +1. **Initialize AI Project Client** with endpoint and credentials +2. **Get Azure AI Search connection** from project +3. **Create agent** with: + - Agent name and model deployment + - Clear instructions requesting citations + - Azure AI Search tool with connection ID, index name, query type + +**For SDK Implementation:** See [../language/python.md](../language/python.md#rag-applications-with-python-sdk) + +**Best Practices:** +- **Always request citations** in agent instructions +- Use **hybrid search** for best results +- Instruct agent to say "I don't know" when info isn't in index +- Format citations consistently + +### Step 4: Test RAG Agent + +1. Query the agent with a test question +2. Stream the response for real-time output +3. Capture citations from response annotations +4. Validate citation formatting + +**For SDK Implementation:** See [../language/python.md](../language/python.md#testing-the-rag-agent) + +--- + +## Creating AI Agents + +### Step 1: List Existing Agents + +**MCP Tool:** `foundry_agents_list` with project endpoint + +### Step 2: Create Basic Agent + +Create with: +- **Model deployment name**: The model to use +- **Agent name**: Unique identifier +- **Instructions**: Clear, specific guidance + +**For SDK Implementation:** See [../language/python.md](../language/python.md#basic-agent) + +### Step 3: Add Custom Function Tools + +1. Define custom functions with clear docstrings +2. Create function toolset with custom functions +3. Create agent with toolset and instructions + +**For SDK Implementation:** See [../language/python.md](../language/python.md#agent-with-custom-function-tools) + +### Step 4: Add Web Search + +Create agent with web search capabilities: +- Optionally specify user location for localized results +- Provide instructions to always cite web sources + +**For SDK Implementation:** See [../language/python.md](../language/python.md#agent-with-web-search) + +### Step 5: Interact with Agent + +1. Create conversation thread +2. Add user messages +3. Run agent to process messages +4. Check run status +5. Retrieve messages +6. Cleanup (delete agent when done) + +**For SDK Implementation:** See [../language/python.md](../language/python.md#interacting-with-agents) + +### Agent Best Practices + +1. **Clear Instructions**: Specific, actionable guidance +2. **Tool Selection**: Only include needed tools +3. **Error Handling**: Check `run.status` for failures +4. **Cleanup**: Delete agents/threads to manage costs +5. **Rate Limits**: Handle 429 errors gracefully + +--- + +## Evaluating Agent Performance + +### Built-in Evaluators + +| Evaluator | What It Measures | Score | +|-----------|------------------|-------| +| `IntentResolutionEvaluator` | User request understanding | 1-5 | +| `TaskAdherenceEvaluator` | Adherence to tasks/instructions | 1-5 | +| `ToolCallAccuracyEvaluator` | Correct function tool calls | 1-5 | + +**Evaluation Output:** +- `{metric_name}`: Numerical score (1-5) +- `{metric_name}_result`: "pass" or "fail" +- `{metric_name}_threshold`: Binarization threshold +- `{metric_name}_reason`: Explanation + +### Single Run Evaluation + +**MCP Tool:** `foundry_agents_query_and_evaluate` + +Parameters: +- Agent ID +- Query text +- Project endpoint +- Azure OpenAI endpoint and deployment +- Comma-separated evaluators + +### Evaluate Existing Response + +**MCP Tool:** `foundry_agents_evaluate` + +### Batch Evaluation + +1. Convert agent thread data to evaluation format +2. Prepare data from multiple thread IDs +3. Set up evaluators +4. Run batch evaluation + +**For SDK Implementation:** See [../language/python.md](../language/python.md#batch-evaluation) + +### Score Interpretation + +| Score | Meaning | +|-------|---------| +| 5 | Excellent - Perfect understanding and execution | +| 4 | Good - Minor issues, overall successful | +| 3 | Acceptable - Threshold for passing | +| 2 | Poor - Significant issues | +| 1 | Failed - Complete misunderstanding | + +--- + +## Troubleshooting + +### Deployment Issues + +**Problem: Deployment Pending/Fails** + +```bash +# Check deployment status +az cognitiveservices account deployment show \ + --name \ + --resource-group \ + --deployment-name \ + --output json + +# Check quota +az cognitiveservices account show \ + --name \ + --resource-group \ + --query "properties.quotaLimit" +``` + +**Resolution:** +1. Check quota limits in Azure Portal +2. Request quota increase +3. Try different region +4. Verify RBAC permissions + +### Agent Issues + +**No Citations in RAG Response:** +- Update instructions to explicitly request citations +- Verify tool choice is "required" or "auto" +- Check Azure AI Search connection + +**Index Not Found:** +- Verify `AI_SEARCH_INDEX_NAME` matches actual index +- Check connection points to correct resource +- Ensure index is created and populated + +**401/403 Authentication:** + +```bash +# Assign role to managed identity +az role assignment create \ + --assignee \ + --role "Search Index Data Contributor" \ + --scope /subscriptions//resourceGroups//providers/Microsoft.Search/searchServices/ +``` + +### Evaluation Issues + +**Dashboard Shows No Data:** +1. Generate new agent traffic +2. Expand time range filter +3. Wait for data ingestion +4. Refresh dashboard + +**Continuous Evaluation Not Running:** +1. Verify evaluation rule is enabled +2. Confirm agent traffic is flowing +3. Check managed identity has **Azure AI User** role +4. Verify OpenAI endpoint is accessible + +### Rate Limiting + +**Error:** `Rate limit is exceeded` (HTTP 429) + +```bash +# Check quota usage +az cognitiveservices usage list \ + --name \ + --resource-group +``` + +**Best Practices:** +- Implement exponential backoff +- Use Dynamic Quota when available +- Monitor usage proactively +- Consider multi-region deployments + +--- + +## Environment Variables + +```bash +# Foundry Project +PROJECT_ENDPOINT=https://.services.ai.azure.com/api/projects/ +MODEL_DEPLOYMENT_NAME=gpt-4o + +# Azure AI Search (for RAG) +AZURE_AI_SEARCH_CONNECTION_NAME=my-search-connection +AI_SEARCH_INDEX_NAME=my-index + +# Evaluation +AZURE_OPENAI_ENDPOINT=https://.openai.azure.com +AZURE_OPENAI_DEPLOYMENT=gpt-4o +```