diff --git a/contributing/samples/bigquery_skills_demo/README.md b/contributing/samples/bigquery_skills_demo/README.md new file mode 100644 index 0000000000..63c6783bb8 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/README.md @@ -0,0 +1,260 @@ +# BigQuery Skills Demo + +This sample demonstrates Anthropic's [Agent Skills Pattern](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) for dynamic skill discovery with BigQuery ML and AI capabilities, enhanced with **callback-based automatic skill loading**. + +## Overview + +This demo showcases: +- **Dynamic Skill Discovery**: Skills are discovered at runtime from SKILL.md files with YAML frontmatter +- **Progressive Disclosure**: Only skill names/descriptions loaded initially; full content on-demand +- **Callback-based Auto-activation**: Skills are automatically activated based on keywords in user messages (no LLM calls needed!) +- **Ephemeral Skill Loading**: Skills are injected into the system prompt (not conversation history) and can be truly unloaded +- **Automatic Cleanup**: Skills are auto-deactivated after each turn to free up context +- **Scalable Design**: Adding new skills requires only a SKILL.md file - no code changes! + +### Available Skills + +1. **bqml** - BigQuery ML for training and deploying ML models in SQL + - Model training (LINEAR_REG, LOGISTIC_REG, KMEANS, ARIMA_PLUS, XGBoost, etc.) + - Model evaluation and prediction + - Feature importance and model analysis + +2. **bq_ai_operator** - Managed AI functions in BigQuery SQL + - AI.CLASSIFY: Categorize text into classes + - AI.IF: Natural language TRUE/FALSE filtering + - AI.SCORE: Rate/rank content by criteria (0.0 to 1.0) + +3. **bq_remote_model** - Remote models connecting to Vertex AI + - CREATE REMOTE MODEL: Connect to Gemini, Claude, Llama, and custom endpoints + - AI.GENERATE_TEXT: Text generation with LLMs + - AI.GENERATE_EMBEDDING: Vector embeddings for semantic search + +## Prerequisites + +1. Google Cloud project with BigQuery and Vertex AI enabled +2. Application Default Credentials configured: + ```bash + gcloud auth application-default login + ``` +3. Set your project ID: + ```bash + export GOOGLE_CLOUD_PROJECT=your-project-id + ``` + +### For AI Functions (bq_ai_operator and bq_remote_model skills) + +Create a BigQuery connection to Vertex AI: +```bash +bq mk --connection \ + --location=us \ + --project_id=$GOOGLE_CLOUD_PROJECT \ + --connection_type=CLOUD_RESOURCE \ + my_ai_connection +``` + +Grant the connection's service account access to Vertex AI: +```bash +# Get the service account +bq show --connection $GOOGLE_CLOUD_PROJECT.us.my_ai_connection + +# Grant access (replace with actual service account) +gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \ + --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \ + --role="roles/aiplatform.user" +``` + +## Running the Demo + +### Option 1: Run with ADK CLI + +```bash +cd contributing/samples/bigquery_skills_demo +adk run . +``` + +### Option 2: Run the web UI + +```bash +adk web contributing/samples --port 8000 +# Open http://127.0.0.1:8000/dev-ui/?app=bigquery_skills_demo +``` + +## Example Prompts + +### BQML Skill (auto-activated by: "train", "model", "predict", "regression", "kmeans") +``` +Train a linear regression model to predict penguin body weight using +the public penguins dataset, then evaluate it and show feature importance. +``` + +### BQ AI Operator Skill (auto-activated by: "classify", "AI.CLASSIFY", "sentiment", "categorize") +``` +Classify 5 BBC news articles by their topic using AI.CLASSIFY with +categories: tech, sport, business, politics, entertainment, other. +``` + +### BQ Remote Model Skill (auto-activated by: "generate text", "gemini", "embeddings", "llm") +``` +Create a remote model using Gemini 2.0 Flash and use it to summarize +product descriptions from my table. +``` + +## How It Works + +### Architecture: Callback-based Skill Management + +This demo uses ADK callbacks instead of LLM tool calls for skill management: + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ User Message │ +│ "Train a model to predict penguin weight" │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ before_model_callback │ +│ 1. Extract keywords from user message │ +│ 2. Match against skill keywords (from SKILL.md frontmatter) │ +│ 3. Activate matching skills: ["bqml"] │ +│ 4. DIRECTLY INJECT skill content into llm_request.system_instruction│ +│ (This ensures skills are available in the FIRST LLM call!) │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ LLM Processing │ +│ System prompt now includes: │ +│ - Base instruction │ +│ - Active skill documentation (BQML syntax, examples) │ +│ Skills are available IMMEDIATELY - no need to wait for tool call! │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ after_agent_callback │ +│ 1. Clear active skills from state │ +│ 2. Context freed for next interaction │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +### Direct Injection vs Instruction Provider + +The callback directly injects skill content into `llm_request.system_instruction`, bypassing the instruction provider timing issue: + +| Approach | When Skills Appear | How It Works | +|----------|-------------------|--------------| +| **Direct Injection** (current) | First LLM call | Callback modifies `llm_request.system_instruction` directly | +| Instruction Provider | Second LLM call | Provider reads from state, but state updated after instruction built | + +This direct injection ensures the LLM has skill documentation from the very first response, enabling it to follow skill examples immediately. + +### Key Components + +1. **Skill Discovery** (`skill_registry.py`) + - Scans `skills/` directory for SKILL.md files + - Parses YAML frontmatter (name, description, keywords) + - Provides instruction provider for ephemeral skill injection + +2. **Skill Callbacks** (`skill_callbacks.py`) + - `before_model_callback`: Detects and activates skills based on keywords + - `after_agent_callback`: Cleans up skills after each turn + - Supports multiple detection modes: keyword (recommended), hybrid, llm + +3. **Agent Configuration** (`agent.py`) + - Registers callbacks with LlmAgent + - Combines base instruction with dynamic skill content + - Manual skill tools available as fallback + +### Callback vs Tool Approach Comparison + +| Aspect | Callback (This Demo) | Tool-based | +|--------|---------------------|------------| +| **LLM Calls** | Zero for skill management | 1-2 per skill activation | +| **Latency** | Instant (keyword matching) | Adds round-trip time | +| **Cost** | No additional tokens | Extra tool call tokens | +| **Control** | System-level, deterministic | LLM decides when to activate | +| **Best For** | Domain-specific terms (BigQuery) | Semantic understanding needed | + +### Why Keyword Detection for BigQuery? + +BigQuery has **highly domain-specific terminology** that makes keyword detection ideal: +- "BQML", "ML.PREDICT", "CREATE MODEL" → bqml skill +- "AI.CLASSIFY", "AI.IF", "AI.SCORE" → bq_ai_operator skill +- "GENERATE_TEXT", "gemini", "embeddings" → bq_remote_model skill + +These terms are unambiguous - you don't need an LLM to understand that "AI.CLASSIFY" relates to the AI operator skill. + +## Code Structure + +``` +bigquery_skills_demo/ +├── __init__.py # Module init +├── agent.py # Agent with BigQuery tools and callbacks +├── skill_registry.py # Dynamic skill discovery + instruction provider +├── skill_callbacks.py # Callback-based auto-activation +├── skill_classifier.py # Optional LLM-based classification (for hybrid mode) +├── skills/ +│ ├── bqml/ +│ │ └── SKILL.md # BQML skill (keywords: train, model, predict, etc.) +│ ├── bq_ai_operator/ +│ │ └── SKILL.md # AI operator skill (keywords: classify, sentiment, etc.) +│ └── bq_remote_model/ +│ └── SKILL.md # Remote model skill (keywords: gemini, embeddings, etc.) +└── README.md # This file +``` + +## Adding New Skills + +Adding a new skill requires **only a SKILL.md file** - no code changes needed! + +1. Create a directory under `skills/` (e.g., `skills/my_skill/`) +2. Add a `SKILL.md` file with YAML frontmatter: + ```markdown + --- + name: my_skill + description: Short description of what this skill does + keywords: + - keyword1 + - keyword2 + - specific_function_name + --- + + # My Skill Documentation + + Detailed instructions, examples, and usage patterns... + ``` +3. The skill will be automatically discovered and keyword patterns built from frontmatter + +### Keyword Guidelines + +- Use domain-specific terms that clearly indicate the skill is needed +- Include function names (e.g., "ML.PREDICT", "AI.CLASSIFY") +- Include common user phrases (e.g., "train", "classify", "embeddings") +- Multiple keywords increase detection coverage + +## Detection Modes + +The `SkillCallbacks` class supports three detection modes: + +```python +# In agent.py +skill_callbacks = SkillCallbacks( + skill_registry, + auto_deactivate=True, + detection_mode="keyword", # "keyword" | "hybrid" | "llm" +) +``` + +| Mode | Description | Best For | +|------|-------------|----------| +| `keyword` | Regex pattern matching from SKILL.md keywords | Domain-specific terms (recommended) | +| `hybrid` | LLM classification with keyword fallback | Mixed semantic/specific queries | +| `llm` | Pure LLM-based semantic classification | Paraphrased/ambiguous requests | + +## References + +- [Anthropic: Equipping Agents with Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) +- [BigQuery ML Documentation](https://cloud.google.com/bigquery/docs/bqml-introduction) +- [BigQuery AI Functions](https://cloud.google.com/bigquery/docs/ai-functions) +- [BigQuery Remote Models](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-remote-model) diff --git a/contributing/samples/bigquery_skills_demo/__init__.py b/contributing/samples/bigquery_skills_demo/__init__.py new file mode 100644 index 0000000000..c48963cdc7 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from . import agent diff --git a/contributing/samples/bigquery_skills_demo/agent.py b/contributing/samples/bigquery_skills_demo/agent.py new file mode 100644 index 0000000000..6dbebba2ef --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/agent.py @@ -0,0 +1,263 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery Skills Demo Agent with Dynamic Skill Discovery. + +This agent demonstrates the Anthropic Skills Pattern for dynamic capability +discovery, as described in: +https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills + +Key Features: +1. **Progressive Disclosure**: Skills are discovered at startup with only + names and descriptions loaded. Full content is loaded on-demand. + +2. **Dynamic Discovery**: Skills are stored as SKILL.md files and discovered + automatically from the skills/ directory. + +3. **load_skill Tool**: The agent can load full skill documentation when + it determines a skill is relevant to the current task. + +Available Skills: +- bqml: BigQuery ML for training/deploying ML models in SQL +- bq_ai_operator: Generative AI functions in SQL + +To run this demo: + cd contributing/samples/bigquery_skills_demo + adk run . + +Or via web UI: + adk web contributing/samples --port 8000 + # Then open http://127.0.0.1:8000/dev-ui/?app=bigquery_skills_demo +""" + +import os + +# Set environment variables for Vertex AI (uses ADC for authentication) +# Users should set GOOGLE_CLOUD_PROJECT to their own project ID +os.environ.setdefault("GOOGLE_GENAI_USE_VERTEXAI", "true") +os.environ.setdefault("GOOGLE_CLOUD_LOCATION", "us-central1") + +from google.adk.agents.llm_agent import LlmAgent +from google.adk.tools import FunctionTool +from google.adk.tools.bigquery import BigQueryCredentialsConfig +from google.adk.tools.bigquery import BigQueryToolset +from google.adk.tools.bigquery.config import BigQueryToolConfig +from google.adk.tools.bigquery.config import WriteMode +import google.auth + +# Import the dynamic skill registry with ephemeral skill loading +from .skill_registry import ( + SkillRegistry, + create_skill_instruction_provider, + activate_skill, + deactivate_skill, + list_active_skills, +) + +# Import callback-based skill management (saves LLM calls) +from .skill_callbacks import SkillCallbacks + +# Agent name +AGENT_NAME = "bigquery_skills_demo_agent" + +# Project configuration - must be set via environment variable +PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT") +if not PROJECT_ID: + raise ValueError( + "GOOGLE_CLOUD_PROJECT environment variable must be set. " + "Set it to your GCP project ID before running this demo." + ) + +# Initialize BigQuery tool config +# Using ALLOWED write mode to enable CREATE MODEL operations +tool_config = BigQueryToolConfig( + write_mode=WriteMode.ALLOWED, + application_name=AGENT_NAME, +) + +# Use application default credentials +application_default_credentials, _ = google.auth.default() +credentials_config = BigQueryCredentialsConfig( + credentials=application_default_credentials +) + +# Initialize BigQuery toolset +bigquery_toolset = BigQueryToolset( + credentials_config=credentials_config, + bigquery_tool_config=tool_config, +) + +# Initialize dynamic skill registry +skill_registry = SkillRegistry() +SKILLS_SUMMARY = skill_registry.get_skills_summary() + +# Create instruction provider for ephemeral skill loading (Claude Code-style) +# This injects active skills into the system prompt, not conversation history +skill_instruction_provider = create_skill_instruction_provider(skill_registry) + +# Create skill management tools (kept for manual override if needed) +activate_skill_tool = FunctionTool(activate_skill) +deactivate_skill_tool = FunctionTool(deactivate_skill) +list_active_skills_tool = FunctionTool(list_active_skills) + +# Create callback-based skill management with keyword detection +# Modes: "keyword" (recommended for BigQuery), "hybrid", "llm" +# Keyword mode is fast, no API calls, and BigQuery terms are domain-specific enough +skill_callbacks = SkillCallbacks( + skill_registry, + auto_deactivate=True, + detection_mode="keyword", # Fast keyword detection - ideal for domain-specific terms +) + +# Base instruction for the agent (static part) +BASE_INSTRUCTION = f"""\ +You are a data science agent with BigQuery capabilities and automatic skill loading. + +## How Skills Work (Keyword-based Auto-activation) + +You have access to specialized skills that provide detailed guidance for complex tasks. +**Skills are automatically activated** based on keywords in the user's message - no API calls needed! + +Skills are: +- **Auto-activated by keywords**: Domain-specific terms trigger skill loading instantly + - "train", "model", "predict", "regression", "kmeans" → bqml + - "classify", "AI.CLASSIFY", "sentiment", "categorize" → bq_ai_operator + - "generate text", "gemini", "embeddings", "remote model" → bq_remote_model +- **Auto-deactivated**: After you complete your response, skills are cleared to free context +- **Multi-skill support**: Multiple skills can be loaded simultaneously if needed + +**This is efficient**: Keyword detection is instant with zero API calls! + +**Current Available Skills:** + +{SKILLS_SUMMARY} + +**Manual Skill Management (optional, for override):** +- `activate_skill(skill_name)`: Manually load a skill if auto-detection missed it +- `deactivate_skill(skill_name)`: Manually unload a skill (rarely needed) +- `list_active_skills()`: See which skills are currently loaded + +**Note**: In most cases, just focus on the task - skills will be loaded automatically! + +## Available BigQuery Tools + +- `execute_sql`: Run any BigQuery SQL (queries, DDL, BQML, AI functions) +- `get_table_info`: Get schema information for a table +- `list_dataset_ids`: List datasets in a project +- `list_table_ids`: List tables in a dataset +- `list_connections`: **ALWAYS use this first** to discover existing BigQuery connections +- `create_connection`: Create a new BigQuery connection (auto-grants Vertex AI User role) + +## Connection Management (IMPORTANT) + +For AI functions and remote models, you need a BigQuery connection to Vertex AI. + +**ALWAYS follow this workflow:** +1. **First**: Call `list_connections(project_id, location)` to discover existing connections +2. **If connections exist**: Use one with `connection_type: "CLOUD_RESOURCE"` +3. **Only if no connections**: Call `create_connection(project_id, location, connection_id)` + - This automatically grants the Vertex AI User IAM role to the service account + +## Project Configuration + +- Project ID: {PROJECT_ID} +- Available public datasets: `bigquery-public-data.ml_datasets` (penguins, census, etc.) + +## Workflow Example + +1. User asks: "Train a model to predict penguin weight" +2. **[AUTOMATIC]** The bqml skill is loaded based on keywords "train", "model", "predict" +3. You follow the skill's examples to CREATE MODEL, EVALUATE, and PREDICT +4. You explain results to the user +5. **[AUTOMATIC]** Skills are cleared after your response + +## Guidelines + +1. **Focus on the task**: Skills load automatically - no need to call activate_skill() +2. **Explore data first**: Use `get_table_info` or `SELECT * LIMIT 5` before complex queries +3. **Use LIMIT**: Prevent large result sets with `LIMIT 10-100` +4. **Explain your steps**: Describe what each query does and interpret results + +## Quick Reference (without loading skills) + +**BQML Quick Start:** +```sql +-- Train: CREATE OR REPLACE MODEL `project.dataset.model` OPTIONS(model_type='LINEAR_REG', ...) +-- Evaluate: SELECT * FROM ML.EVALUATE(MODEL `project.dataset.model`) +-- Predict: SELECT * FROM ML.PREDICT(MODEL `project.dataset.model`, ...) +``` + +**AI Operator Quick Start:** +```sql +-- Classify: AI.CLASSIFY(text, categories => [...], connection_id => 'loc.conn') +-- Filter: AI.IF(text, 'condition', connection_id => 'loc.conn') +-- Score: AI.SCORE(text, 'criteria', connection_id => 'loc.conn') +``` + +**Remote Model Quick Start:** +```sql +-- Create: CREATE REMOTE MODEL `project.dataset.model` REMOTE WITH CONNECTION `loc.conn` OPTIONS(ENDPOINT='gemini-2.0-flash') +-- Generate: SELECT * FROM AI.GENERATE_TEXT(MODEL `project.dataset.model`, (SELECT 'prompt' AS prompt), STRUCT(...)) +-- Embed: SELECT * FROM AI.GENERATE_EMBEDDING(MODEL `project.dataset.model`, (SELECT content FROM table), STRUCT(...)) +``` + +For detailed syntax and examples, skills are loaded automatically based on your question! +""" + + +def create_combined_instruction_provider(base_instruction: str, skill_provider): + """Create an instruction provider that combines base instruction with active skills. + + This follows the Claude Code pattern where skills are injected into the system prompt + (ephemeral) rather than conversation history (persistent). + """ + from google.adk.agents.readonly_context import ReadonlyContext + + def combined_provider(ctx: ReadonlyContext) -> str: + # Get dynamic skill content + skill_content = skill_provider(ctx) + + if skill_content: + return f"{base_instruction}\n\n{skill_content}" + return base_instruction + + return combined_provider + + +# Create the combined instruction provider +instruction_provider = create_combined_instruction_provider( + BASE_INSTRUCTION, skill_instruction_provider +) + +# Create the root agent with BigQuery tools, ephemeral skill loading, and callbacks +root_agent = LlmAgent( + model="gemini-2.5-pro", + name=AGENT_NAME, + description=( + "Data science agent with BigQuery ML and AI capabilities. " + "Uses callback-based automatic skill activation - skills are loaded based on " + "user input keywords and unloaded after each turn to manage context efficiently." + ), + instruction=instruction_provider, + tools=[ + bigquery_toolset, + # Keep manual tools as fallback (agent can still explicitly manage skills) + activate_skill_tool, + deactivate_skill_tool, + list_active_skills_tool, + ], + # Callback-based skill management (saves LLM calls) + before_model_callback=skill_callbacks.before_model_callback, + after_agent_callback=skill_callbacks.after_agent_callback, +) diff --git a/contributing/samples/bigquery_skills_demo/skill_callbacks.py b/contributing/samples/bigquery_skills_demo/skill_callbacks.py new file mode 100644 index 0000000000..9b7d775650 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skill_callbacks.py @@ -0,0 +1,544 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Callback-based Skill Management for ADK Agents. + +This module implements automatic skill activation/deactivation using ADK callbacks, +eliminating the need for explicit LLM tool calls to manage skills. + +Key Features: +1. **Auto-activation via before_model_callback**: Analyzes user input and activates + relevant skills before the LLM processes the request. +2. **Auto-deactivation via after_agent_callback**: Cleans up skills after agent + completes a turn to free context for the next interaction. +3. **Intelligent Detection**: Uses LLM-based classification for semantic understanding + with keyword fallback for reliability. + +Detection Modes: +- "llm": Use LLM classification (most intelligent, understands paraphrases) +- "keyword": Use keyword matching only (fastest, no API calls) +- "hybrid": Try LLM first, fall back to keywords (recommended) + +This approach saves LLM calls compared to having the agent explicitly call +activate_skill/deactivate_skill tools. +""" + +from __future__ import annotations + +import re +from typing import TYPE_CHECKING, Literal + +from .skill_registry import SkillRegistry, ACTIVE_SKILLS_KEY + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + from google.adk.flows.llm_flows.llm_request import LlmRequest + from google.adk.flows.llm_flows.llm_response import LlmResponse + from google.genai import types + +# Detection mode type +DetectionMode = Literal["llm", "keyword", "hybrid"] + + +# Default keyword patterns for auto-activating skills (fallback if not in SKILL.md) +# New skills should define keywords in their SKILL.md frontmatter instead of here +DEFAULT_SKILL_ACTIVATION_PATTERNS: dict[str, list[str]] = { + # These are fallback patterns - skills should define keywords in SKILL.md +} + + +class SkillCallbacks: + """Callback handlers for automatic skill management. + + This class creates callbacks that can be registered with an LlmAgent to + automatically activate and deactivate skills based on user input. + + Supports multiple detection modes: + - "llm": Use LLM-based classification (most intelligent) + - "keyword": Use keyword matching (fastest, no API calls) + - "hybrid": Try LLM first, fall back to keywords (recommended) + + Usage: + registry = SkillRegistry() + skill_callbacks = SkillCallbacks(registry, detection_mode="hybrid") + + agent = LlmAgent( + ... + before_model_callback=skill_callbacks.before_model_callback, + after_agent_callback=skill_callbacks.after_agent_callback, + ) + """ + + def __init__( + self, + registry: SkillRegistry, + auto_deactivate: bool = True, + activation_patterns: dict[str, list[str]] | None = None, + detection_mode: DetectionMode = "hybrid", + ): + """Initialize skill callbacks. + + Args: + registry: The skill registry to load skills from. + auto_deactivate: If True, automatically deactivate all skills after + agent completes a turn. Set to False to keep skills active across + multiple turns. + activation_patterns: Custom keyword patterns for skill activation. + Maps skill name to list of regex patterns. If not provided, + patterns are built dynamically from SKILL.md keywords. + detection_mode: How to detect skills from user input: + - "llm": Use LLM classification (semantic understanding) + - "keyword": Use keyword matching only (fast, no API calls) + - "hybrid": Try LLM first, fall back to keywords (recommended) + """ + self._registry = registry + self._auto_deactivate = auto_deactivate + self._detection_mode = detection_mode + self._classifier = None # Lazy initialization + + # Build patterns from registry keywords (dynamic, scalable) + # Falls back to custom patterns if provided, or default patterns + if activation_patterns is not None: + self._patterns = activation_patterns + else: + self._patterns = self._build_patterns_from_registry() + + # Compile regex patterns for efficiency + self._compiled_patterns: dict[str, list[re.Pattern]] = {} + for skill_name, patterns in self._patterns.items(): + self._compiled_patterns[skill_name] = [ + re.compile(p, re.IGNORECASE) for p in patterns + ] + + def _build_patterns_from_registry(self) -> dict[str, list[str]]: + """Build keyword patterns dynamically from the registry. + + This makes keyword detection scalable - new skills just need to add + keywords to their SKILL.md frontmatter, no code changes needed. + + Returns: + Dict mapping skill names to regex patterns built from keywords. + """ + patterns: dict[str, list[str]] = {} + + # Get all keywords from registry + all_keywords = self._registry.get_all_keywords() + + for skill_name, keywords in all_keywords.items(): + skill_patterns = [] + for keyword in keywords: + # Convert keyword to regex pattern + # Handle multi-word keywords and special characters + escaped = re.escape(keyword) + # Use word boundaries for better matching + # For keywords with dots (like ai.classify), don't add word boundaries + if "." in keyword: + pattern = escaped + else: + pattern = rf"\b{escaped}\b" + skill_patterns.append(pattern) + if skill_patterns: + patterns[skill_name] = skill_patterns + + return patterns + + def _build_skill_content(self, skill_names: list[str]) -> str: + """Build skill content string from activated skills. + + This loads the full skill documentation from the registry and formats + it for injection into the system instruction. + + Args: + skill_names: List of skill names to load. + + Returns: + Formatted string containing all skill documentation. + """ + if not skill_names: + return "" + + skill_sections = [] + for skill_name in skill_names: + skill = self._registry.load_skill_content(skill_name) + if skill: + skill_sections.append(f""" +## Active Skill: {skill.name} + +{skill.description} + +--- + +{skill.content} +""") + + if not skill_sections: + return "" + + return f""" +# Currently Active Skills + +The following skills have been loaded and are available for this task: + +{"".join(skill_sections)} + +--- +**Note**: Use `deactivate_skill(skill_name)` when you're done with a skill to free up context. +""" + + def _inject_skills_into_request( + self, + llm_request: "LlmRequest", + skill_names: list[str], + ) -> None: + """Inject skill content directly into the LLM request's system instruction. + + This is the key fix for the timing issue: by modifying llm_request.config.system_instruction + directly in the before_model_callback, the skills are available in the FIRST LLM call, + not just subsequent calls. + + The LlmRequest has a config.system_instruction field that can be a string. + We use the append_instructions() method which handles string concatenation properly. + + Args: + llm_request: The LLM request to modify. + skill_names: List of skill names to inject. + """ + if not skill_names: + return + + skill_content = self._build_skill_content(skill_names) + if not skill_content: + return + + # Use append_instructions which properly handles string system_instruction + # This concatenates to config.system_instruction using "\n\n" + llm_request.append_instructions([skill_content]) + + print(f"[SkillCallbacks] Injected skill content into system instruction: {skill_names}") + + def _get_classifier(self): + """Lazy initialization of the skill classifier.""" + if self._classifier is None and self._detection_mode in ("llm", "hybrid"): + try: + from .skill_classifier import SkillClassifier + self._classifier = SkillClassifier(self._registry) + except ImportError: + # Fall back to keyword mode if classifier not available + print("[SkillCallbacks] Warning: SkillClassifier not available, using keyword mode") + self._detection_mode = "keyword" + return self._classifier + + def _detect_skills_from_keywords(self, text: str) -> list[str]: + """Detect skills using keyword matching. + + Args: + text: The text to analyze. + + Returns: + List of skill names that matched keywords. + """ + detected_skills = [] + + for skill_name, patterns in self._compiled_patterns.items(): + # Only detect skills that exist in the registry + if skill_name not in self._registry.get_skill_names(): + continue + + for pattern in patterns: + if pattern.search(text): + detected_skills.append(skill_name) + break # One match is enough to activate the skill + + return detected_skills + + def _detect_skills_from_text(self, text: str) -> list[str]: + """Detect which skills should be activated based on text content. + + Uses the configured detection mode (llm, keyword, or hybrid). + + Args: + text: The text to analyze (typically user input). + + Returns: + List of skill names that should be activated. + """ + if self._detection_mode == "keyword": + return self._detect_skills_from_keywords(text) + + elif self._detection_mode == "llm": + classifier = self._get_classifier() + if classifier: + result = classifier.classify(text) + if result.skills: + print(f"[SkillCallbacks] LLM detected: {result.skills} (confidence: {result.confidence:.2f})") + return result.skills + # Fall back to keyword if classifier unavailable + return self._detect_skills_from_keywords(text) + + else: # hybrid mode + classifier = self._get_classifier() + if classifier: + result = classifier.classify_with_fallback( + text, + keyword_detector=self._detect_skills_from_keywords, + ) + if result.skills: + print(f"[SkillCallbacks] Detected: {result.skills} ({result.reasoning})") + return result.skills + # Fall back to keyword if classifier unavailable + return self._detect_skills_from_keywords(text) + + def _get_original_user_message_text(self, llm_request: "LlmRequest") -> str: + """Extract the ORIGINAL user message text from an LLM request. + + This looks for the FIRST user message (not the last), which is the + original user request. In a multi-turn tool-use flow, the conversation + grows but the original user intent is always in the first user message. + + Args: + llm_request: The LLM request containing conversation contents. + + Returns: + The text of the first user message, or empty string if not found. + """ + if not llm_request.contents: + return "" + + # Look for the FIRST user message (original user request) + for content in llm_request.contents: + if content.role == "user" and content.parts: + # Concatenate all text parts + texts = [] + for part in content.parts: + if hasattr(part, "text") and part.text: + texts.append(part.text) + if texts: + return " ".join(texts) + + return "" + + def _get_user_message_text(self, llm_request: "LlmRequest") -> str: + """Extract the latest user message text from an LLM request. + + Args: + llm_request: The LLM request containing conversation contents. + + Returns: + The text of the latest user message, or empty string if not found. + """ + if not llm_request.contents: + return "" + + # Look for the last user message + for content in reversed(llm_request.contents): + if content.role == "user" and content.parts: + # Concatenate all text parts + texts = [] + for part in content.parts: + if hasattr(part, "text") and part.text: + texts.append(part.text) + return " ".join(texts) + + return "" + + def before_model_callback( + self, + callback_context: "CallbackContext", + llm_request: "LlmRequest", + ) -> "LlmResponse | None": + """Auto-activate skills based on user input before LLM processes it. + + This callback analyzes the user message and automatically activates + relevant skills, then DIRECTLY injects their documentation into the + llm_request.system_instruction. This ensures skills are available in + the FIRST LLM call, not just subsequent calls. + + Strategy: + - If NO skills are currently active: Detect from the LATEST user message + (this handles new user requests after skills were cleared) + - If skills ARE already active: Use the ORIGINAL user message to avoid + re-detecting on subsequent LLM calls in the same tool-use flow + - ALWAYS inject skills directly into llm_request.system_instruction + + Args: + callback_context: Context for accessing/modifying state. + llm_request: The LLM request (can be modified). + + Returns: + None to proceed with LLM call (we only modify state, not short-circuit). + """ + # Get current active skills + active_skills: list[str] = list( + callback_context.state.get(ACTIVE_SKILLS_KEY, []) + ) + + # Choose which message to analyze based on current skill state + if not active_skills: + # No skills active: This is likely a NEW user request + # Use the LATEST user message to detect skills + user_text = self._get_user_message_text(llm_request) + if not user_text: + return None + + # Detect skills from the latest message + skills_to_activate = self._detect_skills_from_text(user_text) + + if not skills_to_activate: + print(f"[SkillCallbacks] No skills detected from: {user_text[:100]}...") + return None + + # Activate detected skills + callback_context.state[ACTIVE_SKILLS_KEY] = skills_to_activate + print(f"[SkillCallbacks] Detecting skills from: {user_text[:100]}...") + print(f"[SkillCallbacks] Auto-activated skills: {skills_to_activate}") + + # KEY FIX: Inject skills directly into the llm_request + # This ensures skills are available in the FIRST LLM call + self._inject_skills_into_request(llm_request, skills_to_activate) + else: + # Skills already active: This is a subsequent LLM call in the same flow + # Use the ORIGINAL user message to ensure consistency + user_text = self._get_original_user_message_text(llm_request) + if not user_text: + # Even if no user text, still inject active skills + self._inject_skills_into_request(llm_request, active_skills) + return None + + # Detect which skills should be activated from the original user message + skills_to_activate = self._detect_skills_from_text(user_text) + + # Check if we need to activate any additional skills + newly_activated = [] + for skill_name in skills_to_activate: + if skill_name not in active_skills: + active_skills.append(skill_name) + newly_activated.append(skill_name) + + # Update state if we activated any new skills + if newly_activated: + callback_context.state[ACTIVE_SKILLS_KEY] = active_skills + print(f"[SkillCallbacks] Additional skills detected: {newly_activated}") + + # KEY FIX: Always inject skills into the llm_request for subsequent calls too + self._inject_skills_into_request(llm_request, active_skills) + + # Return None to proceed with LLM call + return None + + def after_agent_callback( + self, + callback_context: "CallbackContext", + ) -> "types.Content | None": + """Auto-deactivate skills after agent completes a turn. + + This callback cleans up active skills to free context for the next + interaction. Skills are ephemeral - they're loaded for a task and + removed when done. + + Args: + callback_context: Context for accessing/modifying state. + + Returns: + None (we don't add any content, just modify state). + """ + if not self._auto_deactivate: + return None + + # Get current active skills + active_skills: list[str] = callback_context.state.get(ACTIVE_SKILLS_KEY, []) + + if active_skills: + # Clear all active skills + callback_context.state[ACTIVE_SKILLS_KEY] = [] + print(f"[SkillCallbacks] Auto-deactivated skills: {active_skills}") + + return None + + +def create_skill_callbacks( + registry: SkillRegistry, + auto_deactivate: bool = True, +) -> SkillCallbacks: + """Factory function to create skill callbacks. + + Args: + registry: The skill registry to use. + auto_deactivate: Whether to auto-deactivate skills after each turn. + + Returns: + SkillCallbacks instance with configured callbacks. + """ + return SkillCallbacks(registry, auto_deactivate=auto_deactivate) + + +# ============================================================================= +# Alternative: Keep skills active until explicitly different task +# ============================================================================= + +class PersistentSkillCallbacks(SkillCallbacks): + """Skill callbacks that keep skills active until task changes. + + This variant doesn't auto-deactivate after each turn. Instead, it only + deactivates skills when the user's request suggests a different task type. + + This is useful for multi-turn conversations about the same topic. + """ + + def __init__(self, registry: SkillRegistry): + super().__init__(registry, auto_deactivate=False) + + def before_model_callback( + self, + callback_context: "CallbackContext", + llm_request: "LlmRequest", + ) -> "LlmResponse | None": + """Auto-activate skills and potentially switch skills if task changes. + + Uses the ORIGINAL user message for skill detection to ensure skills are + activated on the first LLM call, not after tool results come back. + """ + # Extract the ORIGINAL user message (first user message, not last) + user_text = self._get_original_user_message_text(llm_request) + if not user_text: + return None + + # Detect which skills are relevant for this message + relevant_skills = self._detect_skills_from_text(user_text) + + # Get current active skills + active_skills: list[str] = list( + callback_context.state.get(ACTIVE_SKILLS_KEY, []) + ) + + # If we detected new skills, add them + # If we detected different skills (and have some active), switch to them + if relevant_skills: + # Check if this is a task switch (all new skills) + if active_skills and not any(s in active_skills for s in relevant_skills): + # User seems to be switching tasks - deactivate old skills + print(f"[SkillCallbacks] Task switch detected, deactivating: {active_skills}") + active_skills = [] + + # Activate relevant skills + newly_activated = [] + for skill_name in relevant_skills: + if skill_name not in active_skills: + active_skills.append(skill_name) + newly_activated.append(skill_name) + + if newly_activated: + callback_context.state[ACTIVE_SKILLS_KEY] = active_skills + print(f"[SkillCallbacks] Auto-activated skills: {newly_activated}") + + return None diff --git a/contributing/samples/bigquery_skills_demo/skill_classifier.py b/contributing/samples/bigquery_skills_demo/skill_classifier.py new file mode 100644 index 0000000000..f9009c2788 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skill_classifier.py @@ -0,0 +1,291 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Intelligent Skill Classification using LLM. + +This module provides intelligent skill detection using a fast LLM model +to understand user intent and determine which skills are needed. + +Key Features: +1. **LLM-based Classification**: Uses gemini-2.0-flash for fast, accurate intent detection +2. **Semantic Understanding**: Understands paraphrased requests, not just keywords +3. **Confidence Scores**: Returns confidence levels for skill activation decisions +4. **Caching**: Caches recent classifications to avoid redundant API calls +5. **Fallback**: Falls back to keyword matching if LLM is unavailable + +Example use cases that keywords miss but LLM catches: +- "Help me understand patterns in customer behavior" → bqml (clustering) +- "I want to automatically categorize support tickets" → bq_ai_operator +- "Build something to predict next month's sales" → bqml (forecasting) +- "Rate these product descriptions by quality" → bq_ai_operator (AI.SCORE) +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass +from functools import lru_cache +from typing import TYPE_CHECKING + +from google import genai + +from .skill_registry import SkillRegistry + +if TYPE_CHECKING: + pass + + +@dataclass +class SkillClassification: + """Result of skill classification.""" + skills: list[str] # List of skill names to activate + confidence: float # 0.0 to 1.0 + reasoning: str # Brief explanation of why these skills were selected + + +# Classification prompt template - DYNAMIC version +# Skills are injected at runtime from the registry +CLASSIFICATION_PROMPT_TEMPLATE = """\ +You are a skill classifier for a BigQuery data science agent. Your job is to determine which specialized skills (if any) should be loaded based on the user's request. + +Available skills: +{skills_description} + +Analyze the user's request and determine which skills are needed. + +User request: "{user_input}" + +Respond with a JSON object (no markdown, just raw JSON): +{{ + "skills": ["skill_name1", "skill_name2"], + "confidence": 0.85, + "reasoning": "Brief explanation" +}} + +Rules: +- Return empty skills [] if the request is just a simple query (SELECT, list tables, etc.) +- Only return skills from the available list above +- Confidence should be 0.0-1.0 based on how certain you are +- Be conservative: only activate skills when clearly needed +""" + + +class SkillClassifier: + """Intelligent skill classifier using LLM. + + This classifier uses a fast LLM model to understand user intent + and determine which skills should be activated. + + SCALABILITY: Skills are dynamically loaded from the registry, so adding + new skills only requires creating a SKILL.md file - no code changes needed. + """ + + def __init__( + self, + registry: SkillRegistry, + model: str = "gemini-2.0-flash", + confidence_threshold: float = 0.6, + cache_size: int = 100, + ): + """Initialize the skill classifier. + + Args: + registry: The skill registry to validate skill names against. + model: The LLM model to use for classification. + confidence_threshold: Minimum confidence to activate a skill. + cache_size: Number of recent classifications to cache. + """ + self._registry = registry + self._model = model + self._confidence_threshold = confidence_threshold + self._cache_size = cache_size + self._client: genai.Client | None = None + + # Simple in-memory cache for recent classifications + self._cache: dict[str, SkillClassification] = {} + + # Build skills description from registry (dynamic, not hardcoded) + self._skills_description = self._build_skills_description() + + def _build_skills_description(self) -> str: + """Build skills description dynamically from the registry. + + This makes the classifier scalable - new skills are automatically + included without code changes. + """ + descriptions = [] + for i, metadata in enumerate(self._registry.get_all_metadata(), 1): + descriptions.append(f"{i}. **{metadata.name}** - {metadata.description}") + return "\n".join(descriptions) if descriptions else "No skills available." + + def _build_prompt(self, user_input: str) -> str: + """Build the classification prompt with dynamic skills.""" + return CLASSIFICATION_PROMPT_TEMPLATE.format( + skills_description=self._skills_description, + user_input=user_input, + ) + + def _get_client(self) -> genai.Client: + """Get or create the genai client.""" + if self._client is None: + self._client = genai.Client() + return self._client + + def _normalize_input(self, text: str) -> str: + """Normalize input text for caching.""" + # Lowercase and remove extra whitespace + return " ".join(text.lower().split()) + + def _parse_response(self, response_text: str) -> SkillClassification | None: + """Parse the LLM response into a SkillClassification.""" + try: + # Try to extract JSON from the response + # Handle cases where LLM might wrap in markdown code blocks + json_match = re.search(r'\{[^{}]*\}', response_text, re.DOTALL) + if not json_match: + return None + + data = json.loads(json_match.group()) + + # Validate and filter skills + valid_skill_names = set(self._registry.get_skill_names()) + skills = [s for s in data.get("skills", []) if s in valid_skill_names] + + return SkillClassification( + skills=skills, + confidence=float(data.get("confidence", 0.5)), + reasoning=data.get("reasoning", ""), + ) + except (json.JSONDecodeError, KeyError, ValueError): + return None + + def classify(self, user_input: str) -> SkillClassification: + """Classify user input to determine which skills are needed. + + Args: + user_input: The user's message/request. + + Returns: + SkillClassification with recommended skills and confidence. + """ + # Check cache first + cache_key = self._normalize_input(user_input) + if cache_key in self._cache: + return self._cache[cache_key] + + try: + # Call the LLM + client = self._get_client() + prompt = self._build_prompt(user_input) + + response = client.models.generate_content( + model=self._model, + contents=prompt, + config={ + "temperature": 0.1, # Low temperature for consistent classification + "max_output_tokens": 256, + }, + ) + + # Parse the response + result = self._parse_response(response.text) + + if result is None: + # Fallback to empty classification + result = SkillClassification( + skills=[], + confidence=0.0, + reasoning="Failed to parse LLM response", + ) + + # Apply confidence threshold + if result.confidence < self._confidence_threshold: + result = SkillClassification( + skills=[], + confidence=result.confidence, + reasoning=f"Below confidence threshold ({self._confidence_threshold}): {result.reasoning}", + ) + + # Cache the result + if len(self._cache) >= self._cache_size: + # Simple cache eviction: remove first item + first_key = next(iter(self._cache)) + del self._cache[first_key] + self._cache[cache_key] = result + + return result + + except Exception as e: + # Return empty classification on error + return SkillClassification( + skills=[], + confidence=0.0, + reasoning=f"Classification error: {str(e)}", + ) + + def classify_with_fallback( + self, + user_input: str, + keyword_detector: callable | None = None, + ) -> SkillClassification: + """Classify with keyword fallback if LLM fails. + + Args: + user_input: The user's message/request. + keyword_detector: Optional function that returns skill names from text. + + Returns: + SkillClassification with recommended skills. + """ + # Try LLM classification first + result = self.classify(user_input) + + # If LLM returned skills with good confidence, use that + if result.skills and result.confidence >= self._confidence_threshold: + return result + + # Fall back to keyword detection if provided + if keyword_detector is not None: + keyword_skills = keyword_detector(user_input) + if keyword_skills: + return SkillClassification( + skills=keyword_skills, + confidence=0.7, # Medium confidence for keyword match + reasoning="Detected via keyword matching (LLM uncertain)", + ) + + return result + + +# Convenience function for quick classification +def classify_skills( + user_input: str, + registry: SkillRegistry | None = None, +) -> list[str]: + """Quick function to classify skills for a user input. + + Args: + user_input: The user's message. + registry: Optional skill registry (uses default if not provided). + + Returns: + List of skill names to activate. + """ + if registry is None: + registry = SkillRegistry() + + classifier = SkillClassifier(registry) + result = classifier.classify(user_input) + return result.skills diff --git a/contributing/samples/bigquery_skills_demo/skill_registry.py b/contributing/samples/bigquery_skills_demo/skill_registry.py new file mode 100644 index 0000000000..c74a6598c4 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skill_registry.py @@ -0,0 +1,443 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Dynamic Skill Registry following Anthropic's Skills Pattern. + +This module implements the dynamic skill discovery pattern from: +https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills + +Key concepts: +1. Skills are stored as SKILL.md files with YAML frontmatter +2. At startup, only skill names and descriptions are loaded (progressive disclosure) +3. Agent activates/deactivates skills which inject content into system prompt +4. Skill content is in system prompt (not conversation history) - ephemeral! + +This approach mirrors Claude Code's filesystem-based skill loading where: +- Skills are loaded on-demand into the current context +- Skills can be unloaded when no longer needed +- Context doesn't accumulate skill content permanently +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + +from google.adk.agents.readonly_context import ReadonlyContext +from google.adk.tools import ToolContext + + +# State key for tracking active skills (using temp: prefix so it's session-scoped) +ACTIVE_SKILLS_KEY = "active_skills" + + +@dataclass +class SkillMetadata: + """Metadata for a skill (name, description, and keywords).""" + name: str + description: str + path: Path + keywords: list[str] | None = None # Keywords for dynamic pattern matching + + +@dataclass +class SkillContent: + """Full content of a skill including documentation.""" + name: str + description: str + content: str + path: Path + + +class SkillRegistry: + """Registry for dynamically discovering and loading skills. + + Following Anthropic's progressive disclosure pattern: + - Level 1: Skill names and descriptions (loaded at startup) + - Level 2: Full skill content (injected into system prompt when activated) + + Skills are discovered from SKILL.md files in the skills directory. + Each SKILL.md has YAML frontmatter with name and description. + + Example SKILL.md structure: + ```markdown + --- + name: my_skill + description: Short description of what this skill does + --- + + # Full Skill Documentation + + Detailed instructions, examples, and usage patterns... + ``` + """ + + def __init__(self, skills_dir: str | Path | None = None): + """Initialize the skill registry. + + Args: + skills_dir: Directory containing skill subdirectories. + Each subdirectory should have a SKILL.md file. + Defaults to ./skills relative to this module. + """ + if skills_dir is None: + # Default to skills/ directory next to this file + skills_dir = Path(__file__).parent / "skills" + self._skills_dir = Path(skills_dir) + self._skills: dict[str, SkillMetadata] = {} + self._discover_skills() + + def _discover_skills(self) -> None: + """Discover all skills in the skills directory. + + Scans for SKILL.md files and parses YAML frontmatter + to extract name and description (Level 1 disclosure). + """ + if not self._skills_dir.exists(): + return + + for skill_dir in self._skills_dir.iterdir(): + if not skill_dir.is_dir(): + continue + + skill_file = skill_dir / "SKILL.md" + if not skill_file.exists(): + continue + + metadata = self._parse_skill_metadata(skill_file) + if metadata: + self._skills[metadata.name] = metadata + + def _parse_skill_metadata(self, skill_path: Path) -> SkillMetadata | None: + """Parse YAML frontmatter from a SKILL.md file. + + Args: + skill_path: Path to the SKILL.md file + + Returns: + SkillMetadata with name and description, or None if parsing fails + """ + try: + content = skill_path.read_text() + + # Parse YAML frontmatter (between --- delimiters) + frontmatter_match = re.match( + r'^---\s*\n(.*?)\n---\s*\n', + content, + re.DOTALL + ) + + if not frontmatter_match: + return None + + frontmatter = yaml.safe_load(frontmatter_match.group(1)) + + if not frontmatter or 'name' not in frontmatter: + return None + + return SkillMetadata( + name=frontmatter['name'], + description=frontmatter.get('description', ''), + path=skill_path, + keywords=frontmatter.get('keywords'), # Parse keywords from frontmatter + ) + except Exception: + return None + + def get_skill_names(self) -> list[str]: + """Get list of all discovered skill names.""" + return list(self._skills.keys()) + + def get_skill_metadata(self, name: str) -> SkillMetadata | None: + """Get metadata for a specific skill.""" + return self._skills.get(name) + + def get_all_metadata(self) -> list[SkillMetadata]: + """Get metadata for all discovered skills.""" + return list(self._skills.values()) + + def get_all_keywords(self) -> dict[str, list[str]]: + """Get all keywords for all skills. + + Returns a mapping of skill name to list of keywords. + This is used for dynamic keyword pattern matching. + + Returns: + Dict mapping skill names to their keyword lists. + """ + return { + name: metadata.keywords or [] + for name, metadata in self._skills.items() + if metadata.keywords + } + + def load_skill_content(self, name: str) -> SkillContent | None: + """Load the full content of a skill. + + Args: + name: The skill name to load + + Returns: + SkillContent with full documentation, or None if not found + """ + metadata = self._skills.get(name) + if not metadata: + return None + + try: + full_content = metadata.path.read_text() + + # Remove YAML frontmatter for the content + content_match = re.match( + r'^---\s*\n.*?\n---\s*\n(.*)$', + full_content, + re.DOTALL + ) + + if content_match: + content = content_match.group(1).strip() + else: + content = full_content + + return SkillContent( + name=metadata.name, + description=metadata.description, + content=content, + path=metadata.path, + ) + except Exception: + return None + + def get_skills_summary(self) -> str: + """Get a formatted summary of all available skills. + + This is used in the agent's system prompt to inform it + of available skills without loading full content. + + Returns: + Formatted string listing all skills with descriptions + """ + if not self._skills: + return "No skills available." + + lines = ["Available Skills:"] + for name, metadata in sorted(self._skills.items()): + lines.append(f"- **{name}**: {metadata.description}") + + lines.append("") + lines.append("Use `activate_skill(skill_name)` to load a skill's full documentation.") + lines.append("Use `deactivate_skill(skill_name)` to unload a skill when done.") + + return "\n".join(lines) + + +# Module-level registry instance for convenience +_default_registry: SkillRegistry | None = None + + +def get_default_registry() -> SkillRegistry: + """Get the default skill registry (singleton).""" + global _default_registry + if _default_registry is None: + _default_registry = SkillRegistry() + return _default_registry + + +def get_skills_summary() -> str: + """Get summary of all available skills from the default registry.""" + return get_default_registry().get_skills_summary() + + +# ============================================================================= +# Dynamic Instruction Provider (Claude Code-style approach) +# ============================================================================= + +def create_skill_instruction_provider(registry: SkillRegistry): + """Create an instruction provider that injects active skills into system prompt. + + This is the key to ephemeral skill loading! The instruction provider is called + on EVERY LLM request and returns content that goes into the system prompt. + Since it reads from session state, skills can be activated/deactivated and + the system prompt updates automatically. + + Unlike tool responses which persist in conversation history, the system prompt + is rebuilt fresh each time - so deactivated skills truly disappear from context. + + Args: + registry: The skill registry to load skills from + + Returns: + An instruction provider function compatible with LlmAgent.instruction + """ + def instruction_provider(ctx: ReadonlyContext) -> str: + """Generate dynamic instructions based on active skills.""" + # Get active skills from session state + active_skills: list[str] = ctx.state.get(ACTIVE_SKILLS_KEY, []) + + if not active_skills: + return "" # No active skills, no additional instructions + + # Build skill content section + skill_sections = [] + for skill_name in active_skills: + skill = registry.load_skill_content(skill_name) + if skill: + skill_sections.append(f""" +## Active Skill: {skill.name} + +{skill.description} + +--- + +{skill.content} +""") + + if not skill_sections: + return "" + + return f""" +# Currently Active Skills + +The following skills have been loaded and are available for this task: + +{"".join(skill_sections)} + +--- +**Note**: Use `deactivate_skill(skill_name)` when you're done with a skill to free up context. +""" + + return instruction_provider + + +# ============================================================================= +# Skill Activation/Deactivation Tools +# ============================================================================= + +def activate_skill(skill_name: str, tool_context: ToolContext) -> str: + """Activate a skill to load its full documentation into context. + + When activated, the skill's content will be injected into the system prompt + for all subsequent LLM calls. This is ephemeral - the content is NOT stored + in conversation history and can be removed by calling deactivate_skill. + + Args: + skill_name: Name of the skill to activate (e.g., 'bqml', 'bq_ai_operator') + + Returns: + Confirmation message or error if skill not found + """ + registry = get_default_registry() + + # Verify skill exists + if skill_name not in registry.get_skill_names(): + available = ", ".join(registry.get_skill_names()) + return f"Skill '{skill_name}' not found. Available skills: {available}" + + # Get current active skills from state + active_skills: list[str] = list(tool_context.state.get(ACTIVE_SKILLS_KEY, [])) + + # Add skill if not already active + if skill_name not in active_skills: + active_skills.append(skill_name) + tool_context.state[ACTIVE_SKILLS_KEY] = active_skills + + # Get skill metadata for confirmation + metadata = registry.get_skill_metadata(skill_name) + return f"Activated skill '{skill_name}': {metadata.description}\n\nThe skill documentation is now available in your context. Use deactivate_skill('{skill_name}') when done." + else: + return f"Skill '{skill_name}' is already active." + + +def deactivate_skill(skill_name: str, tool_context: ToolContext) -> str: + """Deactivate a skill to remove its documentation from context. + + This removes the skill content from the system prompt, freeing up context + space for other information. The skill can be reactivated later if needed. + + Args: + skill_name: Name of the skill to deactivate + + Returns: + Confirmation message + """ + # Get current active skills from state + active_skills: list[str] = list(tool_context.state.get(ACTIVE_SKILLS_KEY, [])) + + if skill_name in active_skills: + active_skills.remove(skill_name) + tool_context.state[ACTIVE_SKILLS_KEY] = active_skills + return f"Deactivated skill '{skill_name}'. Its documentation has been removed from context." + else: + return f"Skill '{skill_name}' is not currently active." + + +def list_active_skills(tool_context: ToolContext) -> str: + """List all currently active skills. + + Returns: + List of active skill names or message if none active + """ + active_skills: list[str] = tool_context.state.get(ACTIVE_SKILLS_KEY, []) + + if not active_skills: + return "No skills are currently active. Use activate_skill(skill_name) to load a skill." + + registry = get_default_registry() + lines = ["Currently active skills:"] + for name in active_skills: + metadata = registry.get_skill_metadata(name) + if metadata: + lines.append(f"- **{name}**: {metadata.description}") + else: + lines.append(f"- **{name}**: (metadata not found)") + + return "\n".join(lines) + + +# ============================================================================= +# Legacy load_skill function (for backward compatibility) +# ============================================================================= + +def load_skill(skill_name: str) -> str: + """Load a skill's documentation (legacy function). + + Note: This returns the skill content as a string which will persist in + conversation history. For ephemeral skill loading, use activate_skill() + instead with the dynamic instruction provider. + + Args: + skill_name: Name of the skill to load + + Returns: + Full skill documentation + """ + registry = get_default_registry() + skill = registry.load_skill_content(skill_name) + + if skill is None: + available = ", ".join(registry.get_skill_names()) + return f"Skill '{skill_name}' not found. Available skills: {available}" + + return f"""# Skill: {skill.name} + +{skill.description} + +--- + +{skill.content} +""" diff --git a/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md new file mode 100644 index 0000000000..e5440ca822 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md @@ -0,0 +1,357 @@ +--- +name: bq_ai_operator +description: BigQuery AI Operator - Use managed AI functions (AI.CLASSIFY, AI.IF, AI.SCORE) directly in SQL for text classification, filtering, and scoring. Requires a BigQuery connection to Vertex AI. +keywords: + - ai.classify + - ai.if + - ai.score + - ai function + - ai operator + - text classification + - sentiment + - categorize + - categories + - natural language + - filter text + - score text + - score + - rate content + - rank content + - rate + - rank + - classify + - positive + - negative + - vertex ai + - managed ai + - list connections + - connection_id +--- + +# BQ AI Operator Skill (Managed AI Functions in SQL) + +Use managed AI functions directly in BigQuery SQL queries for text classification, filtering, and scoring. + +**IMPORTANT**: These are the NEW managed AI functions that require a `connection_id` to Vertex AI, NOT the older `ML.GENERATE_TEXT` style functions. + +## Prerequisites + +1. **A BigQuery connection to Vertex AI is required** for all AI functions. + +2. **Grant the connection service account access to Vertex AI** + +## Connection Workflow (ALWAYS Follow This) + +**CRITICAL**: AI functions require a `connection_id` to a BigQuery connection to Vertex AI. + +### ⚠️ IMPORTANT: Location Matching Rule + +**The connection location MUST match your dataset location!** + +| Dataset Location | Connection Location | Example | +|------------------|---------------------|---------| +| `US` (multi-region) | `us` | `us.my_ai_connection` | +| `EU` (multi-region) | `eu` | `eu.my_ai_connection` | +| `us-central1` (regional) | `us-central1` | `us-central1.my_ai_connection` | + +**Common Error**: Using `us-central1.my_connection` with a dataset in `US` multi-region will fail with "Dataset not found in location us-central1". + +**How to check dataset location**: +```sql +SELECT option_value FROM `project.dataset.INFORMATION_SCHEMA.SCHEMATA_OPTIONS` WHERE option_name = 'location' +``` + +### Step 1: Determine Your Dataset Location + +Before listing connections, identify where your target dataset is located: +- Most BigQuery public datasets are in `US` multi-region +- Your own datasets might be in `US`, `EU`, or a specific region like `us-central1` + +### Step 2: List Connections in the SAME Location + +Use the `list_connections` tool with the **same location as your dataset**: + +``` +# For datasets in US multi-region: +list_connections(project_id="your-project", location="us") + +# For datasets in us-central1: +list_connections(project_id="your-project", location="us-central1") +``` + +This returns all available connections with their `connection_id` and `service_account`. + +### Step 3: Use an Existing Connection If Available + +If `list_connections` returns connections, **use one of them**. Pick a connection that: +- Has `connection_type: "CLOUD_RESOURCE"` (required for Vertex AI) +- Is in the **SAME location as your dataset** + +Use the `connection_id` from the result, formatted as `location.connection_id`: +- Example: If connection_id is `my_ai_connection` in location `us`, use `us.my_ai_connection` + +### Step 4: Only Create a New Connection If None Exist + +**Only if `list_connections` returns empty or no suitable connections**, create a new one in the **same location as your dataset**: + +``` +# For US multi-region datasets: +create_connection(project_id="your-project", location="us", connection_id="my_ai_connection") + +# For us-central1 datasets: +create_connection(project_id="your-project", location="us-central1", connection_id="my_ai_connection") +``` + +This automatically: +1. Creates the connection +2. Grants the Vertex AI User role to the service account (required for AI functions) + +### Connection ID Formats + +When using connections in SQL: +- `us.my_connection` (location.connection_name) - **Preferred for US multi-region** +- `us-central1.my_connection` - **For regional datasets** +- `project_id.us.my_connection` (fully qualified) + +## Available Managed AI Functions + +| Function | Purpose | Return Type | +|----------|---------|-------------| +| AI.CLASSIFY | Categorize text into classes | STRING | +| AI.IF | Natural language TRUE/FALSE filtering | BOOL | +| AI.SCORE | Rate/rank by criteria (0.0 to 1.0) | FLOAT64 | + +--- + +## AI.CLASSIFY - Categorize Text + +Classify text into one of the provided categories. + +### Syntax +```sql +AI.CLASSIFY( + input, -- STRING: the text to classify + categories => ['cat1', 'cat2'], -- ARRAY: possible categories + connection_id => 'LOCATION.CONNECTION_NAME' +) +``` + +### Examples + +**News article classification:** +```sql +SELECT + title, + body, + AI.CLASSIFY( + body, + categories => ['tech', 'sport', 'business', 'politics', 'entertainment', 'other'], + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS category +FROM `bigquery-public-data.bbc_news.fulltext` +LIMIT 10; +``` + +**Sentiment classification with descriptions:** +```sql +SELECT + review_text, + AI.CLASSIFY( + review_text, + categories => [ + ('positive', 'happy, satisfied, recommends'), + ('negative', 'unhappy, disappointed, complaints'), + ('neutral', 'factual, no strong emotion') + ], + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS sentiment +FROM `project.dataset.reviews` +LIMIT 10; +``` + +--- + +## AI.IF - Natural Language Filtering + +Returns TRUE or FALSE based on a natural language condition. + +### Syntax +```sql +AI.IF( + input, -- STRING: the text to evaluate + condition, -- STRING: natural language condition + connection_id => 'LOCATION.CONNECTION_NAME' +) +``` + +### Examples + +**Filter for eco-friendly products:** +```sql +SELECT product_name, description +FROM `project.products.catalog` +WHERE AI.IF( + description, + 'This product is eco-friendly, sustainable, or environmentally conscious', + connection_id => 'us.my_ai_connection' -- Use your connection: test-project-0728-467323.us.my_ai_connection +) = TRUE +LIMIT 10; +``` + +**Content moderation:** +```sql +SELECT + post_id, + content, + AI.IF( + content, + 'This content is appropriate for all ages and contains no spam, harassment, or explicit material', + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS is_appropriate +FROM `project.social.user_posts` +LIMIT 10; +``` + +--- + +## AI.SCORE - Quality Scoring + +Returns a FLOAT64 score based on your scoring criteria. Commonly used with ORDER BY for ranking. + +### Syntax +```sql +AI.SCORE( + (prompt_with_criteria, column_to_score), -- TUPLE: (STRING literal, column reference) + connection_id => 'LOCATION.CONNECTION_NAME' +) +``` + +**CRITICAL**: The first argument is a **TUPLE** with parentheses containing: +1. A STRING literal describing the scoring criteria +2. A column reference to the text being scored + +### Examples + +**Review helpfulness scoring:** +```sql +SELECT + review_id, + review_text, + star_rating, + AI.SCORE( + ('Rate the helpfulness of this review based on detail level and examples. Review: ', review_text), + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS helpfulness_score +FROM `project.reviews.product_reviews` +ORDER BY helpfulness_score DESC +LIMIT 10; +``` + +**Movie review rating (from official docs):** +```sql +SELECT + AI.SCORE(( + 'On a scale from 1 to 10, rate how much the reviewer liked the movie. Review: ', + review), + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS ai_rating, + reviewer_rating AS human_rating, + review +FROM `bigquery-public-data.imdb.reviews` +WHERE title = 'The English Patient' +ORDER BY ai_rating DESC +LIMIT 10; +``` + +**Negativity scoring:** +```sql +SELECT + review, + AI.SCORE( + ('Rate negativity from 1-10: ', review), + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS negativity_score +FROM product_reviews +ORDER BY negativity_score DESC +LIMIT 5; +``` + +**Relevance scoring:** +```sql +SELECT + document_id, + title, + AI.SCORE( + ('How relevant is this document to machine learning and AI topics? Document: ', content), + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS ml_relevance +FROM `project.docs.articles` +ORDER BY ml_relevance DESC +LIMIT 10; +``` + +--- + +## Complete Pipeline Example + +Combine multiple AI functions for a review intelligence pipeline: + +```sql +-- Step 1: Classify and score reviews +WITH classified AS ( + SELECT + review_id, + review_text, + star_rating, + AI.CLASSIFY( + review_text, + categories => ['positive', 'negative', 'neutral', 'mixed'], + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS sentiment, + AI.SCORE( + ('Rate review quality based on detail and helpfulness. Review: ', review_text), + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS quality_score + FROM `project.reviews.raw_reviews` + LIMIT 100 +) +-- Step 2: Filter appropriate content and categorize +SELECT + review_id, + sentiment, + quality_score, + CASE + WHEN quality_score >= 0.8 THEN 'featured' + WHEN quality_score >= 0.5 THEN 'standard' + ELSE 'low_quality' + END AS tier +FROM classified +WHERE AI.IF( + review_text, + 'Content is appropriate and not spam', + connection_id => 'us.my_ai_connection' -- Use your connection: test-project-0728-467323.us.my_ai_connection +) = TRUE +ORDER BY quality_score DESC; +``` + +--- + +## Important Notes + +1. **Connection Required**: All managed AI functions require a `connection_id` to a Vertex AI connection +2. **Preview Feature**: AI.CLASSIFY, AI.IF, and AI.SCORE are in public preview +3. **Region Support**: Works in all Gemini regions plus US/EU multi-regions +4. **Use LIMIT**: Always use LIMIT to control costs when testing +5. **String Return**: AI.CLASSIFY returns STRING, AI.IF returns BOOL, AI.SCORE returns FLOAT64 +6. **Escape Single Quotes**: When using string literals with apostrophes, escape them by doubling: + - WRONG: `'The surgeon who 'sees' inside patients'` + - CORRECT: `'The surgeon who ''sees'' inside patients'` + +## Troubleshooting + +**Error: "connection not found"** +- Ensure you've created the connection: `CREATE CLOUD RESOURCE CONNECTION` +- Use the correct format: `LOCATION.CONNECTION_NAME` (e.g., `us.my_ai_connection`) + +**Error: "permission denied"** +- Grant the connection's service account access to Vertex AI API diff --git a/contributing/samples/bigquery_skills_demo/skills/bq_remote_model/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bq_remote_model/SKILL.md new file mode 100644 index 0000000000..98d5ff3676 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skills/bq_remote_model/SKILL.md @@ -0,0 +1,429 @@ +--- +name: bq_remote_model +description: BigQuery Remote Models - Create remote models connecting to Vertex AI endpoints for text generation (Gemini, Claude, Llama), embeddings, and custom deployed models. Use AI.GENERATE_TEXT and AI.GENERATE_EMBEDDING functions. +keywords: + - remote model + - create remote model + - vertex ai + - generate text + - ai.generate_text + - generate embedding + - ai.generate_embedding + - gemini + - claude + - llama + - text generation + - embeddings + - llm + - foundation model + - hugging face + - model garden + - endpoint + - list connections + - connection_id +--- + +# BQ Remote Model Skill (Remote Models with Vertex AI) + +Create and use remote models that connect BigQuery to Vertex AI endpoints for text generation, embeddings, and custom ML models. + +**IMPORTANT**: Remote models require a BigQuery connection to Vertex AI. This is different from BQML (which trains models in BigQuery) and BQ AI Operator (which uses managed AI functions like AI.CLASSIFY). + +## Prerequisites + +1. **A BigQuery connection to Vertex AI is required** for all remote models. + +2. **Grant the connection's service account the Vertex AI User role** + +## Connection Workflow (ALWAYS Follow This) + +**CRITICAL**: Remote models require a `connection_id` to a BigQuery connection to Vertex AI. + +### ⚠️ IMPORTANT: Location Matching Rule + +**The connection location MUST match your dataset location!** + +| Dataset Location | Connection Location | Example | +|------------------|---------------------|---------| +| `US` (multi-region) | `us` | `us.my_vertex_connection` | +| `EU` (multi-region) | `eu` | `eu.my_vertex_connection` | +| `us-central1` (regional) | `us-central1` | `us-central1.my_vertex_connection` | + +**Common Error**: Using `us-central1.my_connection` with a dataset in `US` multi-region will fail with "Dataset not found in location us-central1". + +**How to check dataset location**: +```sql +SELECT option_value FROM `project.dataset.INFORMATION_SCHEMA.SCHEMATA_OPTIONS` WHERE option_name = 'location' +``` + +### Step 1: Determine Your Dataset Location + +Before listing connections, identify where your target dataset is located: +- Most BigQuery public datasets are in `US` multi-region +- Your own datasets might be in `US`, `EU`, or a specific region like `us-central1` + +### Step 2: List Connections in the SAME Location + +Use the `list_connections` tool with the **same location as your dataset**: + +``` +# For datasets in US multi-region: +list_connections(project_id="your-project", location="us") + +# For datasets in us-central1: +list_connections(project_id="your-project", location="us-central1") +``` + +This returns all available connections with their `connection_id` and `service_account`. + +### Step 3: Use an Existing Connection If Available + +If `list_connections` returns connections, **use one of them**. Pick a connection that: +- Has `connection_type: "CLOUD_RESOURCE"` (required for Vertex AI) +- Is in the **SAME location as your dataset** + +Use the `connection_id` from the result, formatted as `location.connection_id`: +- Example: If connection_id is `my_ai_connection` in location `us`, use `us.my_ai_connection` + +### Step 4: Only Create a New Connection If None Exist + +**Only if `list_connections` returns empty or no suitable connections**, create a new one in the **same location as your dataset**: + +``` +# For US multi-region datasets: +create_connection(project_id="your-project", location="us", connection_id="my_vertex_connection") + +# For us-central1 datasets: +create_connection(project_id="your-project", location="us-central1", connection_id="my_vertex_connection") +``` + +This automatically: +1. Creates the connection +2. Grants the Vertex AI User role to the service account (required for remote models) + +### Connection ID Formats + +When using connections in SQL: +- `us.my_connection` (location.connection_name) - **Preferred** +- `project_id.us.my_connection` (fully qualified) + +--- + +## CREATE REMOTE MODEL Syntax + +### For Google/Partner Models (Gemini, Claude) + +```sql +CREATE OR REPLACE MODEL `project.dataset.model_name` +REMOTE WITH CONNECTION `project.region.connection_id` +OPTIONS (ENDPOINT = 'endpoint_name'); +``` + +### ⚠️ DEFAULT MODEL: Always Use Gemini 2.5 Pro + +**ALWAYS use `gemini-2.5-pro` as the default model** unless the user specifically requests a different model. + +```sql +-- RECOMMENDED: Use gemini-2.5-pro by default +CREATE OR REPLACE MODEL `project.dataset.gemini_model` +REMOTE WITH CONNECTION `us.my_connection` +OPTIONS (ENDPOINT = 'gemini-2.5-pro'); +``` + +**Common ENDPOINT values:** +| Model | Endpoint | When to Use | +|-------|----------|-------------| +| **Gemini 2.5 Pro** | `gemini-2.5-pro` | **DEFAULT** - Best quality, use for all tasks unless specified otherwise | +| Gemini 2.5 Flash | `gemini-2.5-flash` | Only if user requests faster/cheaper processing | +| Claude 3.5 Sonnet | `claude-3-5-sonnet@20240620` | Only if user specifically requests Claude | +| Text Embedding | `text-embedding-004` | For embeddings/vector search | +| Gemini Embedding | `gemini-embedding-001` | For embeddings (larger dimension) | + +**Legacy models (avoid):** `gemini-2.0-flash`, `gemini-1.5-pro` - Use 2.5 versions instead. + +### For Open Models (Hugging Face / Model Garden) + +```sql +CREATE OR REPLACE MODEL `project.dataset.model_name` +REMOTE WITH CONNECTION `project.region.connection_id` +OPTIONS ( + HUGGING_FACE_MODEL_ID = 'meta-llama/Llama-2-7b-chat-hf', + HUGGING_FACE_TOKEN = 'your_token', -- Optional, for gated models + MACHINE_TYPE = 'n1-standard-4', + MIN_REPLICA_COUNT = 1, + MAX_REPLICA_COUNT = 3, + ENDPOINT_IDLE_TTL = INTERVAL 1 HOUR +); +``` + +--- + +## AI.GENERATE_TEXT - Text Generation + +Generate text using LLMs like Gemini, Claude, or Llama. + +### Basic Syntax + +```sql +SELECT * +FROM AI.GENERATE_TEXT( + MODEL `project.dataset.model_name`, + (SELECT 'Your prompt here' AS prompt), + STRUCT( + 1024 AS max_output_tokens, + 0.7 AS temperature, + 0.95 AS top_p + ) +); +``` + +### ⚠️ CRITICAL: Task-Specific Parameter Settings + +**The `max_output_tokens` parameter is crucial** - set it appropriately for the task type: + +| Task Type | max_output_tokens | temperature | Example Use Case | +|-----------|-------------------|-------------|------------------| +| **Summarization** | `512-2048` | `0.2-0.4` | Summarize articles, extract key points | +| **Long-form generation** | `2048-8192` | `0.5-0.7` | Write essays, detailed explanations | +| **Classification/Labeling** | `50-100` | `0.0-0.2` | Classify text, extract labels | +| **Short answers** | `100-256` | `0.2-0.3` | Q&A, simple extractions | +| **Creative writing** | `1024-4096` | `0.7-0.9` | Stories, creative content | + +**Guidelines:** +- **Summarization tasks**: Use `max_output_tokens` of `512-1024` for single-document summaries, `1024-2048` for multi-document summaries +- **Classification tasks**: Use small `max_output_tokens` (50-100) since output is typically a single word or short phrase +- **Low temperature (0.0-0.3)**: For factual, deterministic outputs (classification, extraction) +- **High temperature (0.5-0.8)**: For creative, varied outputs (writing, brainstorming) + +### Parameters for Gemini Models + +| Parameter | Type | Range | Default | Description | +|-----------|------|-------|---------|-------------| +| `max_output_tokens` | INT64 | 1-8192 | 128 | **IMPORTANT**: Set based on task (see table above) | +| `temperature` | FLOAT64 | 0.0-1.0 | 0.0 | Randomness (0=deterministic, 1=creative) | +| `top_p` | FLOAT64 | 0.0-1.0 | 0.95 | Nucleus sampling threshold | +| `stop_sequences` | ARRAY | - | [] | Stop generation at these sequences | +| `ground_with_google_search` | BOOL | - | FALSE | Enable Google Search grounding | +| `request_type` | STRING | DEDICATED/SHARED | UNSPECIFIED | Resource allocation | + +### Parameters for Claude Models + +| Parameter | Type | Range | Default | +|-----------|------|-------|---------| +| `max_output_tokens` | INT64 | 1-4096 | 128 | +| `top_k` | INT64 | 1-40 | - | +| `top_p` | FLOAT64 | 0.0-1.0 | - | + +### Example: Text Summarization (Large max_output_tokens) + +```sql +-- Step 1: Create the remote model (ALWAYS use gemini-2.5-pro by default) +CREATE OR REPLACE MODEL `project.bq_demo.gemini_model` +REMOTE WITH CONNECTION `us.my_vertex_connection` +OPTIONS (ENDPOINT = 'gemini-2.5-pro'); -- DEFAULT: Always use 2.5-pro unless specified + +-- Step 2: Summarize text (use 512-1024 tokens for summaries) +SELECT + title, + ml_generate_text_result AS summary +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT + title, + CONCAT('Summarize this article in 2-3 paragraphs:\n\n', body) AS prompt + FROM `bigquery-public-data.bbc_news.fulltext` + LIMIT 5), + STRUCT( + 1024 AS max_output_tokens, -- LARGE for summarization + 0.3 AS temperature -- Low for factual output + ) +); +``` + +### Example: Text Classification (Small max_output_tokens) + +```sql +-- Classification task: Use small max_output_tokens since output is just a label +SELECT + review_id, + review_text, + ml_generate_text_result AS sentiment +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT + review_id, + review_text, + CONCAT('Classify the sentiment of this review as POSITIVE, NEGATIVE, or NEUTRAL. Only output the label, nothing else.\n\nReview: ', review_text) AS prompt + FROM `project.dataset.reviews` + LIMIT 10), + STRUCT( + 50 AS max_output_tokens, -- SMALL for classification (just a single word) + 0.0 AS temperature -- Zero for deterministic output + ) +); +``` + +### Example: Batch Summarization from Table + +```sql +SELECT + review_id, + review_text, + ml_generate_text_result AS summary +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT + review_id, + review_text, + CONCAT('Summarize this review in one sentence: ', review_text) AS prompt + FROM `project.dataset.reviews` + LIMIT 10), + STRUCT( + 256 AS max_output_tokens, -- Medium for short summaries + 0.2 AS temperature + ) +); +``` + +--- + +## AI.GENERATE_EMBEDDING - Text Embeddings + +Generate vector embeddings for text, useful for semantic search and similarity. + +### Syntax + +```sql +SELECT * +FROM AI.GENERATE_EMBEDDING( + MODEL `project.dataset.embedding_model`, + (SELECT content FROM table), + STRUCT( + 'RETRIEVAL_DOCUMENT' AS task_type, + 768 AS output_dimensionality + ) +); +``` + +### Task Types + +| Task Type | Description | Use Case | +|-----------|-------------|----------| +| `RETRIEVAL_QUERY` | Optimize for queries | Search queries | +| `RETRIEVAL_DOCUMENT` | Optimize for documents | Document indexing | +| `SEMANTIC_SIMILARITY` | Compute similarity | Finding similar texts | +| `CLASSIFICATION` | Text classification | Categorization | +| `CLUSTERING` | Group similar texts | Topic modeling | +| `QUESTION_ANSWERING` | Q&A tasks | FAQ systems | +| `FACT_VERIFICATION` | Verify facts | Fact checking | +| `CODE_RETRIEVAL_QUERY` | Code search | Code similarity | + +### Dimensionality + +| Model | Dimension Range | Default | +|-------|-----------------|---------| +| `gemini-embedding-001` | 1-3072 | 3072 | +| `text-embedding-004` | 1-768 | 768 | + +### Example: Create Embeddings for Semantic Search + +```sql +-- Step 1: Create embedding model +CREATE OR REPLACE MODEL `project.bq_demo.embedding_model` +REMOTE WITH CONNECTION `us.my_vertex_connection` +OPTIONS (ENDPOINT = 'text-embedding-004'); + +-- Step 2: Generate embeddings for documents +SELECT + doc_id, + title, + ml_generate_embedding_result AS embedding +FROM AI.GENERATE_EMBEDDING( + MODEL `project.bq_demo.embedding_model`, + (SELECT doc_id, title, content FROM `project.dataset.documents` LIMIT 100), + STRUCT('RETRIEVAL_DOCUMENT' AS task_type, 768 AS output_dimensionality) +); +``` + +### Example: Vector Similarity Search + +```sql +-- Find similar documents using cosine distance +WITH query_embedding AS ( + SELECT ml_generate_embedding_result AS embedding + FROM AI.GENERATE_EMBEDDING( + MODEL `project.bq_demo.embedding_model`, + (SELECT 'machine learning best practices' AS content), + STRUCT('RETRIEVAL_QUERY' AS task_type) + ) +) +SELECT + d.doc_id, + d.title, + ML.DISTANCE(d.embedding, q.embedding, 'COSINE') AS similarity +FROM `project.dataset.doc_embeddings` d +CROSS JOIN query_embedding q +ORDER BY similarity ASC +LIMIT 10; +``` + +--- + +## Complete Pipeline Example + +Build a RAG (Retrieval Augmented Generation) pipeline: + +```sql +-- Step 1: Find relevant documents using embeddings +WITH relevant_docs AS ( + SELECT title, content + FROM AI.GENERATE_EMBEDDING( + MODEL `project.bq_demo.embedding_model`, + (SELECT 'What are the benefits of serverless?' AS content), + STRUCT('RETRIEVAL_QUERY' AS task_type) + ) query + CROSS JOIN ( + SELECT doc_id, title, content, embedding + FROM `project.dataset.doc_embeddings` + ) docs + ORDER BY ML.DISTANCE(docs.embedding, query.ml_generate_embedding_result, 'COSINE') + LIMIT 3 +) +-- Step 2: Generate response using retrieved context +SELECT ml_generate_text_result AS answer +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT CONCAT( + 'Based on these documents:\n', + STRING_AGG(content, '\n\n'), + '\n\nAnswer: What are the benefits of serverless?' + ) AS prompt FROM relevant_docs), + STRUCT(512 AS max_output_tokens, 0.3 AS temperature) +); +``` + +--- + +## Important Notes + +1. **Connection Required**: All remote models need a Vertex AI connection +2. **Region Matching**: Dataset and connection must be in the same region +3. **Cost**: Remote model calls incur Vertex AI API costs +4. **Rate Limits**: Be mindful of Vertex AI quotas when processing large datasets +5. **Use LIMIT**: Always use LIMIT when testing to control costs +6. **Escape Single Quotes**: When using string literals with apostrophes, escape them by doubling: + - WRONG: `'The surgeon who 'sees' inside patients'` + - CORRECT: `'The surgeon who ''sees'' inside patients'` + +## Troubleshooting + +**Error: "connection not found"** +- Verify connection exists: `SELECT * FROM region-us.INFORMATION_SCHEMA.CONNECTIONS` +- Use correct format: `project.region.connection_id` + +**Error: "model not found"** +- Check endpoint spelling matches exactly +- Verify model is available in your region + +**Error: "permission denied"** +- Grant Vertex AI User role to connection's service account diff --git a/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md new file mode 100644 index 0000000000..40d5bfb61f --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md @@ -0,0 +1,175 @@ +--- +name: bqml +description: BigQuery ML - Train, evaluate, and deploy machine learning models using SQL. Supports regression, classification, clustering, time series forecasting, and deep learning. +keywords: + - ml + - machine learning + - train + - model + - predict + - cluster + - kmeans + - regression + - classification + - forecast + - arima + - xgboost + - boosted + - random forest + - feature importance + - evaluate +--- + +# BQML Skill (BigQuery Machine Learning) + +Train, evaluate, and deploy ML models directly in BigQuery using SQL. + +## Supported Model Types + +| Model Type | Use Case | SQL Option | +|------------|----------|------------| +| LINEAR_REG | Numeric prediction | `model_type='LINEAR_REG'` | +| LOGISTIC_REG | Binary/multiclass classification | `model_type='LOGISTIC_REG'` | +| KMEANS | Customer segmentation, clustering | `model_type='KMEANS'` | +| BOOSTED_TREE_REGRESSOR | Numeric prediction with XGBoost | `model_type='BOOSTED_TREE_REGRESSOR'` | +| BOOSTED_TREE_CLASSIFIER | Classification with XGBoost | `model_type='BOOSTED_TREE_CLASSIFIER'` | +| RANDOM_FOREST_REGRESSOR | Ensemble numeric prediction | `model_type='RANDOM_FOREST_REGRESSOR'` | +| RANDOM_FOREST_CLASSIFIER | Ensemble classification | `model_type='RANDOM_FOREST_CLASSIFIER'` | +| ARIMA_PLUS | Time series forecasting | `model_type='ARIMA_PLUS'` | +| DNN_REGRESSOR | Deep learning regression | `model_type='DNN_REGRESSOR'` | +| DNN_CLASSIFIER | Deep learning classification | `model_type='DNN_CLASSIFIER'` | + +## Core Workflow + +### Step 1: Train a Model +```sql +CREATE OR REPLACE MODEL `project.dataset.model_name` +OPTIONS( + model_type='LINEAR_REG', + input_label_cols=['target_column'], + enable_global_explain=TRUE +) AS +SELECT feature1, feature2, feature3, target_column +FROM `project.dataset.training_data` +WHERE target_column IS NOT NULL; +``` + +### Step 2: Evaluate the Model +```sql +SELECT * FROM ML.EVALUATE(MODEL `project.dataset.model_name`); +``` + +### Step 3: Get Feature Importance +```sql +SELECT * FROM ML.GLOBAL_EXPLAIN(MODEL `project.dataset.model_name`); +``` + +### Step 4: Make Predictions +```sql +SELECT * FROM ML.PREDICT( + MODEL `project.dataset.model_name`, + (SELECT feature1, feature2, feature3 FROM `project.dataset.new_data`) +); +``` + +### Step 5: Explain Predictions +```sql +SELECT * FROM ML.EXPLAIN_PREDICT( + MODEL `project.dataset.model_name`, + (SELECT feature1, feature2, feature3 FROM `project.dataset.new_data`), + STRUCT(3 as top_k_features) +); +``` + +## Example: Penguin Body Mass Prediction + +```sql +-- Train model +CREATE OR REPLACE MODEL `project.bqml_demo.penguin_weight` +OPTIONS( + model_type='LINEAR_REG', + input_label_cols=['body_mass_g'], + enable_global_explain=TRUE +) AS +SELECT species, island, culmen_length_mm, culmen_depth_mm, + flipper_length_mm, sex, body_mass_g +FROM `bigquery-public-data.ml_datasets.penguins` +WHERE body_mass_g IS NOT NULL AND sex IS NOT NULL; + +-- Evaluate +SELECT * FROM ML.EVALUATE(MODEL `project.bqml_demo.penguin_weight`); + +-- Feature importance +SELECT * FROM ML.GLOBAL_EXPLAIN(MODEL `project.bqml_demo.penguin_weight`); + +-- Predict +SELECT predicted_body_mass_g, species, island +FROM ML.PREDICT( + MODEL `project.bqml_demo.penguin_weight`, + (SELECT 'Adelie' as species, 'Torgersen' as island, + 39.1 as culmen_length_mm, 18.7 as culmen_depth_mm, + 181.0 as flipper_length_mm, 'MALE' as sex) +); +``` + +## Example: K-Means Clustering + +```sql +-- Create clustering model +CREATE OR REPLACE MODEL `project.bqml_demo.penguin_clusters` +OPTIONS( + model_type='KMEANS', + num_clusters=3, + standardize_features=TRUE +) AS +SELECT culmen_length_mm, culmen_depth_mm, flipper_length_mm, body_mass_g +FROM `bigquery-public-data.ml_datasets.penguins` +WHERE body_mass_g IS NOT NULL; + +-- Get cluster assignments +SELECT * FROM ML.PREDICT( + MODEL `project.bqml_demo.penguin_clusters`, + (SELECT culmen_length_mm, culmen_depth_mm, flipper_length_mm, body_mass_g + FROM `bigquery-public-data.ml_datasets.penguins` + WHERE body_mass_g IS NOT NULL) +); + +-- Analyze cluster centroids +SELECT * FROM ML.CENTROIDS(MODEL `project.bqml_demo.penguin_clusters`); +``` + +## Example: XGBoost Classification + +```sql +CREATE OR REPLACE MODEL `project.bqml_demo.species_classifier` +OPTIONS( + model_type='BOOSTED_TREE_CLASSIFIER', + input_label_cols=['species'], + num_parallel_tree=1, + max_tree_depth=6, + subsample=0.8, + data_split_method='AUTO_SPLIT' +) AS +SELECT island, culmen_length_mm, culmen_depth_mm, + flipper_length_mm, body_mass_g, sex, species +FROM `bigquery-public-data.ml_datasets.penguins` +WHERE species IS NOT NULL AND sex IS NOT NULL; + +-- Confusion matrix +SELECT * FROM ML.CONFUSION_MATRIX(MODEL `project.bqml_demo.species_classifier`); + +-- ROC curve (for binary classification) +SELECT * FROM ML.ROC_CURVE(MODEL `project.bqml_demo.species_classifier`); +``` + +## Key ML Functions + +- `ML.EVALUATE()` - Model performance metrics +- `ML.PREDICT()` - Generate predictions +- `ML.EXPLAIN_PREDICT()` - Predictions with feature attributions +- `ML.GLOBAL_EXPLAIN()` - Overall feature importance +- `ML.FEATURE_IMPORTANCE()` - Feature weights for tree models +- `ML.CONFUSION_MATRIX()` - Classification matrix +- `ML.ROC_CURVE()` - ROC curve data +- `ML.CENTROIDS()` - K-means cluster centers +- `ML.TRAINING_INFO()` - Training run details diff --git a/docs/ADK_SKILLS_DESIGN.md b/docs/ADK_SKILLS_DESIGN.md new file mode 100644 index 0000000000..d4bf2a4ab6 --- /dev/null +++ b/docs/ADK_SKILLS_DESIGN.md @@ -0,0 +1,1410 @@ +# ADK Skills Plugin: First-Class Dynamic Knowledge Injection Framework + +**Status:** Proposal +**Created:** December 2025 + +--- + +## Executive Summary + +This document proposes **ADK Skills** as a **first-class plugin system** for the Google Agent Development Kit (ADK). Skills represent a new primitive in the ADK plugin ecosystem, complementing existing primitives (Tools, Callbacks, Extensions) with a dedicated mechanism for **dynamic knowledge injection**. + +### What is a Skill? + +A **Skill** is a self-contained unit of domain knowledge that can be dynamically loaded into an agent's context at runtime. Unlike tools (which provide capabilities) or callbacks (which intercept execution), Skills provide **expertise**—the specialized knowledge an agent needs to perform domain-specific tasks correctly. + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ ADK Plugin Ecosystem │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ +│ │ TOOLS │ │ CALLBACKS │ │ EXTENSIONS │ │ +│ │ │ │ │ │ │ │ +│ │ Capabilities │ │ Interception │ │ Composition │ │ +│ │ "what agent │ │ "when/how │ │ "reusable │ │ +│ │ can DO" │ │ to act" │ │ bundles" │ │ +│ └───────────────┘ └───────────────┘ └───────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ SKILLS (NEW) │ │ +│ │ │ │ +│ │ Domain Knowledge │ Dynamic Loading │ Ephemeral Context │ │ +│ │ "what agent KNOWS" │ "load on-demand" │ "unload when done" │ │ +│ │ │ │ +│ │ Examples: │ │ +│ │ • BigQuery AI Functions syntax and best practices │ │ +│ │ • Kubernetes deployment patterns and troubleshooting │ │ +│ │ • Company coding standards and architecture guidelines │ │ +│ │ • Regulatory compliance requirements (HIPAA, SOC2, GDPR) │ │ +│ │ • API documentation for rapidly evolving services │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +### Key Value Propositions + +| Problem | Skill Solution | Impact | +|---------|----------------|--------| +| **Knowledge Staleness** | Skills can be updated independently of model training | Always current expertise | +| **Context Bloat** | Skills load only when needed, unload when done | 70-90% context savings | +| **First-Call Latency** | Callback-based injection before LLM call | Zero extra round-trips | +| **Expertise Scaling** | Add skills via markdown files, no code changes | O(1) effort per domain | + +--- + +## Table of Contents + +1. [Motivation and Problem Statement](#1-motivation-and-problem-statement) +2. [Skills as an ADK Plugin Primitive](#2-skills-as-an-adk-plugin-primitive) +3. [Skill Architecture and Design](#3-skill-architecture-and-design) +4. [Skill Plugin API Specification](#4-skill-plugin-api-specification) +5. [Implementation Details](#5-implementation-details) +6. [Skill Detection Strategies](#6-skill-detection-strategies) +7. [Domain Case Studies](#7-domain-case-studies) +8. [Performance and Cost Analysis](#8-performance-and-cost-analysis) +9. [Integration Patterns](#9-integration-patterns) +10. [Rollout and Migration](#10-rollout-and-migration) +11. [Future Roadmap](#11-future-roadmap) +12. [Appendix](#appendix) + +--- + +## 1. Motivation and Problem Statement + +### 1.1 The Knowledge Gap in LLM Agents + +LLM-based agents face a fundamental tension: + +``` + Model Training Real World + ┌─────────────┐ ┌─────────────┐ +Knowledge Cutoff ──────►│ Jan 2025 │ Today ─────►│ Dec 2025 │ + └─────────────┘ └─────────────┘ + │ │ + │ KNOWLEDGE GAP │ + │◄─────────────────────────────►│ + │ │ + • Old API versions • New APIs released + • Deprecated syntax • Breaking changes + • Missing best practices • New requirements +``` + +**Impact by Domain:** + +| Domain | Update Frequency | Knowledge Half-Life | Risk of Outdated Guidance | +|--------|------------------|---------------------|---------------------------| +| Cloud AI APIs (BigQuery, Vertex) | Monthly | 3-6 months | HIGH | +| Kubernetes | Quarterly | 6-9 months | MEDIUM-HIGH | +| Security/Compliance | Continuous | 1-3 months | CRITICAL | +| Internal Company Standards | Weekly | 1-2 months | HIGH | +| Programming Languages | Annual | 12-18 months | LOW | + +### 1.2 The Context Efficiency Problem + +Loading all domain knowledge statically is unsustainable: + +```python +# Anti-pattern: Static knowledge loading +agent = LlmAgent( + instruction=""" + You are an expert in: + - BigQuery ML (6,000 tokens) + - BigQuery AI Functions (4,000 tokens) + - BigQuery Remote Models (5,000 tokens) + - Kubernetes (8,000 tokens) + - Terraform (5,000 tokens) + - Python best practices (3,000 tokens) + - Security guidelines (4,000 tokens) + + Total: ~35,000 tokens ALWAYS loaded + Even for: "What's 2 + 2?" + """, +) +``` + +**Context Budget Analysis:** + +| Model | Context Limit | Static Load | Remaining for Conversation | +|-------|---------------|-------------|---------------------------| +| GPT-4 | 128K | 35K (27%) | 93K | +| Gemini 2.5 Pro | 1M | 35K (3.5%) | 965K | +| Claude 3.5 | 200K | 35K (17.5%) | 165K | + +While percentages seem manageable, the real costs are: +1. **Latency**: More tokens = slower time-to-first-token +2. **Cost**: ~$3.50 per 1M input tokens (Gemini) × scale +3. **Attention Dilution**: More context = less focus on relevant information + +### 1.3 Why Existing Solutions Fall Short + +| Approach | Mechanism | Limitation | +|----------|-----------|------------| +| **RAG** | Semantic retrieval | Latency (100-500ms), retrieval quality varies | +| **Fine-tuning** | Model weights | Expensive, slow iteration, can't "unlearn" | +| **Tool-based Loading** | Agent calls `load_skill()` | Extra LLM round-trip (2-5s) | +| **Static System Prompt** | All knowledge upfront | Context waste, staleness | +| **Instruction Provider** | Dynamic prompt building | Timing issue: runs before user input analysis | + +**The Timing Problem with Instruction Providers:** + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ ADK Request Processing Pipeline │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. User Input Received │ +│ │ │ +│ ▼ │ +│ 2. _preprocess_async() ◄──── instruction_provider() called HERE │ +│ │ (Skills NOT detected yet!) │ +│ ▼ │ +│ 3. before_model_callback() ◄──── We CAN detect skills HERE │ +│ │ AND inject via append_instructions() │ +│ ▼ │ +│ 4. LLM.generate() ◄──── Skills available in FIRST call! │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 2. Skills as an ADK Plugin Primitive + +### 2.1 Plugin Primitive Comparison + +ADK provides several extension points. Skills fill a unique gap: + +| Primitive | Purpose | When Used | State | +|-----------|---------|-----------|-------| +| **Tool** | Execute actions | Agent invokes explicitly | Stateless | +| **Callback** | Intercept/modify flow | Automatic at lifecycle points | Can modify request/response | +| **Extension** | Bundle related functionality | Package tools + callbacks | Configured at init | +| **Skill** (NEW) | Provide domain knowledge | Auto-detected or on-demand | Ephemeral per-turn | + +### 2.2 Skill Characteristics + +A Skill in ADK has these defining properties: + +```yaml +# Skill Definition Properties +1. Self-Describing: + - Metadata (name, description, version) + - Keywords for auto-detection + - Dependencies on other skills (optional) + +2. Markdown-Based: + - Human-readable and editable + - Version controlled (Git) + - No code changes to add/update + +3. Ephemeral: + - Loaded into context on-demand + - Cleared after each agent turn + - No permanent context pollution + +4. Injection-Based: + - Content injected into system instruction + - Available in FIRST LLM call + - No tool-call overhead +``` + +### 2.3 Skill vs Tool: When to Use Each + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Decision Matrix: Skill vs Tool │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Use a SKILL when you need to: Use a TOOL when you need to: │ +│ ┌─────────────────────────────┐ ┌─────────────────────────────┐│ +│ │ • Provide domain expertise │ │ • Execute an action ││ +│ │ • Share syntax/patterns │ │ • Query external systems ││ +│ │ • Explain best practices │ │ • Modify state ││ +│ │ • Document API changes │ │ • Compute results ││ +│ │ • Guide decision-making │ │ • Retrieve dynamic data ││ +│ └─────────────────────────────┘ └─────────────────────────────┘│ +│ │ +│ SKILL: "How to write a BigQuery ML query" │ +│ TOOL: "Execute this query against BigQuery" │ +│ │ +│ SKILL: "Kubernetes pod troubleshooting steps" │ +│ TOOL: "kubectl get pods -n namespace" │ +│ │ +│ SKILL: "Company API versioning standards" │ +│ TOOL: "Create a new API endpoint" │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 3. Skill Architecture and Design + +### 3.1 Core Components + +``` +google/adk/skills/ +├── __init__.py # Public API exports +├── skill.py # Skill base class and data types +├── skill_registry.py # Discovery, loading, caching +├── skill_callbacks.py # Callback-based auto-injection +├── skill_detector.py # Detection strategies (keyword, LLM, hybrid) +├── skill_loader.py # File parsing (markdown + frontmatter) +└── builtin/ # ADK-provided skills + ├── bigquery/ + │ ├── bqml.md + │ ├── ai_functions.md + │ └── remote_models.md + ├── kubernetes/ + │ ├── deployments.md + │ └── troubleshooting.md + └── general/ + └── coding_standards.md +``` + +### 3.2 Skill Data Model + +```python +@dataclass +class SkillMetadata: + """Metadata extracted from SKILL.md frontmatter.""" + name: str # Unique identifier + description: str # Human-readable description + version: str = "1.0.0" # Semantic version + keywords: list[str] = field(default_factory=list) # Detection triggers + requires: list[str] = field(default_factory=list) # Skill dependencies + modality: str = "text" # text, multi-modal, code + domain: str = "general" # Categorization + +@dataclass +class Skill: + """Complete skill with metadata and content.""" + metadata: SkillMetadata + content: str # Markdown content (body) + source_path: Path # File location + token_estimate: int # Approximate token count + + def to_injection_format(self) -> str: + """Format skill for system instruction injection.""" + return f""" +## Active Skill: {self.metadata.name} +**Description:** {self.metadata.description} +**Version:** {self.metadata.version} + +--- + +{self.content} +""" +``` + +### 3.3 Skill File Format (SKILL.md) + +```markdown +--- +name: kubernetes_troubleshooting +description: Kubernetes pod and deployment troubleshooting patterns +version: 1.2.0 +keywords: + - pod + - crashloopbackoff + - oomkilled + - imagepullbackoff + - kubectl + - kubernetes + - k8s + - deployment + - not ready +requires: [] +domain: infrastructure +modality: text +--- + +# Kubernetes Troubleshooting Skill + +This skill provides systematic troubleshooting approaches for common Kubernetes issues. + +## Pod Status Analysis + +### CrashLoopBackOff + +**Symptoms:** Pod repeatedly crashes and restarts +**Diagnostic Commands:** +```bash +# Check pod events +kubectl describe pod -n + +# Check logs from current crash +kubectl logs -n + +# Check logs from previous crash +kubectl logs -n --previous +``` + +**Common Causes:** +1. Application error during startup +2. Missing configuration (ConfigMap, Secret) +3. Resource limits too low +4. Liveness probe failing + +[... comprehensive troubleshooting guide ...] +``` + +### 3.4 Runtime Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Skill Runtime Architecture │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Agent Initialization │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ SkillRegistry.discover() │ │ +│ │ └─► Scan skills directories │ │ +│ │ └─► Parse SKILL.md frontmatter (metadata only - Level 1) │ │ +│ │ └─► Build keyword → skill index │ │ +│ │ └─► Compile regex patterns │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ Request Processing (per user message) │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ before_model_callback() │ │ +│ │ │ │ │ +│ │ ├─► 1. Extract user message text │ │ +│ │ │ │ │ +│ │ ├─► 2. Detect skills (keyword/LLM/hybrid) │ │ +│ │ │ └─► Match patterns against text │ │ +│ │ │ └─► Return list of skill names │ │ +│ │ │ │ │ +│ │ ├─► 3. Load skill content (Level 2 - on demand) │ │ +│ │ │ └─► Read full SKILL.md content │ │ +│ │ │ └─► Cache for session │ │ +│ │ │ │ │ +│ │ ├─► 4. Inject into LLM request │ │ +│ │ │ └─► llm_request.append_instructions([skill_content]) │ │ +│ │ │ │ │ +│ │ └─► 5. Store active skills in state │ │ +│ │ └─► callback_context.state["active_skills"] = [...] │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ Turn Completion │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ after_agent_callback() │ │ +│ │ └─► Clear active skills from state │ │ +│ │ └─► Context freed for next turn │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 4. Skill Plugin API Specification + +### 4.1 Core Classes + +#### SkillRegistry + +```python +class SkillRegistry: + """Central registry for skill discovery, loading, and management. + + The registry implements progressive disclosure: + - Level 1: Metadata loaded at startup (fast, low memory) + - Level 2: Full content loaded on-demand (lazy loading) + + Thread-safe for concurrent agent usage. + """ + + def __init__( + self, + skills_dirs: list[str | Path] | None = None, + builtin_skills: bool = True, + cache_content: bool = True, + ): + """Initialize the skill registry. + + Args: + skills_dirs: Directories to scan for SKILL.md files. + Defaults to ./skills relative to caller. + builtin_skills: Include ADK builtin skills (bigquery, k8s, etc.) + cache_content: Cache loaded skill content in memory + """ + + def discover(self) -> dict[str, SkillMetadata]: + """Discover all skills in configured directories. + + Returns: + Dict mapping skill name to metadata + """ + + def get_skill(self, name: str) -> Skill | None: + """Load a skill by name (Level 2 - full content). + + Args: + name: Skill identifier + + Returns: + Complete Skill object or None if not found + """ + + def get_skills(self, names: list[str]) -> list[Skill]: + """Load multiple skills by name. + + Args: + names: List of skill identifiers + + Returns: + List of Skill objects (excludes not found) + """ + + def list_skills(self) -> list[SkillMetadata]: + """List all discovered skill metadata.""" + + def get_skill_summary(self) -> str: + """Generate summary of available skills for system instruction.""" + + def build_keyword_index(self) -> dict[str, list[str]]: + """Build keyword → skill name mapping for detection.""" + + def reload(self, name: str | None = None) -> None: + """Hot reload skill(s) from disk. + + Args: + name: Specific skill to reload, or None for all + """ +``` + +#### SkillCallbacks + +```python +class SkillCallbacks: + """Callback handlers for automatic skill lifecycle management. + + Integrates with LlmAgent callbacks to: + 1. Detect relevant skills from user input + 2. Inject skill content into system instruction + 3. Clean up skills after agent turn completes + + Detection modes: + - "keyword": Fast regex matching (recommended for domain-specific terms) + - "llm": Semantic classification using small model + - "hybrid": LLM with keyword fallback + """ + + def __init__( + self, + registry: SkillRegistry, + detection_mode: Literal["keyword", "llm", "hybrid"] = "keyword", + auto_deactivate: bool = True, + max_skills_per_turn: int = 3, + classifier_model: str = "gemini-1.5-flash", + ): + """Initialize skill callbacks. + + Args: + registry: SkillRegistry instance + detection_mode: How to detect skills from user input + auto_deactivate: Clear skills after each turn + max_skills_per_turn: Limit concurrent skill loading + classifier_model: Model for LLM-based detection + """ + + def before_model_callback( + self, + callback_context: CallbackContext, + llm_request: LlmRequest, + ) -> LlmResponse | None: + """Detect and inject skills before LLM processing. + + This callback: + 1. Extracts user message from llm_request.contents + 2. Detects relevant skills via configured strategy + 3. Loads skill content from registry + 4. Injects via llm_request.append_instructions() + 5. Stores active skills in callback_context.state + + Returns: + None (continue processing) or LlmResponse (short-circuit) + """ + + def after_agent_callback( + self, + callback_context: CallbackContext, + ) -> types.Content | None: + """Clean up skills after agent completes turn. + + Returns: + None (no content to add) + """ + + # Manual control methods + def activate_skills( + self, + skill_names: list[str], + callback_context: CallbackContext, + ) -> list[str]: + """Manually activate specific skills.""" + + def deactivate_skills( + self, + skill_names: list[str] | None, + callback_context: CallbackContext, + ) -> list[str]: + """Manually deactivate skills (None = all).""" + + def get_active_skills( + self, + callback_context: CallbackContext, + ) -> list[str]: + """Get currently active skill names.""" +``` + +### 4.2 Integration with LlmAgent + +```python +from google.adk.agents import LlmAgent +from google.adk.skills import SkillRegistry, SkillCallbacks + +# Method 1: Explicit callback registration +registry = SkillRegistry( + skills_dirs=["./skills", "./custom_skills"], + builtin_skills=True, +) +callbacks = SkillCallbacks( + registry=registry, + detection_mode="keyword", + auto_deactivate=True, +) + +agent = LlmAgent( + model="gemini-2.5-pro", + name="expert_agent", + instruction="You are a helpful assistant.", + tools=[...], + before_model_callback=callbacks.before_model_callback, + after_agent_callback=callbacks.after_agent_callback, +) + +# Method 2: Using SkillExtension (convenience wrapper) +from google.adk.skills import SkillExtension + +agent = LlmAgent( + model="gemini-2.5-pro", + name="expert_agent", + instruction="You are a helpful assistant.", + tools=[...], + extensions=[ + SkillExtension( + skills_dirs=["./skills"], + detection_mode="keyword", + ), + ], +) + +# Method 3: Domain-specific toolset with bundled skills +from google.adk.tools.bigquery import BigQueryToolset + +toolset = BigQueryToolset( + credentials_config=config, + enable_skills=True, # Auto-loads BigQuery skills +) + +agent = LlmAgent( + model="gemini-2.5-pro", + name="bq_agent", + tools=toolset.get_tools(), + **toolset.get_skill_callbacks(), # Injects before/after callbacks +) +``` + +### 4.3 State Management API + +```python +# State keys (session-scoped) +ACTIVE_SKILLS_KEY = "adk:skills:active" +SKILL_HISTORY_KEY = "adk:skills:history" + +# Accessing skill state +active = callback_context.state.get(ACTIVE_SKILLS_KEY, []) +history = callback_context.state.get(SKILL_HISTORY_KEY, []) + +# Skill state structure +{ + "adk:skills:active": ["bqml", "bq_remote_model"], + "adk:skills:history": [ + {"turn": 1, "skills": ["bqml"], "detected_from": "train model"}, + {"turn": 2, "skills": ["bq_remote_model"], "detected_from": "gemini"}, + ], +} +``` + +--- + +## 5. Implementation Details + +### 5.1 The Injection Mechanism + +The critical implementation detail is HOW skills are injected into the LLM request: + +```python +def _inject_skills_into_request( + self, + llm_request: LlmRequest, + skills: list[Skill], +) -> None: + """Inject skill content directly into the LLM request. + + Uses llm_request.append_instructions() which: + 1. Concatenates to config.system_instruction using "\\n\\n" + 2. Handles both string and Content types + 3. Works BEFORE the LLM call (not deferred) + """ + if not skills: + return + + # Build formatted skill content + sections = [] + for skill in skills: + sections.append(skill.to_injection_format()) + + skill_block = f""" +# Currently Active Skills + +The following domain expertise has been loaded for this task. +Follow the guidance in these skills carefully. + +{"".join(sections)} + +--- +""" + + # Inject into request (modifies config.system_instruction) + llm_request.append_instructions([skill_block]) + + logger.info(f"Injected skills: {[s.metadata.name for s in skills]}") +``` + +### 5.2 Keyword Detection Implementation + +```python +class KeywordSkillDetector: + """Fast keyword-based skill detection using compiled regex.""" + + def __init__(self, registry: SkillRegistry): + self._registry = registry + self._patterns: dict[str, list[re.Pattern]] = {} + self._build_patterns() + + def _build_patterns(self) -> None: + """Compile regex patterns from skill keywords.""" + for skill_name, metadata in self._registry.list_skills(): + patterns = [] + for keyword in metadata.keywords: + # Escape special chars + escaped = re.escape(keyword.lower()) + # Add word boundaries for non-dotted keywords + if "." not in keyword: + pattern = rf"\b{escaped}\b" + else: + pattern = escaped + patterns.append(re.compile(pattern, re.IGNORECASE)) + self._patterns[skill_name] = patterns + + def detect(self, text: str) -> list[str]: + """Detect skills from text using keyword matching. + + Args: + text: User message or query + + Returns: + List of detected skill names + """ + detected = [] + text_lower = text.lower() + + for skill_name, patterns in self._patterns.items(): + for pattern in patterns: + if pattern.search(text_lower): + detected.append(skill_name) + break # One match per skill is sufficient + + return detected +``` + +### 5.3 Multi-Turn Handling + +```python +def before_model_callback( + self, + callback_context: CallbackContext, + llm_request: LlmRequest, +) -> LlmResponse | None: + """Handle skill injection across multi-turn tool use.""" + + # Check if we already have active skills (continuation) + active_skills = callback_context.state.get(ACTIVE_SKILLS_KEY, []) + + if not active_skills: + # NEW turn - detect from latest user message + user_text = self._extract_user_message(llm_request) + detected = self._detector.detect(user_text) + + # Apply limits + if len(detected) > self._max_skills: + logger.warning(f"Limiting skills from {len(detected)} to {self._max_skills}") + detected = detected[:self._max_skills] + + # Store for this turn + callback_context.state[ACTIVE_SKILLS_KEY] = detected + active_skills = detected + + # Record in history + history = callback_context.state.get(SKILL_HISTORY_KEY, []) + history.append({ + "turn": len(history) + 1, + "skills": detected, + "detected_from": user_text[:100], + }) + callback_context.state[SKILL_HISTORY_KEY] = history + + # Load and inject skills + if active_skills: + skills = self._registry.get_skills(active_skills) + self._inject_skills_into_request(llm_request, skills) + + return None # Continue processing +``` + +--- + +## 6. Skill Detection Strategies + +### 6.1 Strategy Comparison + +| Strategy | Latency | Accuracy | Best For | +|----------|---------|----------|----------| +| **Keyword** | <1ms | 95%+ for domain terms | Technical domains with unique vocabulary | +| **LLM** | 500-1500ms | 98%+ | Natural language, paraphrased queries | +| **Hybrid** | 500-1500ms | 99%+ | Mixed workloads | + +### 6.2 Keyword Strategy (Recommended Default) + +```python +# Keyword matching excels when domains have unique terminology + +# BigQuery Skills +"AI.CLASSIFY" → bq_ai_operator (unambiguous) +"CREATE MODEL" → bqml (unambiguous) +"gemini" → bq_remote_model (context: BigQuery agent) + +# Kubernetes Skills +"CrashLoopBackOff" → k8s_troubleshooting (unambiguous) +"kubectl" → k8s_* (namespace indicator) +"OOMKilled" → k8s_troubleshooting (unambiguous) + +# Security Skills +"HIPAA" → compliance_hipaa (unambiguous) +"SOC2" → compliance_soc2 (unambiguous) +``` + +### 6.3 LLM Strategy (Semantic Understanding) + +```python +class LLMSkillDetector: + """LLM-based skill detection for semantic understanding.""" + + CLASSIFICATION_PROMPT = """ + Given the user query and available skills, identify which skills + would help the agent respond accurately. + + Available Skills: + {skill_summaries} + + User Query: {query} + + Return a JSON array of skill names that should be activated. + Only include skills directly relevant to the query. + Return [] if no skills are needed. + """ + + async def detect(self, text: str) -> list[str]: + """Detect skills using LLM classification.""" + prompt = self.CLASSIFICATION_PROMPT.format( + skill_summaries=self._registry.get_skill_summary(), + query=text, + ) + + response = await self._classifier.generate(prompt) + return json.loads(response) +``` + +### 6.4 Hybrid Strategy (Fallback Chain) + +```python +class HybridSkillDetector: + """Hybrid detection: LLM primary, keyword fallback.""" + + async def detect(self, text: str) -> list[str]: + # Try LLM first + try: + detected = await self._llm_detector.detect(text) + if detected: + return detected + except Exception as e: + logger.warning(f"LLM detection failed: {e}") + + # Fallback to keywords + return self._keyword_detector.detect(text) +``` + +--- + +## 7. Domain Case Studies + +### 7.1 BigQuery AI (Reference Implementation) + +**Domain Characteristics:** +- Rapidly evolving (new Gemini versions, AI functions) +- Highly specific syntax (SQL extensions) +- Strong keyword signals ("AI.CLASSIFY", "CREATE REMOTE MODEL") + +**Skill Structure:** +``` +bigquery/ +├── bqml.md # ML model training (6,000 tokens) +├── ai_functions.md # AI.CLASSIFY, AI.IF, AI.SCORE (4,000 tokens) +└── remote_models.md # Remote model creation (5,000 tokens) +``` + +**Detection Keywords:** +| Skill | Keywords | +|-------|----------| +| bqml | train, model, predict, LINEAR_REG, KMEANS, ML.EVALUATE | +| ai_functions | AI.CLASSIFY, AI.IF, AI.SCORE, classify, sentiment | +| remote_models | gemini, remote model, AI.GENERATE_TEXT, embeddings | + +**Real-World Impact:** +``` +User: "Classify 5 BBC news articles by topic using AI functions" + +Without Skills: +- Agent might use deprecated ML.GENERATE_TEXT +- Miss connection_id requirement (added Q2 2025) +- Use wrong parameter format + +With Skills: +- Agent uses AI.CLASSIFY (current API) +- Includes proper connection_id syntax +- Follows location matching rules +``` + +### 7.2 Kubernetes Operations + +**Domain Characteristics:** +- Version-specific behaviors (1.28 vs 1.29) +- Complex troubleshooting patterns +- Strong error message signals + +**Skill Structure:** +``` +kubernetes/ +├── deployments.md # Deployment patterns (4,000 tokens) +├── troubleshooting.md # Error diagnosis (6,000 tokens) +├── networking.md # Service/Ingress (3,500 tokens) +└── security.md # RBAC, NetworkPolicy (3,000 tokens) +``` + +**Detection Keywords:** +| Skill | Keywords | +|-------|----------| +| troubleshooting | CrashLoopBackOff, OOMKilled, ImagePullBackOff, not ready | +| deployments | deployment, rollout, strategy, replica | +| networking | service, ingress, loadbalancer, nodeport | +| security | rbac, networkpolicy, serviceaccount, podsecuritypolicy | + +### 7.3 Enterprise Compliance + +**Domain Characteristics:** +- Regulatory requirements (must be current) +- Organization-specific policies +- Critical accuracy requirements + +**Skill Structure:** +``` +compliance/ +├── hipaa.md # Healthcare data requirements +├── soc2.md # Security controls +├── gdpr.md # EU data privacy +└── internal/ + └── data_handling.md # Company-specific policies +``` + +**Use Case:** +``` +User: "I need to store patient health records in our application" + +Detected Skills: [hipaa, internal/data_handling] + +Injected Knowledge: +- PHI encryption requirements +- Access logging mandates +- Data retention policies +- Company-specific approval workflows +``` + +### 7.4 Internal Development Standards + +**Domain Characteristics:** +- Company-specific (not in public training data) +- Frequently updated +- Critical for consistency + +**Skill Structure:** +``` +company_standards/ +├── api_design.md # REST API conventions +├── error_handling.md # Error response formats +├── logging.md # Structured logging standards +├── testing.md # Test coverage requirements +└── security.md # Security review checklist +``` + +**Integration Pattern:** +```python +# Company-wide agent with internal skills +agent = LlmAgent( + model="gemini-2.5-pro", + name="dev_assistant", + instruction="Help engineers follow company standards.", + extensions=[ + SkillExtension( + skills_dirs=[ + "/shared/skills/company_standards", + "/team/skills/backend", + ], + detection_mode="keyword", + ), + ], +) +``` + +--- + +## 8. Performance and Cost Analysis + +### 8.1 Latency Impact + +| Scenario | Without Skills | With Skills | Delta | +|----------|---------------|-------------|-------| +| Simple query (no skill needed) | 1.5s | 1.5s | +0ms | +| Domain query (1 skill) | 1.5s | 1.6s | +100ms | +| Complex query (3 skills) | 1.5s | 1.8s | +300ms | +| Tool-based loading (comparison) | 1.5s | 4.5s | +3000ms | + +**Key Insight:** Skill injection adds ~50-100ms per skill (token processing), while tool-based loading adds 2-3s per skill (extra LLM round-trip). + +### 8.2 Token Efficiency + +**Comparison: Always-On vs Dynamic Skills** + +| Approach | Tokens/Query (avg) | Annual Tokens (1M queries) | Annual Cost | +|----------|-------------------|---------------------------|-------------| +| All skills always | 35,000 | 35B | $350,000 | +| Dynamic (50% need skills) | 8,500 | 8.5B | $85,000 | +| **Savings** | **76%** | **26.5B** | **$265,000** | + +### 8.3 Detection Accuracy + +**Keyword Detection (BigQuery Domain):** + +| Metric | Value | Notes | +|--------|-------|-------| +| Precision | 99.2% | Very few false positives | +| Recall | 96.8% | Comprehensive keyword lists | +| F1 Score | 98.0% | Excellent overall accuracy | +| Latency | 0.3ms | Compiled regex | + +**Failure Modes:** +- False Positive: "I love training for marathons" → bqml (rare) +- False Negative: "Help me build a predictive system" → no match (add "predictive" keyword) + +--- + +## 9. Integration Patterns + +### 9.1 Pattern: Toolset with Bundled Skills + +```python +class BigQueryToolset: + """BigQuery tools with integrated skill support.""" + + def __init__( + self, + credentials_config: CredentialsConfig, + enable_skills: bool = True, + skill_detection_mode: str = "keyword", + ): + self._tools = [ + execute_query, + list_tables, + get_schema, + list_connections, + create_connection, + ] + + if enable_skills: + self._skill_registry = SkillRegistry( + skills_dirs=[Path(__file__).parent / "skills"], + builtin_skills=False, + ) + self._skill_callbacks = SkillCallbacks( + registry=self._skill_registry, + detection_mode=skill_detection_mode, + ) + + def get_tools(self) -> list[Tool]: + return self._tools + + def get_skill_callbacks(self) -> dict: + """Return callbacks dict for LlmAgent kwargs.""" + return { + "before_model_callback": self._skill_callbacks.before_model_callback, + "after_agent_callback": self._skill_callbacks.after_agent_callback, + } +``` + +### 9.2 Pattern: Multi-Domain Agent + +```python +# Agent with skills from multiple domains +agent = LlmAgent( + model="gemini-2.5-pro", + name="platform_agent", + instruction="Help with cloud infrastructure tasks.", + tools=[...], + extensions=[ + SkillExtension( + skills_dirs=[ + "./skills/bigquery", + "./skills/kubernetes", + "./skills/terraform", + ], + detection_mode="keyword", + max_skills_per_turn=3, + ), + ], +) +``` + +### 9.3 Pattern: Skill Composition + +```python +# Skills with dependencies +# terraform/modules.md +--- +name: terraform_modules +requires: + - terraform_basics # Load basics first +--- + +# Automatically loads both when terraform_modules is detected +``` + +### 9.4 Pattern: Conditional Skills + +```python +class ConditionalSkillCallbacks(SkillCallbacks): + """Skills that activate based on runtime conditions.""" + + def before_model_callback(self, ctx, req): + # Add compliance skills based on user context + if ctx.state.get("user_department") == "healthcare": + self._force_activate(["hipaa"], ctx) + + # Continue with normal detection + return super().before_model_callback(ctx, req) +``` + +--- + +## 10. Rollout and Migration + +### 10.1 Phase 1: Core Framework + +**Deliverables:** +- `google.adk.skills` module in ADK core +- SkillRegistry, SkillCallbacks, SkillExtension +- Documentation and examples + +**API Surface:** +```python +from google.adk.skills import ( + Skill, + SkillMetadata, + SkillRegistry, + SkillCallbacks, + SkillExtension, + KeywordSkillDetector, +) +``` + +### 10.2 Phase 2: Builtin Skills + +**Deliverables:** +- BigQuery skills (BQML, AI Functions, Remote Models) +- Kubernetes skills (Deployments, Troubleshooting) +- General skills (Python, Security) + +**Integration:** +```python +from google.adk.skills.builtin import ( + BIGQUERY_SKILLS, + KUBERNETES_SKILLS, +) + +registry = SkillRegistry( + builtin_skills=True, # Includes all builtin + # OR + builtin_skills=BIGQUERY_SKILLS, # Specific subset +) +``` + +### 10.3 Phase 3: Toolset Integration + +**Deliverables:** +- BigQueryToolset with enable_skills parameter +- KubernetesToolset with enable_skills parameter +- Auto-configuration patterns + +### 10.4 Phase 4: Skill Ecosystem + +**Deliverables:** +- Skill marketplace/registry +- Versioned skill packages +- Community contribution guidelines +- Skill analytics dashboard + +--- + +## 11. Future Roadmap + +### 11.1 Multi-Modal Skills + +```yaml +--- +name: architecture_diagrams +modality: multi-modal +--- + +# Architecture Patterns + +![Microservices Pattern](./images/microservices.png) + +Use this pattern when: +- Services need independent scaling +- Teams need deployment autonomy +``` + +### 11.2 Executable Skills + +```yaml +--- +name: code_generator +modality: executable +entrypoint: generate_code +--- + +```python +def generate_code(context: SkillContext) -> str: + """Generate code based on context.""" + template = load_template(context.language) + return template.render(context.params) +``` +``` + +### 11.3 Federated Skills + +```python +# Load skills from remote registry +registry = SkillRegistry( + remote_registries=[ + "https://skills.google.com/bigquery", + "https://internal.company.com/skills", + ], + cache_ttl=3600, # Refresh hourly +) +``` + +### 11.4 Skill Learning + +```python +# Track skill effectiveness +analytics = SkillAnalytics(registry) + +# After agent interaction +analytics.record_outcome( + skill_name="bqml", + query="train a regression model", + outcome="success", + user_satisfaction=5, +) + +# Optimize keyword detection +analytics.suggest_keywords("bqml") +# Returns: ["predictive model", "forecast"] based on user patterns +``` + +--- + +## Appendix + +### A.1 Complete SKILL.md Template + +```markdown +--- +# Required fields +name: skill_name # Unique identifier (alphanumeric + underscore) +description: Brief description # One-line summary for listings + +# Optional fields +version: 1.0.0 # Semantic version +keywords: # Detection triggers + - keyword1 + - multi word keyword + - function.name +requires: [] # Skill dependencies +domain: general # Category (bigquery, kubernetes, etc.) +modality: text # text, multi-modal, executable +author: team@company.com # Maintainer contact +updated: 2025-12-01 # Last update date +--- + +# Skill Title + +Brief introduction explaining what this skill provides. + +## Prerequisites + +List any setup requirements. + +## Core Concepts + +### Concept 1 + +Explanation with examples: + +```language +// Code example +``` + +### Concept 2 + +More content... + +## Examples + +### Example 1: Common Use Case + +```language +// Complete working example +``` + +### Example 2: Advanced Use Case + +```language +// Advanced example +``` + +## Best Practices + +1. Best practice 1 +2. Best practice 2 + +## Troubleshooting + +**Error: "common error message"** +- Cause: Why this happens +- Solution: How to fix + +## References + +- [Official Documentation](https://...) +- [Related Guide](https://...) +``` + +### A.2 Debugging Skills + +```python +import logging + +# Enable skill debugging +logging.getLogger("google.adk.skills").setLevel(logging.DEBUG) + +# Output: +# [SkillRegistry] Discovered 5 skills in ./skills +# [SkillCallbacks] Detecting from: "Train a regression model" +# [KeywordDetector] Matched "train" → bqml +# [KeywordDetector] Matched "regression" → bqml +# [SkillCallbacks] Activating skills: ['bqml'] +# [SkillCallbacks] Loaded bqml (6,234 tokens) +# [SkillCallbacks] Injected into system instruction +``` + +### A.3 Testing Skills + +```python +import pytest +from google.adk.skills import SkillRegistry, KeywordSkillDetector + +class TestSkillDetection: + @pytest.fixture + def registry(self): + return SkillRegistry(skills_dirs=["./test_skills"]) + + @pytest.fixture + def detector(self, registry): + return KeywordSkillDetector(registry) + + def test_detects_bqml_from_train(self, detector): + detected = detector.detect("Train a model to predict sales") + assert "bqml" in detected + + def test_no_detection_for_unrelated(self, detector): + detected = detector.detect("What's the weather today?") + assert len(detected) == 0 + + def test_multiple_skills_detected(self, detector): + detected = detector.detect( + "Create a Gemini model to classify news articles" + ) + assert "bq_remote_model" in detected + assert "bq_ai_operator" in detected +``` + +### A.4 Skill Metrics + +```python +@dataclass +class SkillMetrics: + """Metrics collected per skill.""" + name: str + activation_count: int + avg_turn_duration_ms: float + avg_tokens_used: int + success_rate: float # Based on user feedback + common_triggers: list[str] # Most frequent detection keywords +``` + +--- + +## References + +1. [Anthropic: Equipping Agents with Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) +2. [Google ADK Documentation](https://cloud.google.com/docs/adk) +3. [LlmAgent Callbacks Reference](https://cloud.google.com/docs/adk/callbacks) +4. [BigQuery ML Documentation](https://cloud.google.com/bigquery/docs/bqml-introduction) +5. [BigQuery AI Functions](https://cloud.google.com/bigquery/docs/ai-functions) diff --git a/src/google/adk/tools/bigquery/__init__.py b/src/google/adk/tools/bigquery/__init__.py index 9e6b1166b0..1b80ed2b91 100644 --- a/src/google/adk/tools/bigquery/__init__.py +++ b/src/google/adk/tools/bigquery/__init__.py @@ -25,12 +25,20 @@ etc. 4. We want to provide extra access guardrails in those tools. For example, execute_sql can't arbitrarily mutate existing data. + +Skills are also provided for programmatic tool calling (PTC): +- BQMLSkill: Machine learning with BigQuery ML +- BQAIOperatorSkill: AI functions (AI.GENERATE, AI.CLASSIFY, etc.) """ from .bigquery_credentials import BigQueryCredentialsConfig from .bigquery_toolset import BigQueryToolset +from .skills import BQAIOperatorSkill +from .skills import BQMLSkill __all__ = [ "BigQueryToolset", "BigQueryCredentialsConfig", + "BQMLSkill", + "BQAIOperatorSkill", ] diff --git a/src/google/adk/tools/bigquery/bigquery_toolset.py b/src/google/adk/tools/bigquery/bigquery_toolset.py index f38bf95f62..2640133a1a 100644 --- a/src/google/adk/tools/bigquery/bigquery_toolset.py +++ b/src/google/adk/tools/bigquery/bigquery_toolset.py @@ -81,6 +81,8 @@ async def get_tools( metadata_tool.list_dataset_ids, metadata_tool.list_table_ids, metadata_tool.get_job_info, + metadata_tool.list_connections, + metadata_tool.create_connection, query_tool.get_execute_sql(self._tool_settings), query_tool.forecast, query_tool.analyze_contribution, diff --git a/src/google/adk/tools/bigquery/metadata_tool.py b/src/google/adk/tools/bigquery/metadata_tool.py index af50f54f3f..575d92bc55 100644 --- a/src/google/adk/tools/bigquery/metadata_tool.py +++ b/src/google/adk/tools/bigquery/metadata_tool.py @@ -299,6 +299,262 @@ def get_table_info( } +def list_connections( + project_id: str, + location: str, + credentials: Credentials, + settings: BigQueryToolConfig, +) -> list[dict]: + """List BigQuery connections in a Google Cloud project and location. + + BigQuery connections are used to connect to external data sources like + Vertex AI for AI functions (AI.CLASSIFY, AI.IF, AI.SCORE) and remote + models (AI.GENERATE_TEXT, AI.GENERATE_EMBEDDING). + + Args: + project_id (str): The Google Cloud project id. + location (str): The location/region (e.g., 'us', 'eu', 'us-central1'). + credentials (Credentials): The credentials to use for the request. + settings (BigQueryToolConfig): The BigQuery tool settings. + + Returns: + list[dict]: List of connections with their properties. + + Examples: + >>> list_connections("my-project", "us") + [ + { + "name": "projects/my-project/locations/us/connections/my_ai_connection", + "connection_id": "my_ai_connection", + "location": "us", + "connection_type": "CLOUD_RESOURCE", + "friendly_name": "My AI Connection", + "description": "Connection for Vertex AI" + } + ] + """ + try: + from google.cloud import bigquery_connection_v1 + + connection_client = bigquery_connection_v1.ConnectionServiceClient( + credentials=credentials + ) + parent = f"projects/{project_id}/locations/{location}" + + connections = [] + for conn in connection_client.list_connections(parent=parent): + # Extract connection_id from the full name + # Format: projects/{project}/locations/{location}/connections/{connection_id} + parts = conn.name.split("/") + connection_id = parts[-1] if parts else conn.name + + conn_info = { + "name": conn.name, + "connection_id": connection_id, + "location": location, + "friendly_name": conn.friendly_name or "", + "description": conn.description or "", + } + + # Add connection type based on which field is set + if conn.cloud_resource: + conn_info["connection_type"] = "CLOUD_RESOURCE" + if conn.cloud_resource.service_account_id: + conn_info["service_account"] = conn.cloud_resource.service_account_id + elif conn.cloud_sql: + conn_info["connection_type"] = "CLOUD_SQL" + elif conn.spark: + conn_info["connection_type"] = "SPARK" + else: + conn_info["connection_type"] = "UNKNOWN" + + connections.append(conn_info) + + return connections + except Exception as ex: + return { + "status": "ERROR", + "error_details": str(ex), + } + + +def _grant_vertex_ai_role( + project_id: str, + service_account: str, + credentials: Credentials, +) -> dict: + """Grant the Vertex AI User role to a service account. + + Args: + project_id (str): The Google Cloud project id. + service_account (str): The service account email to grant the role to. + credentials (Credentials): The credentials to use for the request. + + Returns: + dict: Status of the IAM operation. + """ + try: + from google.cloud import resourcemanager_v3 + from google.iam.v1 import iam_policy_pb2, policy_pb2 + + # Create the Resource Manager client + client = resourcemanager_v3.ProjectsClient(credentials=credentials) + + # Get the current IAM policy + resource = f"projects/{project_id}" + request = iam_policy_pb2.GetIamPolicyRequest(resource=resource) + policy = client.get_iam_policy(request=request) + + # Check if the binding already exists + role = "roles/aiplatform.user" + member = f"serviceAccount:{service_account}" + + binding_exists = False + for binding in policy.bindings: + if binding.role == role: + if member in binding.members: + binding_exists = True + break + else: + # Add member to existing role binding + binding.members.append(member) + binding_exists = True + break + + # If role binding doesn't exist, create a new one + if not binding_exists: + new_binding = policy_pb2.Binding(role=role, members=[member]) + policy.bindings.append(new_binding) + + # Set the updated IAM policy + set_request = iam_policy_pb2.SetIamPolicyRequest( + resource=resource, policy=policy + ) + client.set_iam_policy(request=set_request) + + return { + "status": "SUCCESS", + "message": f"Granted {role} to {service_account}", + } + except Exception as ex: + return { + "status": "ERROR", + "error_details": f"Failed to grant IAM role: {str(ex)}", + } + + +def create_connection( + project_id: str, + location: str, + connection_id: str, + credentials: Credentials, + settings: BigQueryToolConfig, + friendly_name: str = "", + description: str = "", + grant_vertex_ai_role: bool = True, +) -> dict: + """Create a BigQuery Cloud Resource connection for Vertex AI. + + This creates a connection that can be used with AI functions like + AI.CLASSIFY, AI.IF, AI.SCORE and remote models for AI.GENERATE_TEXT, + AI.GENERATE_EMBEDDING. + + By default, this function also grants the connection's service account + the "Vertex AI User" role, which is required to use AI functions. + + Args: + project_id (str): The Google Cloud project id. + location (str): The location/region (e.g., 'us', 'eu', 'us-central1'). + connection_id (str): The connection id to create (e.g., 'my_ai_connection'). + credentials (Credentials): The credentials to use for the request. + settings (BigQueryToolConfig): The BigQuery tool settings. + friendly_name (str): Optional friendly name for the connection. + description (str): Optional description for the connection. + grant_vertex_ai_role (bool): If True (default), automatically grants the + Vertex AI User role to the connection's service account. + + Returns: + dict: The created connection details including the service account. + + Examples: + >>> create_connection("my-project", "us", "my_ai_connection") + { + "status": "SUCCESS", + "connection_id": "my_ai_connection", + "location": "us", + "full_connection_path": "us.my_ai_connection", + "service_account": "bqcx-123456789-xxxx@gcp-sa-bigquery-condel.iam.gserviceaccount.com", + "iam_status": "Granted roles/aiplatform.user to service account" + } + """ + try: + from google.cloud import bigquery_connection_v1 + + connection_client = bigquery_connection_v1.ConnectionServiceClient( + credentials=credentials + ) + parent = f"projects/{project_id}/locations/{location}" + + # Create a Cloud Resource connection (for Vertex AI) + connection = bigquery_connection_v1.Connection( + friendly_name=friendly_name or connection_id, + description=description or "BigQuery connection for Vertex AI", + cloud_resource=bigquery_connection_v1.CloudResourceProperties(), + ) + + created_conn = connection_client.create_connection( + parent=parent, + connection_id=connection_id, + connection=connection, + ) + + service_account = "" + if created_conn.cloud_resource and created_conn.cloud_resource.service_account_id: + service_account = created_conn.cloud_resource.service_account_id + + result = { + "status": "SUCCESS", + "connection_id": connection_id, + "location": location, + "full_connection_path": f"{location}.{connection_id}", + "service_account": service_account, + } + + # Automatically grant Vertex AI User role if requested + if grant_vertex_ai_role and service_account: + iam_result = _grant_vertex_ai_role(project_id, service_account, credentials) + if iam_result["status"] == "SUCCESS": + result["iam_status"] = f"Granted roles/aiplatform.user to {service_account}" + else: + result["iam_status"] = "FAILED" + result["iam_error"] = iam_result["error_details"] + result["manual_command"] = ( + f"gcloud projects add-iam-policy-binding {project_id} " + f"--member='serviceAccount:{service_account}' " + f"--role='roles/aiplatform.user'" + ) + elif not grant_vertex_ai_role: + result["note"] = ( + "IAM role not granted. To use AI functions, grant the Vertex AI User role: " + f"gcloud projects add-iam-policy-binding {project_id} " + f"--member='serviceAccount:{service_account}' " + f"--role='roles/aiplatform.user'" + ) + + return result + except Exception as ex: + error_msg = str(ex) + if "already exists" in error_msg.lower(): + return { + "status": "ERROR", + "error_details": f"Connection '{connection_id}' already exists in {location}. Use list_connections to see existing connections.", + } + return { + "status": "ERROR", + "error_details": error_msg, + } + + def get_job_info( project_id: str, job_id: str, diff --git a/src/google/adk/tools/bigquery/skills/__init__.py b/src/google/adk/tools/bigquery/skills/__init__.py new file mode 100644 index 0000000000..5f1a5529c3 --- /dev/null +++ b/src/google/adk/tools/bigquery/skills/__init__.py @@ -0,0 +1,29 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery Skills for ADK. + +This module provides specialized skills for BigQuery operations: + +- BQMLSkill: Machine learning model training and prediction with BigQuery ML +- BQAIOperatorSkill: AI functions (AI.GENERATE, AI.CLASSIFY, etc.) +""" + +from .bq_ai_operator_skill import BQAIOperatorSkill +from .bqml_skill import BQMLSkill + +__all__ = [ + "BQMLSkill", + "BQAIOperatorSkill", +] diff --git a/src/google/adk/tools/bigquery/skills/bq_ai_operator_skill.py b/src/google/adk/tools/bigquery/skills/bq_ai_operator_skill.py new file mode 100644 index 0000000000..d2aa636b80 --- /dev/null +++ b/src/google/adk/tools/bigquery/skills/bq_ai_operator_skill.py @@ -0,0 +1,377 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery AI Operator Skill for ADK. + +This skill provides AI/ML functions for generative AI operations directly +in BigQuery SQL, including: + +- AI.GENERATE: Flexible text/multimodal generation +- AI.CLASSIFY: Categorize data into user-defined classes +- AI.EXTRACT: Extract structured data from unstructured text +- AI.IF: Filter data based on natural language conditions +- AI.SCORE: Rate/rank data by quality or relevance +- AI.EMBED: Generate embeddings for semantic search +- AI.SIMILARITY: Compare embeddings for clustering/recommendations + +For full documentation, see: +https://cloud.google.com/bigquery/docs/generative-ai-overview +""" + +from __future__ import annotations + +from typing import Any +from typing import List + +from pydantic import Field + +from ....skills.base_skill import BaseSkill +from ....skills.base_skill import SkillConfig +from ....utils.feature_decorator import experimental + + +@experimental +class BQAIOperatorSkill(BaseSkill): + """Skill for BigQuery AI Operator functions. + + This skill bundles AI functions for generative AI operations in BigQuery, + allowing agents to use LLMs directly within SQL queries. + + Key capabilities: + - Text generation and summarization + - Classification into custom categories + - Entity/data extraction from unstructured text + - Semantic filtering with natural language + - Quality scoring and ranking + - Embedding generation for vector search + - Similarity comparison for recommendations + + Example: + ```python + from google.adk.tools.bigquery.skills import BQAIOperatorSkill + + skill = BQAIOperatorSkill() + agent = Agent( + name="ai_analyst", + skills=[skill], + enable_programmatic_tool_calling=True, + ) + ``` + + Note: Requires BigQuery Enterprise edition and appropriate permissions. + """ + + name: str = Field(default="bq_ai_operator") + description: str = Field( + default=( + "AI-powered analysis in BigQuery using generative AI. " + "Generate text, classify data, extract entities, filter with " + "natural language, score/rank content, create embeddings, and " + "compute semantic similarity - all directly in SQL." + ) + ) + config: SkillConfig = Field( + default_factory=lambda: SkillConfig( + timeout_seconds=180.0, # AI operations can take time + max_parallel_calls=10, + ) + ) + + def get_tool_declarations(self) -> List[dict[str, Any]]: + """Return tool declarations for BQ AI Operator functions.""" + return [ + { + "name": "ai_generate", + "description": ( + "Generate text using AI.GENERATE. The most flexible inference " + "function - analyze any combination of text and unstructured " + "data. Use for summarization, translation, Q&A, content " + "generation, and more." + ), + "parameters": { + "input_data": "SQL query or table with input data", + "prompt": "The prompt template with column references", + "model": "Optional model name (default: gemini-pro)", + "output_column": "Name for the output column", + }, + }, + { + "name": "ai_generate_table", + "description": ( + "Generate structured table output using AI.GENERATE_TABLE. " + "Extracts structured data into a table with a custom schema " + "from unstructured input." + ), + "parameters": { + "input_data": "SQL query or table with input data", + "prompt": "The prompt describing what to extract", + "output_schema": "Schema for output table columns", + "model": "Optional model name", + }, + }, + { + "name": "ai_classify", + "description": ( + "Classify data into user-defined categories using AI.CLASSIFY. " + "Groups text/multimodal data into predefined classes." + ), + "parameters": { + "input_data": "SQL query or table with data to classify", + "input_column": "Column containing text to classify", + "categories": "List of category names/labels", + "output_column": "Name for the classification result column", + }, + }, + { + "name": "ai_extract", + "description": ( + "Extract structured information from unstructured text using " + "AI. Useful for entity extraction, parsing documents, etc." + ), + "parameters": { + "input_data": "SQL query or table with unstructured text", + "input_column": "Column containing text to process", + "extraction_spec": "What to extract (entities, fields, etc.)", + "output_columns": "Names for extracted data columns", + }, + }, + { + "name": "ai_if", + "description": ( + "Filter data using natural language conditions with AI.IF. " + "Returns TRUE/FALSE based on whether rows match the condition." + ), + "parameters": { + "input_data": "SQL query or table to filter", + "input_column": "Column to evaluate", + "condition": "Natural language condition to check", + "output_column": "Name for the boolean result column", + }, + }, + { + "name": "ai_score", + "description": ( + "Rate/score data using AI.SCORE. Assigns numeric scores to " + "rank rows by quality, relevance, or other criteria." + ), + "parameters": { + "input_data": "SQL query or table to score", + "input_column": "Column containing content to score", + "criteria": "Scoring criteria in natural language", + "output_column": "Name for the score column", + }, + }, + { + "name": "ai_embed", + "description": ( + "Generate embeddings using AI.EMBED or AI.GENERATE_EMBEDDING. " + "Creates high-dimensional vectors for semantic search, " + "clustering, and similarity comparisons." + ), + "parameters": { + "input_data": "SQL query or table with text to embed", + "input_column": "Column containing text", + "model": "Embedding model (default: text-embedding-004)", + "output_column": "Name for the embedding vector column", + }, + }, + { + "name": "ai_similarity", + "description": ( + "Compare embeddings using AI.SIMILARITY. Computes cosine " + "similarity between embedding vectors for clustering, " + "recommendations, and duplicate detection." + ), + "parameters": { + "embedding1": "First embedding column or value", + "embedding2": "Second embedding column or value", + "output_column": "Name for the similarity score column", + }, + }, + { + "name": "ai_forecast", + "description": ( + "Generate time series forecasts using AI.FORECAST. " + "Uses TimesFM model for point and interval predictions." + ), + "parameters": { + "input_data": "SQL query or table with time series data", + "timestamp_col": "Column containing timestamps", + "data_col": "Column containing values to forecast", + "horizon": "Number of time steps to forecast", + "id_cols": "Optional columns identifying multiple series", + }, + }, + ] + + def get_orchestration_template(self) -> str: + """Return example orchestration code for BQ AI Operator functions.""" + return ''' +async def analyze_customer_feedback(tools): + """Analyze customer feedback using AI functions.""" + import asyncio + + # Classify sentiment and extract entities in parallel + classify_result, extract_result = await asyncio.gather( + tools.ai_classify( + input_data=""" + SELECT feedback_id, feedback_text + FROM `my_project.my_dataset.customer_feedback` + WHERE feedback_date >= '2024-01-01' + """, + input_column="feedback_text", + categories=["positive", "negative", "neutral", "mixed"], + output_column="sentiment" + ), + tools.ai_extract( + input_data=""" + SELECT feedback_id, feedback_text + FROM `my_project.my_dataset.customer_feedback` + WHERE feedback_date >= '2024-01-01' + """, + input_column="feedback_text", + extraction_spec="product names, feature requests, complaints", + output_columns=["products", "feature_requests", "complaints"] + ) + ) + + # Summarize the results + sentiment_counts = {} + for row in classify_result.get("rows", []): + sentiment = row.get("sentiment") + sentiment_counts[sentiment] = sentiment_counts.get(sentiment, 0) + 1 + + return { + "total_feedback": len(classify_result.get("rows", [])), + "sentiment_distribution": sentiment_counts, + "sample_extractions": extract_result.get("rows", [])[:10] + } + + +async def semantic_search(tools): + """Perform semantic search using embeddings.""" + + # Generate embeddings for search query + query_embedding = await tools.ai_embed( + input_data="SELECT 'How do I reset my password?' as query", + input_column="query", + model="text-embedding-004", + output_column="query_embedding" + ) + + # Find similar documents + similar_docs = await tools.ai_similarity( + embedding1="query_embedding", + embedding2="doc_embedding", + output_column="similarity_score" + ) + + # Filter to most relevant results + relevant = [ + row for row in similar_docs.get("rows", []) + if row.get("similarity_score", 0) > 0.7 + ] + + return { + "query": "How do I reset my password?", + "num_results": len(relevant), + "top_matches": relevant[:5] + } + + +async def content_moderation(tools): + """Filter inappropriate content using AI.IF.""" + + # Filter content based on natural language criteria + filtered = await tools.ai_if( + input_data=""" + SELECT post_id, content, author_id + FROM `my_project.my_dataset.user_posts` + WHERE created_at >= CURRENT_DATE() - 7 + """, + input_column="content", + condition="content is appropriate for all ages and does not contain spam, harassment, or explicit material", + output_column="is_appropriate" + ) + + # Score remaining content for quality + scored = await tools.ai_score( + input_data=""" + SELECT post_id, content + FROM appropriate_posts + """, + input_column="content", + criteria="helpfulness, clarity, and engagement potential on a scale of 1-10", + output_column="quality_score" + ) + + appropriate_count = sum( + 1 for row in filtered.get("rows", []) + if row.get("is_appropriate") + ) + + return { + "total_posts": len(filtered.get("rows", [])), + "appropriate_posts": appropriate_count, + "flagged_posts": len(filtered.get("rows", [])) - appropriate_count, + "top_quality_posts": sorted( + scored.get("rows", []), + key=lambda x: x.get("quality_score", 0), + reverse=True + )[:10] + } +''' + + def filter_result(self, result: Any) -> Any: + """Filter BQ AI Operator results to reduce context size. + + - Truncates large result sets + - Truncates very long generated text + - Summarizes embedding vectors (too large to return in full) + """ + if not isinstance(result, dict): + return result + + filtered = result.copy() + + # Truncate large row results + if "rows" in filtered and len(filtered["rows"]) > 50: + filtered["rows"] = filtered["rows"][:50] + filtered["result_truncated"] = True + filtered["total_rows"] = len(result["rows"]) + + # Process individual rows + if "rows" in filtered: + for row in filtered["rows"]: + if isinstance(row, dict): + for key, value in row.items(): + # Truncate very long text outputs + if isinstance(value, str) and len(value) > 2000: + row[key] = value[:2000] + "... [truncated]" + + # Summarize embedding vectors (don't send full vector) + if isinstance(value, list) and len(value) > 100: + if all(isinstance(x, (int, float)) for x in value[:10]): + row[key] = { + "type": "embedding_vector", + "dimensions": len(value), + "sample": value[:5], + "note": "Full vector truncated for context efficiency", + } + + # Round numeric values + if isinstance(value, float): + row[key] = round(value, 4) + + return filtered diff --git a/src/google/adk/tools/bigquery/skills/bqml_skill.py b/src/google/adk/tools/bigquery/skills/bqml_skill.py new file mode 100644 index 0000000000..fc2bea3028 --- /dev/null +++ b/src/google/adk/tools/bigquery/skills/bqml_skill.py @@ -0,0 +1,323 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery ML (BQML) Skill for ADK. + +This skill provides machine learning capabilities using BigQuery ML, +allowing agents to train, evaluate, and use ML models directly in SQL. + +Supported model types: +- Linear/logistic regression +- K-means clustering +- Matrix factorization +- Time series (ARIMA_PLUS) +- Deep neural networks (via Vertex AI) +- XGBoost/Random Forest +- And more + +For full documentation, see: +https://cloud.google.com/bigquery/docs/bqml-introduction +""" + +from __future__ import annotations + +from typing import Any +from typing import List + +from pydantic import Field + +from ....skills.base_skill import BaseSkill +from ....skills.base_skill import SkillConfig +from ....utils.feature_decorator import experimental + + +@experimental +class BQMLSkill(BaseSkill): + """Skill for BigQuery ML operations. + + This skill bundles tools for machine learning operations in BigQuery, + including model creation, training, evaluation, and prediction. + + Supported model types: + - LINEAR_REG: Linear regression for numeric predictions + - LOGISTIC_REG: Logistic regression for classification + - KMEANS: K-means clustering for segmentation + - MATRIX_FACTORIZATION: Recommendation systems + - ARIMA_PLUS: Time series forecasting + - DNN_CLASSIFIER/DNN_REGRESSOR: Deep neural networks + - BOOSTED_TREE_CLASSIFIER/BOOSTED_TREE_REGRESSOR: XGBoost models + - RANDOM_FOREST_CLASSIFIER/RANDOM_FOREST_REGRESSOR: Random forest + + Example: + ```python + from google.adk.tools.bigquery.skills import BQMLSkill + + skill = BQMLSkill() + agent = Agent( + name="ml_analyst", + skills=[skill], + enable_programmatic_tool_calling=True, + ) + ``` + """ + + name: str = Field(default="bqml") + description: str = Field( + default=( + "Machine learning with BigQuery ML. Train, evaluate, and deploy " + "ML models using SQL. Supports regression, classification, " + "clustering, time series forecasting, and deep learning models." + ) + ) + config: SkillConfig = Field( + default_factory=lambda: SkillConfig( + timeout_seconds=300.0, # ML training can take time + max_parallel_calls=5, + ) + ) + + def get_tool_declarations(self) -> List[dict[str, Any]]: + """Return tool declarations for BQML operations.""" + return [ + { + "name": "create_model", + "description": ( + "Create and train a BigQuery ML model. " + "Supports various model types including linear regression, " + "logistic regression, k-means, time series, and more." + ), + "parameters": { + "model_name": ( + "Fully qualified model name (project.dataset.model)" + ), + "model_type": ( + "Type of model: LINEAR_REG, LOGISTIC_REG, KMEANS, " + "ARIMA_PLUS, DNN_CLASSIFIER, BOOSTED_TREE_REGRESSOR, etc." + ), + "training_data": "SQL query or table for training data", + "options": "Additional model options as key-value pairs", + }, + }, + { + "name": "evaluate_model", + "description": ( + "Evaluate a trained model's performance. Returns metrics " + "like accuracy, precision, recall, RMSE, MAE depending on " + "model type." + ), + "parameters": { + "model_name": "Fully qualified model name", + "eval_data": "Optional evaluation dataset (SQL or table)", + }, + }, + { + "name": "predict", + "description": ( + "Generate predictions using a trained model. " + "Returns predicted values along with input features." + ), + "parameters": { + "model_name": "Fully qualified model name", + "input_data": "SQL query or table for prediction input", + }, + }, + { + "name": "explain_predict", + "description": ( + "Generate predictions with feature attributions. " + "Shows which features contributed most to each prediction." + ), + "parameters": { + "model_name": "Fully qualified model name", + "input_data": "SQL query or table for prediction input", + "top_k_features": "Number of top features to show", + }, + }, + { + "name": "get_model_info", + "description": ( + "Get information about a model including training stats, " + "feature info, and hyperparameters." + ), + "parameters": { + "model_name": "Fully qualified model name", + }, + }, + { + "name": "list_models", + "description": "List all models in a dataset.", + "parameters": { + "dataset": "Fully qualified dataset (project.dataset)", + }, + }, + { + "name": "drop_model", + "description": "Delete a model from the dataset.", + "parameters": { + "model_name": "Fully qualified model name", + }, + }, + { + "name": "feature_importance", + "description": ( + "Get feature importance scores for tree-based models " + "(XGBoost, Random Forest)." + ), + "parameters": { + "model_name": "Fully qualified model name", + }, + }, + { + "name": "confusion_matrix", + "description": ( + "Generate confusion matrix for classification models. " + "Shows true/false positives and negatives." + ), + "parameters": { + "model_name": "Fully qualified model name", + "eval_data": "Optional evaluation dataset", + }, + }, + { + "name": "roc_curve", + "description": ( + "Generate ROC curve data for binary classification models. " + "Returns threshold, TPR, FPR values." + ), + "parameters": { + "model_name": "Fully qualified model name", + "eval_data": "Optional evaluation dataset", + }, + }, + ] + + def get_orchestration_template(self) -> str: + """Return example orchestration code for BQML operations.""" + return ''' +async def train_and_evaluate(tools): + """Train a model and evaluate its performance.""" + import asyncio + + # Create a linear regression model + create_result = await tools.create_model( + model_name="my_project.my_dataset.sales_predictor", + model_type="LINEAR_REG", + training_data=""" + SELECT + store_id, + product_category, + day_of_week, + is_holiday, + sales_amount + FROM `my_project.my_dataset.historical_sales` + WHERE sales_date >= '2024-01-01' + """, + options={ + "input_label_cols": ["sales_amount"], + "enable_global_explain": True + } + ) + + if create_result.get("status") != "SUCCESS": + return {"error": create_result.get("error_details")} + + # Evaluate and predict in parallel + eval_result, predictions = await asyncio.gather( + tools.evaluate_model( + model_name="my_project.my_dataset.sales_predictor" + ), + tools.predict( + model_name="my_project.my_dataset.sales_predictor", + input_data=""" + SELECT store_id, product_category, day_of_week, is_holiday + FROM `my_project.my_dataset.upcoming_dates` + """ + ) + ) + + return { + "model_created": True, + "evaluation_metrics": eval_result.get("rows", []), + "sample_predictions": predictions.get("rows", [])[:10], + "total_predictions": len(predictions.get("rows", [])) + } + + +async def cluster_customers(tools): + """Segment customers using k-means clustering.""" + # Create k-means clustering model + await tools.create_model( + model_name="my_project.my_dataset.customer_segments", + model_type="KMEANS", + training_data=""" + SELECT + customer_id, + total_purchases, + avg_order_value, + purchase_frequency, + days_since_last_purchase + FROM `my_project.my_dataset.customer_metrics` + """, + options={ + "num_clusters": 5, + "standardize_features": True + } + ) + + # Get cluster assignments + clusters = await tools.predict( + model_name="my_project.my_dataset.customer_segments", + input_data="SELECT * FROM `my_project.my_dataset.customer_metrics`" + ) + + # Summarize by cluster + cluster_counts = {} + for row in clusters.get("rows", []): + centroid_id = row.get("CENTROID_ID") + cluster_counts[centroid_id] = cluster_counts.get(centroid_id, 0) + 1 + + return { + "num_clusters": len(cluster_counts), + "cluster_sizes": cluster_counts, + "sample_assignments": clusters.get("rows", [])[:20] + } +''' + + def filter_result(self, result: Any) -> Any: + """Filter BQML results to reduce context size. + + - Truncates large result sets + - Rounds numeric metrics for readability + - Removes verbose internal fields + """ + if not isinstance(result, dict): + return result + + filtered = result.copy() + + # Truncate large row results + if "rows" in filtered and len(filtered["rows"]) > 100: + filtered["rows"] = filtered["rows"][:100] + filtered["result_truncated"] = True + filtered["total_rows"] = len(result["rows"]) + + # Round numeric values in metrics for readability + if "rows" in filtered: + for row in filtered["rows"]: + if isinstance(row, dict): + for key, value in row.items(): + if isinstance(value, float): + row[key] = round(value, 6) + + return filtered