Update of GitHub Copilot Instructions file for current MCP/RAG system

.github/copilot-instructions.md

@@ -10,57 +10,6 @@ Use GitHub MCP tools for gathering repository information repos outside of globa
 
 This document provides comprehensive guidance for AI agents working on the NOAA Global Workflow system - a complex weather forecasting framework supporting multiple operational and research workflows.
 
-## Essential Developer Workflows
-
-### Build System Commands
-```bash
-# Build all components (from sorc/)
-./build_all.sh                    # Default build
-./build_all.sh -d                 # Debug mode
-./build_all.sh -f                 # Fast build with -DFASTER=ON
-./build_all.sh -v                 # Verbose output
-./build_all.sh -k                 # Kill all builds if any fails
-
-# Build specific systems
-./build_all.sh gfs               # GFS forecast system
-./build_all.sh gefs              # GEFS ensemble system  
-./build_all.sh sfs               # Seasonal forecast system
-./build_all.sh gcafs             # Climate analysis system
-./build_all.sh gsi               # GSI data assimilation
-./build_all.sh gdas              # GDAS system
-./build_all.sh all               # All systems
-```
-
-### Experiment Setup Workflow
-```bash
-# 1. Environment setup (CRITICAL - must be done first)
-source ush/detect_machine.sh
-module use modulefiles
-module load module_gwsetup.${MACHINE_ID}
-source dev/workflow/gw_setup.sh
-
-# 2. Create experiment
-cd dev/workflow
-python setup_expt.py gfs forecast-only \
-  --pslot EXPERIMENT_NAME \
-  --configdir parm/config/gfs \
-  --comroot /path/to/data \
-  --expdir /path/to/experiment
-
-# 3. Generate workflow XML
-python setup_xml.py /path/to/experiment
-```
-
-### Platform-Specific Development
-```bash
-# Supported platforms (use detect_machine.sh)
-WCOSS2    # Tier 1 - Full operational support
-Hercules  # Tier 1 - MSU, no TC Tracker  
-Hera      # Tier 2 - NOAA RDHPCS
-Orion     # Tier 2 - MSU, GSI runs slowly
-Gaea      # Cloud platforms via EPIC
-```
-
 ## System Architecture Overview
 
 ### Core Components
@@ -167,6 +116,58 @@ ush/                       # Utility scripts and environment setup
 └── detect_machine.sh      # Machine detection and module loading
 ```
 
+## Essential Developer Workflows
+
+### Build System Commands
+```bash
+# Build all components (from sorc/)
+./build_all.sh                    # Default build
+./build_all.sh -d                 # Debug mode
+./build_all.sh -f                 # Fast build with -DFASTER=ON
+./build_all.sh -v                 # Verbose output
+./build_all.sh -k                 # Kill all builds if any fails
+
+# Build specific systems
+./build_all.sh gfs               # GFS forecast system
+./build_all.sh gefs              # GEFS ensemble system  
+./build_all.sh sfs               # Seasonal forecast system
+./build_all.sh gcafs             # Climate analysis system
+./build_all.sh gsi               # GSI data assimilation
+./build_all.sh gdas              # GDAS system
+./build_all.sh all               # All systems
+```
+
+### Experiment Setup Workflow
+```bash
+# 1. Environment setup (CRITICAL - must be done first)
+source ush/detect_machine.sh
+module use modulefiles
+module load module_gwsetup.${MACHINE_ID}
+source dev/workflow/gw_setup.sh
+
+# 2. Create experiment
+cd dev/workflow
+python setup_expt.py gfs forecast-only \
+  --pslot EXPERIMENT_NAME \
+  --configdir parm/config/gfs \
+  --comroot /path/to/data \
+  --expdir /path/to/experiment
+
+# 3. Generate workflow XML
+python setup_xml.py /path/to/experiment rocoto
+```
+
+### Platform-Specific Development
+```bash
+# Supported platforms (use detect_machine.sh)
+WCOSS2    # Tier 1 - Full operational support
+Hercules  # Tier 1 - MSU, no TC Tracker  
+Hera      # Tier 2 - NOAA RDHPCS
+Orion     # Tier 2 - MSU, GSI runs slowly
+Gaea-C6   # Tier 1 - Fully supported platform capable of running retrospectives
+Ursa      # Tier 1 - Fully supported, but cannot run high resolution or GCAFS cases
+```
+
 ## Key Architectural Patterns
 
 ### Factory Pattern Usage
@@ -430,74 +431,215 @@ def test_task_creation():
 3. Create modulefiles for environment setup
 4. Update environment configurations in `env/` directory
 
-## MCP/RAG Tool Integration
+## EIB MCP/RAG Server Integration (v3.6.2)
 
-### Available Global Workflow MCP Tools
-This repository includes a specialized Model Context Protocol (MCP) server with Retrieval-Augmented Generation (RAG) capabilities. These tools provide intelligent access to workflow documentation, operational guidance, and contextual analysis.
+The **EIB MCP-RAG Server** provides AI-assisted access to the Global Workflow system through the Model Context Protocol (MCP) with Retrieval-Augmented Generation (RAG) capabilities. This server is purpose-built to support NOAA's operational GFS, GEFS, and SFS forecasting infrastructure.
 
-#### When to Use MCP Tools
-**USE MCP tools when:**
-- Researching unfamiliar workflow components or operational procedures
-- Need to search across documentation for specific concepts or patterns
-- Seeking operational guidance for HPC systems or deployment procedures
-- Analyzing dependencies or workflow relationships
-- Looking for code patterns similar to current implementation
-- Need contextual explanations that require deep domain knowledge
+### Architecture Overview
 
-**DON'T use MCP tools when:**
-- You have direct access to specific files and can read them efficiently
-- Task is simple file creation or editing without research needs
-- Information is already available in current context
-- RAG system components are not initialized (will return placeholder responses)
-
-#### Available MCP Tools
-
-**Core Workflow Tools:**
-- `mcp_global-workfl_get_workflow_structure` - System architecture and component overview
-- `mcp_global-workfl_list_job_scripts` - Complete inventory of workflow job scripts
-- `mcp_global-workfl_get_system_configs` - HPC platform-specific configurations
-- `mcp_global-workfl_explain_workflow_component` - Deep component analysis and explanation
-
-**RAG-Enhanced Intelligence Tools:**
-- `mcp_global-workfl_search_documentation` - Semantic search across all workflow documentation
-- `mcp_global-workfl_explain_with_context` - Contextual explanations using RAG knowledge base
-- `mcp_global-workfl_find_similar_code` - Vector-based code pattern matching and similarity search
-- `mcp_global-workfl_get_operational_guidance` - HPC operational procedures and best practices
-- `mcp_global-workfl_analyze_workflow_dependencies` - Graph-based dependency analysis and mapping
-
-**GitHub Ecosystem Tools (Live Repository Access):**
-- `github_search_repositories` - Search NOAA-EMC repositories with filtering
-- `github_get_repository_content` - Access files from any repository in real-time
-- `github_search_code` - Code pattern search across entire NOAA-EMC ecosystem
-- `github_get_issues` - GitHub issues for troubleshooting context
-- `github_cross_repo_analysis` - Advanced analysis across multiple repositories
-
-#### Proper Tool Usage Display
-When using MCP tools, acknowledge their usage to demonstrate intelligent tool selection:
-
-```markdown
-**Research Approach:** Using `mcp_globalworkflo_search_documentation` to find relevant 
-examples and `mcp_globalworkflo_get_operational_guidance` for HPC-specific procedures.
+```
+AI Clients (VS Code Copilot, LangFlow, Claude Desktop)
+                    │
+                    ▼ HTTP/MCP Protocol
+         ┌──────────────────────────┐
+         │  Docker MCP Gateway      │  Port 18888
+         │  (Streaming Transport)   │
+         └──────────────────────────┘
+                    │ spawns
+                    ▼
+         ┌──────────────────────────┐
+         │  MCP Server Container    │  34 tools
+         │  eib-mcp-rag:latest      │
+         └──────────────────────────┘
+                    │
+         ┌──────────┴──────────┐
+         ▼                     ▼
+    ChromaDB (8080)      Neo4j (7687)
+    Vector Embeddings    Code Graph DB
 ```
 
-**Example Integration:**
-```markdown
-Let me research this using the MCP tools to ensure comprehensive coverage:
+### Tool Categories & Separation of Concerns
 
-[Tool usage and results]
+The MCP server implements **7 tool modules** with clear separation of concerns (SOC):
 
-Based on the MCP analysis above, here's the recommended approach...
-```
+---
+
+#### 1. Workflow Info Tools (3 tools - Static, No DB)
+**Module**: `WorkflowInfoTools.js`
+**Purpose**: File system-based access to workflow structure - NO database dependencies
+
+| Tool | Description |
+|------|-------------|
+| `get_workflow_structure` | System architecture overview (jobs, scripts, parm, ush, sorc, env, docs) |
+| `get_system_configs` | HPC platform configurations (HERA, HERCULES, ORION, WCOSS2, GAEA) |
+| `describe_component` | File system-based component descriptions (static analysis) |
+
+**When to use**: Quick lookups, understanding directory structure, platform configs
+
+---
+
+#### 2. Code Analysis Tools (4 tools - Neo4j Graph)
+**Module**: `CodeAnalysisTools.js`
+**Purpose**: Code structure analysis via graph database traversal
+
+| Tool | Description |
+|------|-------------|
+| `analyze_code_structure` | File/function/class analysis with dependency tree (depth 1-3) |
+| `find_dependencies` | Upstream imports and downstream importers (both directions) |
+| `trace_execution_path` | Call chain tracing from any function (max depth 5) |
+| `find_callers_callees` | Fan-in/fan-out analysis with complexity scoring |
+
+**When to use**: Understanding code relationships, refactoring impact, debugging call chains
+
+---
 
-#### MCP Tool Development Context
-These tools are actively being developed and refined on the `MCP_node.js-RAG_development` branch. When the RAG components are not fully initialized, tools may return placeholder responses indicating the need for vector database setup or document ingestion.
+#### 3. Semantic Search Tools (6 tools - ChromaDB Vectors + Graph Hybrid)
+**Module**: `SemanticSearchTools.js`
+**Purpose**: RAG-powered documentation and code search
 
-**MCP Server Location**: All MCP tools are implemented in `dev/ci/scripts/utils/Copilot/mcp_server_node/`:
-- `mcp-server-rag.js` - Main RAG-enhanced server with 9 workflow tools
-- `mcp-server-github-rag.js` - GitHub ecosystem integration with 14 total tools  
-- `start-mcp-server-node.sh` - Primary startup script
-- Configuration files: `mcp-config.env`, `package.json`, `package-rag.json`
+| Tool | Description |
+|------|-------------|
+| `search_documentation` | Hybrid semantic + graph search across ingested docs |
+| `find_related_files` | Find files with similar import dependencies |
+| `explain_with_context` | Multi-source RAG explanations (technical, operational, config) |
+| `get_knowledge_base_status` | Vector DB + Graph DB health and statistics |
+| `list_ingested_urls` | Show all documentation sources ingested into RAG |
+| `get_ingested_urls_array` | Structured URL array for programmatic access |
 
-**Note for Development:** If you encounter placeholder responses from RAG-enhanced tools, this indicates the vector database needs initialization or document ingestion. The core workflow tools should always provide functional responses.
+**When to use**: Conceptual questions, finding relevant documentation, understanding unfamiliar components
 
+---
+
+#### 4. EE2 Compliance Tools (5 tools - Standards Validation)
+**Module**: `EE2ComplianceTools.js`
+**Purpose**: NOAA NWS EE2 (Enterprise Environmental 2) standards validation
+
+| Tool | Description |
+|------|-------------|
+| `search_ee2_standards` | Search EE2 compliance standards documentation |
+| `analyze_ee2_compliance` | Analyze code for EE2 compliance with recommendations |
+| `generate_compliance_report` | Generate compliance reports (summary, detailed, checklist) |
+| `scan_repository_compliance` | Full repository scan with Phase 2 SME-corrected patterns |
+| `extract_code_for_analysis` | Extract code snippets for LLM passthrough analysis |
+
+**When to use**: Pre-commit compliance checks, NCO production readiness, code reviews
+
+**Phase 2 Corrections Applied**:
+- `set -eu` is NOT required (80% false positive rate) - only `set -x` for debug logging
+- Uses `err_chk`/`err_exit` utilities instead of explicit exit statements
+- Evidence-based analysis with RST line references
+
+---
+
+#### 5. Operational Tools (3 tools - HPC Procedures)
+**Module**: `OperationalTools.js`
+**Purpose**: HPC operational guidance and workflow explanations
+
+| Tool | Description |
+|------|-------------|
+| `get_operational_guidance` | Platform-specific procedures with urgency levels (routine/urgent/emergency) |
+| `explain_workflow_component` | Graph-enriched component explanations |
+| `list_job_scripts` | Categorized job script inventory (analysis, forecast, post, archive) |
+
+**When to use**: HPC deployment questions, understanding job scripts, operational procedures
+
+---
+
+#### 6. GitHub Integration Tools (4 tools - Live Repository Access)
+**Module**: `GitHubTools.js`
+**Purpose**: Cross-repository analysis and issue tracking
+
+| Tool | Description |
+|------|-------------|
+| `search_issues` | Search NOAA-EMC GitHub issues (open/closed/all) |
+| `get_pull_requests` | PR information with diff context |
+| `analyze_workflow_dependencies` | Cross-repo dependency analysis (upstream/downstream/circular) |
+| `analyze_repository_structure` | Multi-repo structure comparison (global-workflow, GSI, UFS_UTILS) |
+
+**When to use**: Bug investigation, PR reviews, understanding cross-repo impacts
+
+---
+
+#### 7. SDD Workflow Tools (7 tools - Development Orchestration)
+**Module**: `SDDWorkflowTools.js`
+**Purpose**: Software Design Document (SDD) framework execution
+
+| Tool | Description |
+|------|-------------|
+| `list_sdd_workflows` | List available SDD workflows |
+| `get_sdd_workflow` | Get workflow phases, steps, and metadata |
+| `execute_sdd_workflow` | Execute workflow with dry-run option |
+| `get_sdd_execution_history` | View execution history |
+| `validate_sdd_compliance` | Validate code against SDD framework |
+| `get_sdd_framework_status` | Framework integration status and metrics |
+| `execute_sdd_workflow_supervised` | Human-in-loop execution with approval gates |
+
+**When to use**: Feature development following SDD methodology, multi-step workflows
+
+---
+
+#### 8. Utility Tools (2 tools - Server Management)
+**Built into**: `UnifiedMCPServer.js`
+
+| Tool | Description |
+|------|-------------|
+| `get_server_info` | MCP server info, tool counts, configuration |
+| `mcp_health_check` | Empirical health validation (heartbeat, collections, documents, queries) |
+
+**When to use**: Debugging MCP issues, verifying RAG system health
+
+---
+
+### RAG Knowledge Base Sources
+
+The RAG system ingests documentation from multiple tiers:
+
+| Tier | Sources | Purpose |
+|------|---------|---------|
+| **Tier 1 Critical** | global-workflow RTD, EE2 Standards | Core workflow documentation |
+| **Tier 2 Workflow** | Rocoto, ecFlow, wxflow, PyFlow | Workflow engine documentation |
+| **Tier 3 Models** | UFS Weather Model, JEDI, FV3 | Forecast model documentation |
+| **Tier 4 Build** | Spack, spack-stack, hpc-stack | Build system documentation |
+| **Tier 5 Standards** | Google Shell Style, PEP8, NumPy docstrings | Coding standards |
+
+### When to Use MCP Tools
+
+**USE MCP tools when:**
+- Searching for concepts across documentation (semantic search)
+- Analyzing code dependencies and call chains (graph traversal)
+- Checking EE2 compliance before committing code
+- Understanding HPC platform-specific procedures
+- Investigating GitHub issues related to a component
+- Following SDD development workflows
+
+**DON'T use MCP tools when:**
+- You already have the file open and can read it directly
+- Simple file edits that don't need context research
+- The question is answered in the current conversation context
+
+### Tool Availability by Connection Mode
+
+| Mode | Available Tools | Notes |
+|------|-----------------|-------|
+| **VS Code Local** | All 34 tools | Direct stdio connection |
+| **Docker Gateway** | All 34 tools | HTTP via port 18888 |
+| **Remote (no container)** | Core workflow tools only | Use `content` parameter for file analysis |
+
+### Content Abstraction for Remote Access
+
+For remote MCP clients (e.g., LangFlow) without filesystem access, tools support content parameters:
+
+```javascript
+// Instead of filesystem path:
+analyze_ee2_compliance({ content: "#!/bin/bash\nset -x\n..." })
+
+// Batch file analysis:
+scan_repository_compliance({ 
+  files: [
+    { name: "JGFS_FORECAST", content: "..." },
+    { name: "exgfs_fcst.sh", content: "..." }
+  ]
+})
+```
+---
 Remember: This is a production weather forecasting system. Changes must be thoroughly tested and should not disrupt operational workflows. Always follow the existing patterns and conventions when extending the system