configurable auto refresh for automatic tools/prompts/resources sync during health checks

[kevalmahajan]

✨ Feature / Enhancement PR

🔗 Epic / Issue

Closes #1910, #172

---

🚀 Summary (1-2 sentences)

Added configurable AUTO_REFRESH_SERVERS feature that automatically syncs gateway tools, resources, and prompts with the database during health checks. When enabled, the system detects changes from MCP servers and performs add/update/delete operations only when necessary.

The tools, prompts, and resources are compared against those stored in the database, and any required add, update, or delete operations are performed.

---

🔧 Usage

Enable via environment variable:

AUTO_REFRESH_SERVERS=true

1. HEALTH_CHECK_INTERVAL=30 # Optional: shorter interval for testing
2. Add an mcp server with 3 tools in the gateway.
3. Update the mcp server by reducing the number of tools or adding some tools.
4. Wait for the health check and notice that the new tools are automatically refreshed in the DB.
5. Same applies for prompts and resources as well.

---

🧪 Checks

• make lint passes
• make test passes
• CHANGELOG updated (if user-facing)

---

📓 Notes (optional)

flowchart TD
    A[Health Check Loop] -->|Every HEALTH_CHECK_INTERVAL| B{Gateway Healthy?}
    B -->|No| C[Handle Failure]
    B -->|Yes| D{AUTO_REFRESH_SERVERS?}
    D -->|No| E[Update last_seen]
    D -->|Yes| F[Fetch tools from MCP Server]
    F --> G{Any Changes?}
    G -->|No| E
    G -->|Yes| H[Sync DB: Add/Update/Delete]
    H --> I[Invalidate Caches]
    I --> E

Loading

• The feature reuses existing _update_or_create_tools/resources/prompts helper methods
• Uses fresh_db_session() to avoid holding DB connections during HTTP calls to MCP servers
• SQLAlchemy's db.dirty tracking detects in-place updates to determine if commit is needed
• Tool lookup cache is invalidated per-gateway for efficiency

[crivetimihai]

Thanks for the PR — it covers health-check auto-refresh. For REQ-002 we still need manual refresh API, per-gateway intervals, list_changed event handling, and full tool spec sync (annotations/outputSchema). Tracking those gaps in #1984.

[crivetimihai]

Code Review Changes

Rebased onto main, squashed 7 commits into 1, and addressed the following issues:

Fixes Applied

Issue	Severity	Fix
Cache invalidation for resource/prompt updates	High	Added per-type update tracking (tools_updated, resources_updated, prompts_updated) instead of global db.dirty count. Now properly invalidates each cache type when updates occur.
Empty server clearing stale rows	Medium	Changed logic to only skip empty responses for authorization_code OAuth gateways (incomplete auth flow). Non-auth_code gateways with empty responses now properly clear stale items.
Double OAuth token requests	Medium (perf)	Added pre_auth_headers parameter to _initialize_gateway and _refresh_gateway_tools_resources_prompts. Health check now passes pre-authenticated headers to avoid duplicate token fetch.
Deletion scope too broad	Medium	Narrowed mcp_created_via_values to only actual MCP discovery sources: {"MCP", "federation", "health_check", "oauth", "update"}. User-created entries ("api", "ui", None/legacy) are now preserved.
Missing tests	Low	Added 9 new unit tests in test_gateway_auto_refresh.py covering auto-refresh paths, cache invalidation, pre-auth headers, and deletion scope.

Key Implementation Details

• Per-type dirty tracking: Tracks db.dirty objects before/after _update_or_create_* calls to count updates per type (tools/resources/prompts)
• Selective cache invalidation: Only invalidates caches for types that actually changed (add/remove/update)
• Safe deletion scope: Only removes items with created_via in known MCP discovery sources, preserving user-created entries
• OAuth token reuse: Passes pre_auth_headers from health check to _initialize_gateway to avoid fetching tokens twice per health check cycle

Files Changed

• mcpgateway/services/gateway_service.py - Core implementation fixes
• tests/unit/mcpgateway/services/test_gateway_auto_refresh.py - New test file (9 tests)

All tests pass.

[crivetimihai]

Approved - created [FEATURE REQUEST]: Full tool list/spec refresh (polling + API + list_changed) #1984 for remaining features