[Feature Request]: REST Passthrough APIs with Pre/Post Plugins (JSONPath and filters)

[crivetimihai]

Feature Request: REST Passthrough APIs with Pre/Post Plugins

Summary

Add first-class passthrough support for upstream REST APIs. When a REST API is registered as a tool, the gateway also exposes a corresponding HTTP passthrough endpoint that forwards requests to the upstream, with plugin hooks before and after the call for validation, enrichment, filtering, transformation, and auditing.

Motivation

• Unify how external REST services are consumed: as MCP tools and as plain HTTP.
• Enable consistent cross-cutting concerns (auth, rate limit, PII redaction, deny/allow, response shaping) via the existing plugin framework.
• Reduce duplication between “tool invocation” and “custom reverse proxy” patterns.
• Make REST integrations discoverable and self-documented via OpenAPI.
• Supports JSONPath filtering to apply plugins selectively

Goals

• Register a REST integration once and use it as:
  • An MCP tool (tools/call), and
  • A passthrough HTTP endpoint (/passthrough/...).
• Allow attaching pre/post plugin chains at the route and/or tool level.
• Support path templating, query/header mapping, and body pass-through.
• Enforce security (allowlist, auth injection, SSRF safeguards), rate limiting, and observability.

Non-Goals (Initial)

• Full API gateway feature parity (e.g., complex routing DSL, websocket upgrades).
• Automatic OpenAPI → tool generation (could be a follow-up).

Proposed Design

1) Model additions

• New tool integration type: rest (if not already present) with fields:
  • id, name, base_url, path_template, method
  • query_mapping (dict), header_mapping (dict), timeout_ms
  • expose_passthrough (bool, default true)
  • auth (enum: none|static_header|bearer_env|bearer_secret|mcp_token|plugin)
  • allowlist (list of allowed upstream hosts/schemes)
  • plugin_chain_pre, plugin_chain_post (optional, override defaults)

2) Router

• New router under mcpgateway/routers/passthrough.py:
  • Base path: /passthrough/{namespace}/{tool_id} (configurable base, e.g. /api/tools/)
  • Methods supported: mirror the tool’s method (GET/POST/PUT/PATCH/DELETE)
  • Path segments beyond tool_id appended to path_template if trailing /** is declared
  • Query, headers, and body forwarded according to mapping rules

3) Request processing pipeline

• Pipeline per request:
  1. Resolve tool by {namespace}/{tool_id} and validate expose_passthrough
  2. Construct upstream URL from base_url + path_template using path/query/header/body mapping
  3. Run pre-plugins (validation, auth enrichment, deny/allow, rate limit, logging)
  4. Forward to upstream with timeout, stream or buffer response
  5. Run post-plugins (redaction, shape, cache, transform, audit)
  6. Return response with appropriate status, headers, and body

4) Plugins

• Use existing plugin framework (e.g., built-ins: pii_filter, deny_filter, regex_filter, resource_filter).
• Define plugin hooks:
  • Pre: on_passthrough_request(context, request) (can mutate request or raise)
  • Post: on_passthrough_response(context, request, response) (can mutate response)
• Chain selection order:
  • Route-specific: tool-level plugin_chain_pre/post if provided
  • Fallback: default passthrough chains from plugins/config.yaml

5) Configuration

• New section in config to declare defaults and chains:

# plugins/config.yaml
passthrough:
  enabled: true
  base_path: "/passthrough"
  default_timeout_ms: 20000
  default_plugin_chains:
    pre: ["deny_filter", "regex_filter", "pii_filter"]
    post: ["pii_filter"]

# Example tool definition (illustrative; align to existing tool config format)
tools:
  - id: "github.search"
    type: rest
    name: "GitHub Search"
    base_url: "https://api.github.com"
    path_template: "/search/repositories"
    method: GET
    query_mapping:
      q: "{{params.query}}"
      sort: "{{params.sort|default('stars')}}"
    header_mapping:
      Accept: "application/vnd.github+json"
      Authorization: "Bearer {{env.GITHUB_TOKEN}}"
    expose_passthrough: true
    plugin_chain_pre: ["deny_filter", "rate_limit"]
    plugin_chain_post: ["pii_filter", "response_shape"]

6) Security

• Enforce upstream host allowlist; reject requests to non-allowed hosts/schemes.
• Optionally block private IP ranges to prevent SSRF.
• Require JWT bearer auth for passthrough endpoints by default; scope to {namespace}:{tool_id}.
• Support header redaction in logs and traces.

7) Observability

• Add spans around the pipeline:
  • passthrough.request (attrs: tool.id, method, url, tenant, user)
  • passthrough.upstream (attrs: status_code, duration_ms, retries)
  • Record plugin execution as events or child spans.

8) Errors and edge cases

• Upstream errors surface with preserved status code and structured error body when possible.
• Timeouts return 504; connection failures 502; policy violations 403/422.
• Size limits for request/response bodies; configurable.

OpenAPI Exposure (optional, but desirable)

• Generate OpenAPI entries for exposed passthrough routes based on tool metadata.
• Tag routes by namespace; include summaries and example requests.

Acceptance Criteria

• Can declare a REST tool once and:
  • Invoke it via tools/call (MCP)
  • Call it via HTTP at /passthrough/{namespace}/{tool_id} with identical semantics
• Pre- and post-plugin chains execute and can mutate request/response.
• Security guardrails: allowlist + SSRF protections + auth requirement documented and enforced.
• Basic OpenAPI entries appear for exposed routes, or explicitly deferred with a follow-up issue.
• Unit tests cover routing, mappings, plugin invocation, and error paths.

Implementation Plan (phased)

• Phase 1: Data model + config parsing for rest tools and passthrough defaults
• Phase 2: Router + pipeline skeleton with mock upstream
• Phase 3: Plugin hooks + built-in filters integration
• Phase 4: Security guardrails (allowlist + SSRF) and auth
• Phase 5: Observability spans + metrics
• Phase 6: Docs + examples + tests

Risks / Mitigations

• SSRF: strict allowlist + private range blocking + URL normalization
• Secret leakage: header redaction + post-filtering + audit logs
• Backwards compat: default expose_passthrough: false for existing tools if needed; true for new rest type

Notes

• This complements existing federation/gateway features and stdio/SSE/WebSocket transports.
• Keep scope focused on single-upstream REST calls; multi-hop/mashups can be handled by tools or future compositors.

[rakdutta]

@crivetimihai @madhav165
Problem Statement

We need to align the requested fields with the existing Tool configuration schema to determine which fields are already supported, which need to be added in the Add Tool form, and which should be editable in the Edit Tool form. Example values marked as ? need clarification and agreement on what defaults or formats should be applied (timeout_ms, allowlist).

---

Requested Fields – Add/Edit Tool Form Analysis

S.No	Requested Field	Already Present in Table	Available in Add Tool Form UI	Available in Edit Tool Form UI	Example Value	Mapping	Remarks
1	id	Yes	–	–	–	–	–
2	name	Yes	–	–	–	–	–
3	base_url	No	No	TBD	https://api.github.com	Extracted from URL field	–
4	path_template	No	No	TBD	/search/repositories	Extracted from URL field	–
5	method	Yes	–	–	–	request_type	Additional field not required – method value is already captured through request_type.
6	query_mapping (dict)	No	Yes	Yes	q: "{{params.query}}" sort: "{{params.sort	default('stars')}}"	–
7	header_mapping (dict)	No	Yes	Yes	Accept: "application/vnd.github+json" Authorization: "Bearer {{env.GITHUB_TOKEN}}"	–	–
8	timeout_ms	No	TBD	–	?	–	Example not defined – needs clarity on default value or acceptable range (ms).
9	expose_passthrough (bool, default=true)	No	Yes	Yes	TRUE	–	–
10	auth (enum: none | static_header | bearer_env | bearer_secret | mcp_token | plugin)	Yes	–	–	–	–	Additional auth field not required – can be handled via existing fields.
11	allowlist (list of allowed upstream hosts/schemes)	No	Yes	Yes	?	–	Example not defined – need clarity on expected format (e.g., hostnames, regex, or full schemes).
12	plugin_chain_pre	No	Yes	Yes	["deny_filter", "rate_limit"]	–	–
13	plugin_chain_post (optional, override defaults)	No	Yes	Yes	["pii_filter", "response_shape"]	–	–

---

Action Items

1. Confirm timeout_ms

   • Define whether a default value should be set.
   • Clarify acceptable range (milliseconds).

2. Clarify allowlist format

   • Should it include only hostnames?
   • Should regex patterns or complete scheme definitions be supported?