Skip to content

feat(search): add SearXNG search provider support - #151

Open
alanpeng wants to merge 9 commits into
deer-flow:mainfrom
alanpeng:feat/searxng-pr
Open

alanpeng wants to merge 9 commits into
deer-flow:mainfrom
alanpeng:feat/searxng-pr

Conversation

@alanpeng

Copy link
Copy Markdown

Summary

Adds SearXNG as a third search provider for the built-in web_search tool (alongside Brave and Firecrawl). SearXNG is a self-hosted, privacy-respecting metasearch engine that exposes a JSON API — no API key required.

What changes

  • Search settings (SearchSettings): new searxngBaseUrl field, default http://localhost:8080, persisted via the existing settings manager.
  • Runtime (packages/runtime): web_search dispatches to the SearXNG JSON API (GET {base}/search?q=...&format=json&language=auto) with Accept: application/json and X-Forwarded-For/X-Real-IP: 127.0.0.1 headers. Non-OK responses surface the SearXNG error body instead of a bare status message.
  • Settings UI (desktop): SearXNG option in the search provider dropdown + a service-URL field shown when SearXNG is selected.
  • LangGraph generator parity: the generated Python web_search.py mirrors the TS implementation (same URL/params/headers/error handling), per the project's generator-parity requirement; generated sources regenerated.
  • Codegen (generate-project-button): the generated project env now forwards SEARXNG_BASE_URL instead of writing undefined.
  • Tests: TS + generated-Python coverage for query building, result parsing, non-OK error bodies, and settings persistence.

Configuration

Set the provider to SearXNG in Settings → Search and point the service URL at a self-hosted SearXNG instance (default http://localhost:8080), or set SEARXNG_BASE_URL in the environment.

alanpeng and others added 9 commits August 22, 2026 11:30
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…tings

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Task 1 made SearchSettings.searxngBaseUrl required; latent call sites the
changed-file checks never saw (untouched test fixtures and the playground
codegen button) failed the full-repo typecheck that CI runs. Also fixes a
real gap: a searxng provider generating a project would have written
SEARXNG_BASE_URL=undefined into .env.

Co-Authored-By: Claude <noreply@anthropic.com>
@alanpeng

Copy link
Copy Markdown
Author

Friendly ping for review — happy to iterate on anything.

Short version: adds searxng as a built-in web-search provider for self-hosted SearXNG instances — provider entry + base-URL field in search settings, JSON format API integration in the built-in web tools, and the matching backend in the LangGraph generator so generated projects stay in parity. No new npm dependencies; failures surface the service's error body instead of failing silently. Design notes and the test matrix are in the repo (docs/superpowers/specs/, packages/core/tests).

This is one of three open PRs from us (#150 Windows support, #152 i18n) — independent of each other, any review/merge order works.

@MagicCube MagicCube left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding SearXNG support — this is a useful provider for self-hosted and privacy-conscious setups. I found three issues that should be addressed before merging: the result limit is not enforced, the client-IP headers are unsafe/unnecessary, and the setup/error handling assumes JSON is enabled even though stock SearXNG commonly disables it. The TypeScript and generated Python implementations should remain in parity when fixing these.

url.searchParams.set("format", "json");
url.searchParams.set("language", "auto");
url.searchParams.set("pageno", "1");
url.searchParams.set("limit", String(limit));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Enforce limit locally. SearXNG’s documented Search API does not define a limit request parameter, so instances may ignore this value; the code then maps and returns every item in json.results. That means web_search({ limit: 5 }) can return an instance-sized page and consume substantially more model context than requested. Please apply slice(0, limit) (with the normal bounds) to the response, and mirror the fix in generated web_search.py. API reference: https://docs.searxng.org/dev/search_api.html

// SearXNG botdetection rejects requests without a client IP; the
// instance must trust this proxy (server.trust_x_forwarded_for).
"X-Forwarded-For": "127.0.0.1",
"X-Real-IP": "127.0.0.1",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not forge loopback client-IP headers. SearXNG falls back to the connection’s REMOTE_ADDR when forwarding headers are absent, so these headers are not required for bot detection. On an instance configured to trust forwarded headers, hard-coding 127.0.0.1 misattributes every caller and can undermine IP-based logging, rate limiting, or access policy. Please omit both headers in the desktop and generated Python implementations. Reference: https://docs.searxng.org/src/searx.botdetection.html

onBlur={() => void persist(settings)}
/>
<p className="text-muted-foreground text-xs">
Self-hosted SearXNG instance. No API key required.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Document the required JSON format and handle non-JSON errors. SearXNG only serves format=json when JSON is enabled under search.formats; otherwise it returns 403, and the default configuration commonly enables only HTML. The UI currently implies that entering a URL is sufficient. Please mention the JSON requirement, and make the provider check res.ok/read text defensively before calling res.json(), since HTML 403/429 responses currently surface only a JSON parse error. References: https://docs.searxng.org/dev/search_api.html and https://docs.searxng.org/admin/settings/settings_search.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants