Skip to content

feat: add Brave Search & Baidu Search integration with sub-source support#42

Open
YizukiAme wants to merge 7 commits intoTHU-MAIC:mainfrom
YizukiAme:feat/brave-baidu-search
Open

feat: add Brave Search & Baidu Search integration with sub-source support#42
YizukiAme wants to merge 7 commits intoTHU-MAIC:mainfrom
YizukiAme:feat/brave-baidu-search

Conversation

@YizukiAme
Copy link
Copy Markdown
Contributor

Summary

Adds Brave Search and Baidu Search as web search providers, with Baidu sub-source configuration (Web Search, Baike, Scholar).

Changes

New Search Providers

  • Brave Search: Uses public page scraping — no API key required. Parses Brave's Svelte-rendered HTML to extract search results.
  • Baidu Search: Integrates Web Search, Baike (encyclopedia), and Scholar APIs. Users can toggle individual sub-sources on/off.

Provider Icons

  • Added icons for Brave (public/icons/brave.png), Baidu (public/icons/baidu.png), and Tavily (public/icons/tavily.jpg)
  • Icons displayed in toolbar pill, provider dropdown, and settings sidebar

UI Improvements

  • Baidu sub-source toggles use standard Switch components (consistent with rest of the UI)
  • Brave API key field shown but marked optional with explanatory notice
  • Search toggle enabled when any provider is available (not just selected one)

Technical Details

  • lib/web-search/brave.ts: Regex-based HTML parser for Brave's current Svelte-rendered page structure
  • lib/web-search/baidu.ts: Accepts subSources parameter, queries enabled sources in parallel
  • lib/store/settings.ts: Added baiduSubSources state with per-source toggles
  • app/api/web-search/route.ts: Brave bypasses API key check; Baidu passes sub-source config
  • components/generation/generation-toolbar.tsx: Fixed webSearchAvailable logic

Files Changed

File Description
lib/web-search/brave.ts Fixed HTML parser for current Brave Search structure
lib/web-search/baidu.ts Added sub-source toggle support
lib/web-search/types.ts Added BaiduSubSources interface
lib/web-search/constants.ts Added sub-source config, icon paths
lib/store/settings.ts Added Baidu sub-source state
app/api/web-search/route.ts Brave no-key logic, Baidu sub-sources
app/generation-preview/page.tsx Pass sub-sources in API call
components/settings/web-search-settings.tsx Sub-source toggles, optional key UI
components/generation/generation-toolbar.tsx Icons, fixed availability logic
public/icons/* Provider icons

Testing

  • ✅ Baidu Search returns results (verified via API call with key)
  • ✅ Brave Search parser matches current HTML structure (tested with saved HTML, 20 results parsed)
  • ✅ Build passes (pnpm build exit code 0)
  • ✅ UI verified: icons display correctly, search toggle enables properly, Baidu sub-source switches render consistently

YizukiAme and others added 6 commits March 17, 2026 02:46
- Add 'brave' and 'baidu' to WebSearchProviderId union type
- Register providers in WEB_SEARCH_PROVIDERS with configs
- Create brave.ts: HTML scraping provider (no API key required)
- Create baidu.ts: Qianfan REST API provider (Bearer token auth)
- Update API route for multi-provider dispatch via 'provider' param
- Generalize resolveWebSearchApiKey for provider-agnostic key resolution
- Add BRAVE_API_KEY and BAIDU_API_KEY to .env.example
- Update settings store with default configs for new providers
- Update i18n strings (zh-CN + en-US) to be provider-agnostic
- Update web-search-settings UI with conditional API key display
- Pass provider param in generation-preview page fetch call
- Fix snippet regex to match 'class=snippet svelte-...' (not exact 'class=snippet')
- Update title extraction to use 'search-snippet-title' class
- Extract descriptions from 'generic-snippet' div (current) with fallback to 'snippet-description' p (legacy)
- Handle relative date prefixes ('2 days ago -')
- Add provider icons (Brave, Baidu, Tavily)
- Update Baidu sub-source toggles to use Switch components
- Fix webSearchAvailable logic to enable search when any provider is available
- Make Brave API key optional in settings UI
# Conflicts:
#	components/settings/web-search-settings.tsx
- Add shared normalizeWebSearchQuery util (400-char limit)
- Apply query normalization to Brave and Baidu providers
- Refactor Tavily to use shared normalizer
- Add isSelectedProviderUsable check in toolbar toggle
- Add pre-submit web search provider validation in page.tsx
- Add i18n strings for provider unavailable toast

Fixes toolbar allowing web search enable when selected provider
has no API key configured, while another provider (Brave) is
available without a key.
@YizukiAme YizukiAme force-pushed the feat/brave-baidu-search branch from edff981 to 6cc35da Compare March 19, 2026 18:43
ChaseAfterLight pushed a commit to ChaseAfterLight/OpenMAIC that referenced this pull request Apr 7, 2026
Copy link
Copy Markdown
Contributor

wyuc commented May 5, 2026

Thanks for the contribution, and sorry for the long delay on my side.

This PR is now quite far behind main and conflicts with the current web-search architecture, so it is not mergeable as-is. There are also a few areas that would need another look after an update, including how the new providers plug into the current searchWeb flow, server-side classroom generation, Baidu sub-source handling, and test coverage for the Brave parser.

If you’re still interested in maintaining this PR, please rebase it onto the latest main and adapt it to the current web-search provider architecture. I’ll be happy to review it again after that.

@YizukiAme
Copy link
Copy Markdown
Contributor Author

Got it! I'll rebase this onto main and update the code to fit the current architecture in a couple of days when I have some bandwidth.
Thanks for the review~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants