Make webSearch into Plugins system

[itaybar]

I thought about making the web search concept more generic, Like an "RAG marketplace" or plugins, meaning anyone can configure there own RAG systems into ChatUI, allowing adding plugins for proprietary data when deploying ChatUI on-prem.

• Rename websearch into plugin or RAG
• allow to configure RAG from external service, for example http request that sends the input, and get in exchange the tok k results (all the embedding, vector similarity, hybrid search (using both TFIDF/BM25 AND vector similarity) etc in the external service
• make the webseach toggle into dropdown to select what RAG do you wanna use (allowing multiple can be a bit harder, so maybe start with selecting only one from the dropdown)
• The plugin can return the sources to show them (just like in websearch)
  POST https://my-custom-plugin/retrieve
  {
  "input": "some user question",
  "user_email": "[email protected]"
  }
  returned json
  {
  retrieved_docs: [{"text":"this is some retrieve info from private source", source: "someurl"}]
  }

the user_email field is an extra security level to check rather this email actually allowed to access the proprietary data

[mishig25]

this is great idea !

as I work on #641, I realize that we would indeed need to make general RAG interface/system.

websearch, pdf-chat would be implementations of this RAG interface. And we would add implementation for vector DBs as well

[julien-c]

I realize that we would indeed need to make general RAG interface/system.

This makes a lot of sense IMO @mishig25 🔥

[itaybar]

Another good Idea.
you can you langchain as your plugins (you can host langchain with langserve), and make an API to the plugin, In this the most generic and convenient way, since Langchain is the most popular tool for RAGs