Merge branch 'main' into DS-4821b

sid-swirl · web-flow · commit 9d4a2b707ccc · 2025-09-15T16:52:00.000-04:00
diff --git a/docs/AI-Search-Assistant.md b/docs/AI-Search-Assistant.md
@@ -73,6 +73,8 @@ Example: OpenAI GPT-4 Configured for Chat & RAG
 }
 ```
 
+For more details, see the **AI Search, Enterprise Edition** section: [Connecting to Enterprise GAI and LLMs](AI-Search#connecting-to-generative-ai-gai-and-large-language-models-llms).
+
 ## Launching Assistant
 
 Once the AI provider is configured correctly, **Assistant should be accessible via a browser**.  
@@ -81,9 +83,6 @@ For a default installation, go to: [http://localhost:8000/galaxy/chat](http://lo
 
 ![SWIRL Assistant discussion](images/swirl_40_chat_scop.png)
 
-For more details, see the **AI Search, Enterprise Edition** section: [Connecting to Enterprise GAI and LLMs](AI-Search#connecting-to-generative-ai-gai-and-large-language-models-llms).
-
-
 ## Listing Sources
 
 Instruct the Assistant to: `List sources`
@@ -112,6 +111,84 @@ Queries information on 7 million companies worldwide, including number of employ
 
 The `description` field can be up to 2k in length.
 
+## Using Prompts
+
+SWIRL Enterprise includes a set of pre-loaded, standard prompts. Each consiss of three key components:
+
+| Field | Description |
+| ----- | ----------- |
+| `prompt` | The main body of the prompt. Use `{query}` to represent the SWIRL query. |
+| `note` | Text appended to search result data sent to the LLM for insight generation. |
+| `footer` | Additional instructions appended after the prompt and RAG data. This is ideal for formatting guidance. |
+
+The name of the `prompt` has no importance. SWIRL uses the `tags` field to determine which prompt is used for a given function. 
+
+The following table presents the `tags` options:
+
+| Tag | LLM Role | 
+| --- | -------- | 
+| chat | Used by AI Search Assistant for chat conversations, including company background; not technical | 
+| chat-rag | Used by AI Search Assistant to answer questions and summarize data via RAG; somewhat technical | 
+| search-rag | Used by AI Search, `Generate AI Insight` (RAG) switch, somewhat technical | 
+
+Note that there must be at least one `active` prompt for each of these tags for the relevant SWIRL features to work.
+
+### Modifying the Standard Prompts
+
+{.warning}
+**Warning: never modify the standard prompts!** Any such changes will be discarded when SWIRL updates. 
+
+Use the [Customizing Prompts](#customizing-the-ai-search-assistant-prompts) procedure instead.
+
+### Customizing the AI Search Assistant Prompts
+
+The following procedure below below to copy the standard prompts, modify them, then make them active. 
+New prompts that you create won't be disturbed during upgrades.
+
+1. Open [http://localhost:8000/swirl/prompts/](http://localhost:8000/swirl/prompts/)
+
+2. Locate the `chat_rag_instructions_standard`, `chat_rag_standard` or `chat_rag_deeplink` prompts and note the `id`. 
+
+3. Add the `id` to the URL to view just that prompt. 
+
+4. Click the `Raw data` tab at the bottom of the page. Copy the entire JSON record to the clipboard. 
+
+5. Click the `HTML form` tab at the bottom of the page. Set `active` to `false`. Click `PUT` to save the change. 
+
+6. Go back to [http://localhost:8000/swirl/prompts/](http://localhost:8000/swirl/prompts/) and scroll to the bottom of the form. 
+
+7. Select the `Raw data` tab if necessary. Paste the prompt copied in step #4 into `Content:` block. **Do not hit `Put` yet**.
+
+8. Remove the `id`, `owner`, `date_created` and `date_updated` fields. Change the `name` field to something descriptive. Also, make sure `active` is set to `true`. Finally, if you don't wish to share this prompt with other users, set `shared` to `false`. Feel free to hit `PUT` now to save the record. 
+
+9. Modify the `prompt` field, and change `SWIRL Corporation` to the name of your organization. We also recommend adding adding 2-3 sentences describing the organization as well. immediately afterwards. 
+
+For example:
+
+```json
+{
+    "prompt": "You are an expert online assistant and reference librarian working for **<your-company-name>. <Your-company-name> is located in <description> and operates in <industry> etc**... Your job is...
+ }
+```
+
+{: .warning }
+**Warning:** Removing important sections of any prompt such as variables like `{header}` and `{query}` may cause AI Insight generation to fail or not contain important features such as follow-up questions or citations. 
+
+10. Hit `PUT`. This will save the prompt.
+
+![SWIRL Prompt Object](images/swirl_prompt_form.png)
+
+## Restoring Standard Prompts
+
+To go back to a standard prompt after creating a new one:
+
+1. Edit the new prompt and set `active` to `false`. 
+2. Edit the standard prompt and set `active` to `true`.
+
+### Restoring All Prompts to Default
+
+To restore all prompts to the default, refer to the [Admin Guide on Resetting Prompts](Admin-Guide.html#resetting-prompts).
+
 ## Advanced Querying
 
 To enable the Assistant to query in advanced query languages like SQL, the elastic API or Mongo MQL, add an LLM generated instruction set to this structure, including:
diff --git a/docs/AI-Search.md b/docs/AI-Search.md
@@ -284,60 +284,99 @@ This ensures automatic updates for AI provider credentials, reducing manual inte
 
 # Managing Prompts
 
-## Default Prompts
+SWIRL Enterprise includes a set of pre-loaded standard prompts used when Generating AI Insights via RAG. 
 
-{: .warning }
-**Warning:** If you edit a default system prompt, your changes will be lost when SWIRL is updated.
+Each consiss of three key components:
+
+| Field | Description |
+| ----- | ----------- |
+| `prompt` | The main body of the prompt. Use `{query}` to represent the SWIRL query. |
+| `note` | Text appended to search result data sent to the LLM for insight generation. |
+| `footer` | Additional instructions appended after the prompt and RAG data. This is ideal for formatting guidance. |
+
+The name of the `prompt` has no importance. SWIRL uses the `tags` field to determine which prompt is used for a given function. 
+
+The following table presents the `tags` options:
 
-View the default prompts at:  
-[http://localhost:8000/swirl/prompts/](http://localhost:8000/swirl/prompts/)
+| Tag | LLM Role | 
+| --- | -------- | 
+| search-rag | Used by AI Search, `Generate AI Insight` (RAG) switch, somewhat technical | 
+| chat | Used by AI Search Assistant for chat conversations, including company background; not technical | 
+| chat-rag | Used by AI Search Assistant to answer questions and summarize data via RAG; somewhat technical | 
 
-## Creating New Saved Prompts
+Note that there must be at least one `active` prompt for each of these tags for the relevant SWIRL features to work.
 
-To create a new saved prompt:
+### Modifying the Standard Prompts
+
+{.warning}
+**Warning: never modify the standard prompts!** Any such changes will be discarded when SWIRL updates. 
+
+Use the [Customizing Prompts](#customizing-the-ai-search-rag-prompt) procedure instead.
+
+### Customizing the AI Search RAG Prompt
+
+The following procedure below below to copy the standard prompts, modify them, then make them active. 
+New prompts that you create won't be disturbed during upgrades.
 
 1. Open [http://localhost:8000/swirl/prompts/](http://localhost:8000/swirl/prompts/)
-2. Use the form at the bottom of the page to create a new prompt **or** paste raw JSON and click `POST`.
 
-**Example: Creating a Pirate-Themed Prompt**
+2. Locate the `search_rag_standard` or, if using [Deep Linking](./User-Guide-Enterprise.md#deep-linked-citations) `search_rag_deeplink` prompt, and note the `id`. 
+
+3. Add the `id` to the URL to view just that prompt. 
+
+4. Click the `Raw data` tab at the bottom of the page. Copy the entire JSON record to the clipboard. 
+
+5. Click the `HTML form` tab at the bottom of the page. Set `active` to `false`. Click `PUT` to save the change. 
 
-Modify the default prompt to generate responses in pirate-speak:
+6. Go back to [http://localhost:8000/swirl/prompts/](http://localhost:8000/swirl/prompts/) and scroll to the bottom of the form. 
 
+7. Select the `Raw data` tab if necessary. Paste the prompt copied in step #4 into `Content:` block. **Do not hit `Put` yet**.
+
+8. Remove the `id`, `owner`, `date_created` and `date_updated` fields. Change the `name` field to something descriptive. Also, make sure `active` is set to `true`. Finally, if you don't wish to share this prompt with other users, set `shared` to `false`. Feel free to hit `PUT` now to save the record. 
+
+9. Modify the `prompt`, `note` and/or `footer` as needed, while retaining all critical instructions. For example, to instruct the LLM to use pirate-speak:
 ```json
 {
-    "name": "pirate",
+    "name": "search_rag_standard_pirate",
     "...": "... etc ...",
     "footer": "--- Final Instructions ---
 In your response, pretend you are a pirate comedian, but keep it clean!",
     "tags": ["search-rag"]
 }
 ```
 
-This will produce:
-
+10. Hit `PUT`. This will save the prompt.
 ![SWIRL Prompt Object](images/swirl_prompt_form.png)
 
-## Specifying a Saved Prompt in a Query
+{: .warning }
+**Warning:** Removing important sections of any prompt such as variables like `{header}` and `{query}` may cause AI Insight generation to fail or not contain important features such as follow-up questions or citations. 
+
+## Specifying a Saved Prompt when Generating AI Insights
+
+There are two ways to select a saved prompt: 
+
+1. Use the prompt selector on the AI Search form:
+![SWIRL AI Search Form w/Prompt Selector](./images/swirl_40_search_prompts.png)
 
-Test a saved prompt using the **prompt operator**:
+2. Use the **prompt operator** in your query. For example: 
 
 ```
 swirl AI Search prompt:pirate
 ```
 
-The response should be generated in pirate-speak:
-
+In either case, the response is generated in pirate-speak as the prompt instructs:
 ![SWIRL RAG response in pirate speak](images/swirl_prompt_pirate.png)
 
-## Understanding Saved Prompts
+## Restoring Standard Prompts
 
-SWIRL prompts consist of three key components:
+To go back to a standard prompt after creating a new one:
 
-| Field | Description |
-| ----- | ----------- |
-| `prompt` | The main body of the prompt. Use `{query}` to represent the SWIRL query. |
-| `note` | Text appended to RAG data chunks, annotated by the [Text Analyzer](#text-summarization). |
-| `footer` | Additional instructions appended after the prompt and RAG data. This is ideal for formatting guidance. |
+1. Edit the new prompt and set `active` to `false`. 
+2. Edit the standard prompt and set `active` to `true`.
+
+### Restoring All Prompts to Default
+
+To restore all prompts to the default, refer to the [Admin Guide on Resetting Prompts](https://docs.swirlaiconnect.com/Admin-Guide#resetting-prompts).
 
 ## Using a Prompt in a Query Processor or Connector
 
diff --git a/docs/Developer-Guide.md b/docs/Developer-Guide.md
@@ -200,11 +200,9 @@ To address this, use the **[DateFindingResultProcessor](#find-dates-in-bodytitle
 
 ## Use an LLM to Rewrite Queries
 
-SWIRL AI Search supports **query rewriting** using `ChatGPTQueryProcessor` (for the Community edition) or the `GenAIQueryProcessor` (for Enterprise).
+SWIRL AI Search supports **query rewriting** using an LLM. To set this up, add the `GenAIQueryProcessor` to some  `SearchProvider.query_processors` list. 
 
-To enable it, add the processor to `SearchProvider.query_processors`.  
-
-For details, see: [Developer Reference - Query Processors](./Developer-Reference#query-processors).
+For details, see: [Developer Reference - GenAIQueryProcessor](./Developer-Reference.md#genai-and-chatgpt-connectors)
 
 ## Adjust `swirl_score` for Starred Results in Galaxy UI
 
@@ -214,7 +212,7 @@ For details, see: [Developer Reference - Query Processors](./Developer-Reference
 
 **SWIRL Enterprise Edition  **
 - Configured via `"minimumConfidenceScore"` in `static/api/config/default`.
-- Default: `0.7`. Increase this to reduce starred results.
+- Default: `0.7`. Increase this to reduce the number of starred results and use only highly relevant results.
 
 ![SWIRL AI Search 4.0 Results](images/swirl_40_results.png)
 
diff --git a/docs/Developer-Reference.md b/docs/Developer-Reference.md
@@ -297,14 +297,13 @@ The included [BigQuery SearchProvider](https://github.com/swirlai/swirl-search/b
 
 More information: [BigQuery Documentation](https://cloud.google.com/bigquery/docs)
 
-## GenAI and ChatGPT Connectors
+## GenAI Connectors
 
-SWIRL includes connectors that can be used to obtain direct answers from LLMs, as if they were regular SearchProviders. The connector is the `ChatGPTConnector` (for the Community edition) or the `GenAIConnector` (for Enterprise).
-
-Both connectors return at most one result.
+SWIRL includes the `GenAIConnector` which can ask for direct answers from LLMs, as if they were regular SearchProviders. It returns at most one result.
 
 The included [SearchProvider](https://github.com/swirlai/swirl-search/blob/main/SearchProviders/gen_ai.json) is pre-configured with a "Tell me about: ..." prompt. 
 
+
 ```json
 {
     "name": "GenAI - OpenAI",
@@ -316,7 +315,7 @@ The included [SearchProvider](https://github.com/swirlai/swirl-search/blob/main/
     "query_processors": [
         "AdaptiveQueryProcessor"
     ],
-    "query_mappings": "",
+    "query_mappings": "PROMPT='Tell me about: {query_to_provider}'",
     "result_processors": [
         "GenericResultProcessor",
         "CosineRelevancyResultProcessor"
@@ -332,40 +331,37 @@ The included [SearchProvider](https://github.com/swirlai/swirl-search/blob/main/
 }
 ```
 
-The `Question` and `GenAI` SearchProvider Tags enable targeted queries. For example:
-
-```shell
-Question: Tell me about knowledge management software?
-```
-
 ### Setting the GenAI Prompt or Role
 
-The following tags adjust the prompt or role passed to GenAI. 
+The following `query_mappings` **including quotes** adjust the prompt or role passed to GenAI. 
 
 * `CHAT_QUERY_REWRITE_PROMPT`: Customizes the prompt used to refine the query.
 
 ```json
-"CHAT_QUERY_REWRITE_PROMPT:Write a more precise query of similar length to this: {query_string}"
+"CHAT_QUERY_REWRITE_PROMPT: Write a more precise query of similar length to this: {query_to_provider}"
 ```
 
 * `CHAT_QUERY_REWRITE_GUIDE`: Overrides the `system` role.
 
 ```json
-"CHAT_QUERY_REWRITE_GUIDE:You are a helpful assistant that responds like a pirate captain"
+"CHAT_QUERY_REWRITE_GUIDE: You are a helpful assistant that responds like a pirate captain"
 ```
 
 * `CHAT_QUERY_DO_FILTER`: Enables/disables internal filtering of LLM responses.
 
 ```json
-"CHAT_QUERY_DO_FILTER:false"
+"CHAT_QUERY_DO_FILTER: false"
 ```
 
-The following `query_mapping` adjusts the default role passed to GenAI:
+For example:
 
-```shell
-"query_mappings": "PROMPT='Tell me about: {query_to_provider}',CHAT_QUERY_REWRITE_GUIDE='You are a helpful assistant that responds like a pirate captain'",
+```json
+"query_mappings": "CHAT_QUERY_REWRITE_PROMPT='Tell me about: {query_to_provider}',CHAT_QUERY_REWRITE_GUIDE='You are a helpful assistant that talks like a pirate captain but keeps it clean!'",
 ```
 
+{.warning}
+Do not use commas inside of a prompt passed in this manner.
+
 # Elastic & OpenSearch
 
 The [Elastic](https://github.com/swirlai/swirl-search/blob/main/swirl/connectors/elastic.py) and [OpenSearch](https://github.com/swirlai/swirl-search/blob/main/swirl/connectors/opensearch.py) Connectors use each engine's Python client.
@@ -917,22 +913,15 @@ Query Processors modify queries. The field they operate on depends on where they
 | Processor | Description | Notes | 
 | ---------- | ---------- | ---------- | 
 | **AdaptiveQueryProcessor** | Rewrites queries based on `query_mappings` for a given SearchProvider | Not for `pre_query_processors` |
-| **ChatGPTQueryProcessor** | Uses OpenAI to rewrite queries, making them broader, more specific, boolean, or in another language | Community edition only |
-| **GenAIQueryProcessor** | Uses an LLM to rewrite queries, making them broader, more specific, boolean, or in another language | Enterprise edition only |
+| **GenAIQueryProcessor** | Uses an LLM to rewrite queries, making them broader, more specific, boolean, or in another language |  |
 | **GenericQueryProcessor** | Removes special characters from queries |  |
 | **SpellcheckQueryProcessor** | Uses [TextBlob](https://textblob.readthedocs.io/en/dev/quickstart.html#spelling-correction) to correct spelling errors | Best used in `SearchProvider.query_processors`; avoid with Google PSE |
 | **NoModQueryProcessor** | Removes only leading SearchProvider Tags, leaving query terms unchanged | For repositories allowing non-search characters (e.g., brackets) |
 | **RemovePIIQueryProcessor** | Removes PII entities from queries without replacing them |  |
 
 ## GenAIQueryProcessor
 
-The `GenAIQueryProcessor` calls an LLM with the user's query and a custom prompt to enable rewriting. For example this processor can:
-
-* Translate queries between languages
-* Rewrite queries as booleans
-* Perform phrase detection
-
-And much more. 
+The `GenAIQueryProcessor` calls an LLM with the user's query and a custom prompt to enable rewriting, translation phrase detection and much more. 
 
 ### Configuring the SearchProvider
 
@@ -954,18 +943,20 @@ To alter the query sent to all Searchproviders, add it to the `search.pre_query_
 }
 ```
 
-Note: this requires calling the Swirl Search API and is not supported via Galaxy.
+Note that the example above only applies to calling the SWIRL API. Please [contact support](#support) for instructions on how to configure this behavior with the Galaxy UI. 
 
 ### Configuring the QueryProcessor
 
-The GenAIQueryProcessor supports the same tags and configuration options as the [GenAIConnector](#setting-the-genai-prompt-or-role).
+The GenAIQueryProcessor supports the same tags and configuration options as the [GenAIConnector](#setting-the-genai-prompt-or-role) but using a colon (':') as a delimeter.
 
 For example:
 
 ```json
 "tags": [
-            "PROMPT='Translate this query to japanese, if necessary: {query_string}"
-        ],
+    "CHAT_QUERY_REWRITE_PROMPT:If this query is in Japanese, tokenize it, and output ONLY the tokenized japanese, NOTHING ELSE -> {query_string}",
+    "CHAT_QUERY_REWRITE_GUIDE:You are a speedy tokenizer!",
+    "CHAT_QUERY_DO_FILTER:False"
+],
 ```
 
 # Result Processors
diff --git a/docs/User-Guide-Enterprise.md b/docs/User-Guide-Enterprise.md
