Regarding summarisation for title generation

[QuietRocket]

First off, I'd like to say great job on the featv2 revamp! And I like how it's now its own separate repository. Great work 👏

While reading the code, I noticed there's a functionality to update the conversation title based on the prompt.
I find this feature to be pretty nifty, but wouldn't it be good to make it optional? I just glazed over the repo, but it seems to me like if I use GPT-4, the summary will be made via GPT-4. This consists of its own request copying the prompt and possibly the entire conversation history. It seems like it will chew through API credits pretty quickly.

If I understood this situation correctly, and this indeed could be an issue, there could be two possible fixes:

1. Make a toggle for whether this system should be put in place.
2. For each request, have GPT-4 generate a structured or segmented output containing a small summary / title in the end. That way, only a single request is used for the entire process.

Looking forward to your response,

QuietRocket

[ddiu8081]

Thank you for your message!

There are several points to note about title-summarisation:

1. It will always use gpt-3.5-turbo, no matter what model you have selected. It uses a separate set of parameters that are not affected by the parameters in the settings panel.
   

   anse/src/providers/openai/handler.ts

   Lines 34 to 53 in 4dc4029

   	export const handleRapidPrompt: Provider['handleRapidPrompt'] = async(prompt, globalSettings) => {
   	const rapidPromptPayload = {
   	conversationId: 'temp',
   	globalSettings: {
   	...globalSettings,
   	model: 'gpt-3.5-turbo',
   	temperature: 0.4,
   	maxTokens: 2048,
   	top_p: 1,
   	stream: false,
   	},
   	conversationSettings: {},
   	systemRole: '',
   	mockMessages: [],
   	} as HandlerPayload
   	const result = await handleChatCompletion([{ role: 'user', content: prompt }], rapidPromptPayload)
   	if (typeof result === 'string')
   	return result
   	return ''
   	}

2. The content used to generate the summary includes: system info (not yet implemented), and the first prompt sent by user.
3. It is generated in continuous conversations where the title is not set manually, and is only operated once in the first round.

Making the feature optional is great and I will put it into roadmap.