Skip to content

Implement audio chat provider #227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
julien-nc merged 11 commits into from
Jul 10, 2025
Merged

Implement audio chat provider #227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f178063
implement audio chat provider
julien-nc Jul 1, 2025
6334cae
adjust AudioToAudioChatTaskType
julien-nc Jul 3, 2025
e3a6d2d
feat(audio-chat): add condition to register provider and task type
julien-nc Jul 3, 2025
b875987
review adjustments
julien-nc Jul 3, 2025
2054ea6
fix tests
julien-nc Jul 3, 2025
456a8a1
change defaults and enum values if using openai or not, use llmModel …
julien-nc Jul 4, 2025
f00717e
use service name in logs instead of hardcoded value, fall back to app ID
julien-nc Jul 4, 2025
47b9c86
remove fallback audio chat task type, register the provider only if t…
julien-nc Jul 7, 2025
42c644c
isolate oneStep audio2audio processing in a provider method
julien-nc Jul 9, 2025
cd67b7f
try to return the remote audio ID when using the chat endpoint with a…
julien-nc Jul 9, 2025
3ea9b77
return remote audio expiration date
julien-nc Jul 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions lib/AppInfo/Application.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

use OCA\OpenAi\Capabilities;
use OCA\OpenAi\OldProcessing\Translation\TranslationProvider as OldTranslationProvider;
use OCA\OpenAi\TaskProcessing\AudioToAudioChatProvider;
use OCA\OpenAi\TaskProcessing\AudioToTextProvider;
use OCA\OpenAi\TaskProcessing\ChangeToneProvider;
use OCA\OpenAi\TaskProcessing\ChangeToneTaskType;
Expand Down Expand Up @@ -131,6 +132,22 @@ public function register(IRegistrationContext $context): void {
$context->registerTaskProcessingProvider(TextToImageProvider::class);
}

// only register audio chat stuff if we're using OpenAI or stt+llm+tts are enabled
$serviceUrl = $this->appConfig->getValueString(Application::APP_ID, 'url');
$isUsingOpenAI = $serviceUrl === '' || $serviceUrl === Application::OPENAI_API_BASE_URL;
if (
$isUsingOpenAI
|| (
$this->appConfig->getValueString(Application::APP_ID, 'stt_provider_enabled', '1') === '1'
&& $this->appConfig->getValueString(Application::APP_ID, 'llm_provider_enabled', '1') === '1'
&& $this->appConfig->getValueString(Application::APP_ID, 'tts_provider_enabled', '1') === '1'
)
) {
if (class_exists('OCP\\TaskProcessing\\TaskTypes\\AudioToAudioChat')) {
$context->registerTaskProcessingProvider(AudioToAudioChatProvider::class);
}
}

$context->registerCapability(Capabilities::class);
}

Expand Down
28 changes: 25 additions & 3 deletions lib/Service/OpenAiAPIService.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,8 @@ public function createCompletion(
* @param array|null $extraParams
* @param string|null $toolMessage JSON string with role, content, tool_call_id
* @param array|null $tools
* @return array{messages: array<string>, tool_calls: array<string>}
* @param string|null $userAudioPromptBase64
* @return array{messages: array<string>, tool_calls: array<string>, audio_messages: list<array<string, mixed>>}
* @throws Exception
*/
public function createChatCompletion(
Expand All @@ -451,6 +452,7 @@ public function createChatCompletion(
?array $extraParams = null,
?string $toolMessage = null,
?array $tools = null,
?string $userAudioPromptBase64 = null,
): array {
if ($this->isQuotaExceeded($userId, Application::QUOTA_TYPE_TEXT)) {
throw new Exception($this->l10n->t('Text generation quota exceeded'), Http::STATUS_TOO_MANY_REQUESTS);
Expand Down Expand Up @@ -494,8 +496,24 @@ public function createChatCompletion(
$messages[] = $message;
}
}
if ($userPrompt !== null) {
$messages[] = ['role' => 'user', 'content' => $userPrompt];
if ($userPrompt !== null || $userAudioPromptBase64 !== null) {
$message = ['role' => 'user', 'content' => []];
if ($userPrompt !== null) {
$message['content'][] = [
'type' => 'text',
'text' => $userPrompt,
];
}
if ($userAudioPromptBase64 !== null) {
$message['content'][] = [
'type' => 'input_audio',
'input_audio' => [
'data' => $userAudioPromptBase64,
'format' => 'mp3',
],
];
}
$messages[] = $message;
}
if ($toolMessage !== null) {
$msgs = json_decode($toolMessage, true);
Expand Down Expand Up @@ -555,6 +573,7 @@ public function createChatCompletion(
$completions = [
'messages' => [],
'tool_calls' => [],
'audio_messages' => [],
];

foreach ($response['choices'] as $choice) {
Expand Down Expand Up @@ -583,6 +602,9 @@ public function createChatCompletion(
if (isset($choice['message']['content']) && is_string($choice['message']['content'])) {
$completions['messages'][] = $choice['message']['content'];
}
if (isset($choice['message']['audio'], $choice['message']['audio']['data']) && is_string($choice['message']['audio']['data'])) {
$completions['audio_messages'][] = $choice['message'];
}
}

return $completions;
Expand Down
Loading
Loading