CosyVoice2 requires prompt_speech

[peterliang5678]

System Info / 系統信息

cuda12.4 py3.10

CosyVoice2-0.5B

Traceback (most recent call last):
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/api/restful_api.py", line 1539, in create_speech
out = await model.speech(
^^^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 231, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 667, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 558, in on_receive
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 102, in wrapped_func
ret = await fn(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xoscar/api.py", line 462, in _wrapper
r = await func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/utils.py", line 94, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 916, in speech
return await self._call_wrapper_binary(
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 563, in _call_wrapper_binary
return await self._call_wrapper("binary", fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 131, in _async_wrapper
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 585, in _call_wrapper
ret = await asyncio.to_thread(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/home/zhdk_admin/miniconda3/envs/xinference/lib/python3.11/site-packages/xinference/model/audio/cosyvoice.py", line 187, in speech
assert prompt_speech is not None, "CosyVoice2 requires prompt_speech"
^^^^^^^^^^^^^^^^^
AssertionError: [address=0.0.0.0:41723, pid=3177802] CosyVoice2 requires prompt_speech

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

Release: v1.1.1

The command used to start Xinference / 用以启动 xinference 的命令

ui启动

Reproduction / 复现过程

dify 调用文字转语音，和 客户端使用 from xinference.client import Client

client = Client("http://10.10.227.10:9997")

model = client.get_model("CosyVoice2-0.5B")
speech_bytes = model.speech(
input="你好朋友",
voice="中文女"
)
with open('1.mp3', 'wb') as f:
f.write(speech_bytes) 都报错 ，官方找不到其他例子

Expected behavior / 期待表现

其他正常

[qinxuye]

目前 CosyVoice2 只能提供 prompt_speech 用于语音克隆。

@codingl2k1 这个有用法上的更新吗？

[peterliang5678]

目前 CosyVoice2 只能提供 prompt_speech 用于语音克隆。

@codingl2k1 这个有用法上的更新吗？

xinference 中 如果启动 这个 传 prompt_speech 怎么写命令？就是不知道怎么写

[qinxuye]

https://inference.readthedocs.io/zh-cn/latest/models/model_abilities/audio.html#cosyvoice-usage

可以参考 cosyvoice 一代的克隆声音的用法。

[qinxuye]

此外，我们在考虑统一提供 voice 的选项。

这样不需要 prompt_speech 也可以直接合成语音。

[peterliang5678]

https://inference.readthedocs.io/zh-cn/latest/models/model_abilities/audio.html#cosyvoice-usage

可以参考 cosyvoice 一代的克隆声音的用法。

传prompt_speech 可以正常调用了，但是dify框架调用 xinference的 cosyvoice 没有地方传prompt_speech 代码例子可以，建议修改

[qinxuye]

此外，我们在考虑统一提供 voice 的选项。

这样不需要 prompt_speech 也可以直接合成语音。

@peterliang5678 正在考虑按这个方式修改，修改后就不需要传入 prompt_speech。