diff --git a/src/langsmith/trace-with-pipecat.mdx b/src/langsmith/trace-with-pipecat.mdx index 0058a1d421..f50af9572e 100644 --- a/src/langsmith/trace-with-pipecat.mdx +++ b/src/langsmith/trace-with-pipecat.mdx @@ -14,7 +14,7 @@ Trace your [Pipecat](https://pipecat.ai/) voice agents to LangSmith with the Lan The Pipecat integration requires `langsmith[pipecat]>=0.9.7`. -The integration hooks into the spans Pipecat already emits and maps them onto LangSmith's tracing format, so each conversation becomes a single LangSmith trace, with a span per pipeline stage (STT, LLM, TTS). +The integration hooks into the spans Pipecat already emits and maps them onto LangSmith's tracing format, so each conversation becomes a single LangSmith trace, with a span per pipeline stage (STT, LLM, TTS). This covers both the STT/LLM/TTS cascade and speech-to-speech (realtime) models. Realtime models (for example, `OpenAIRealtimeLLMService`) need one extra call to capture the user's transcript. See [When using Pipecat with a realtime model](#when-using-pipecat-with-a-realtime-model). ## Install @@ -95,6 +95,59 @@ configure_pipecat() set_thread_id(conversation_id) ``` +## When using Pipecat with a realtime model + +With a speech-to-speech (realtime) model there is no separate speech-to-text stage, so the user's transcript is never emitted as an OTel span. Instead it arrives through the user context aggregator's `on_user_turn_message_added` callback, which Pipecat fires once it has the finalized user text. Without wiring it up, the trace shows only the assistant side. + +Call `instrument_user_aggregator` once, right after building the context aggregator, so the SDK subscribes to that callback for you and pairs each transcript with its turn. It correlates by the id you pass to `set_thread_id`, so set that first and pass the same id: + + +`instrument_user_aggregator` requires `langsmith[pipecat]>=0.10.6`. + + +```python +from langsmith.integrations.pipecat import configure_pipecat, set_thread_id +from pipecat.processors.aggregators.llm_context import LLMContext +from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair +from pipecat.services.openai.realtime.events import ( + AudioConfiguration, + AudioInput, + AudioOutput, + InputAudioTranscription, + SessionProperties, +) +from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService + +conversation_id = "..." # any id that identifies the conversation +span_processor = configure_pipecat() +set_thread_id(conversation_id) + +llm = OpenAIRealtimeLLMService( + api_key=openai_api_key, + settings=OpenAIRealtimeLLMService.Settings( + model="gpt-realtime", + session_properties=SessionProperties( + # Enable input-audio transcription: OpenAI Realtime sends the model + # raw audio and, by default, produces no user-side text. Without this + # the context aggregator never fires, so the user's turns never reach + # the trace. + audio=AudioConfiguration( + input=AudioInput(transcription=InputAudioTranscription()), + output=AudioOutput(voice="marin"), + ), + ), + ), +) + +context = LLMContext(messages=[{"role": "system", "content": "..."}]) +context_aggregator = LLMContextAggregatorPair(context, realtime_service_mode=True) +span_processor.instrument_user_aggregator(context_aggregator, conversation_id) # capture the user transcript +``` + +For OpenAI Realtime, also enable input-audio transcription (`InputAudioTranscription`) on the session; otherwise the model receives raw audio and produces no user-side text, so the aggregator never fires. Other realtime services that surface the user's text through the aggregator themselves (for example, Gemini Live) need only the `instrument_user_aggregator` call, with no extra session configuration. + +Only call `instrument_user_aggregator` for realtime models. In the STT/LLM/TTS cascade the transcript is already captured (from the speech-to-text stage), so calling it there would record the user's turns a second time. + ## Record the conversation audio Attach the conversation audio to the trace using Pipecat's [`AudioBufferProcessor`](https://docs.pipecat.ai/server/utilities/audio/audio-recording). Place it after `transport.output()` so it captures what was actually played (after any barge-in truncation), hand it to the integration, and start it once the session is running: