Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 54 additions & 1 deletion src/langsmith/trace-with-pipecat.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Trace your [Pipecat](https://pipecat.ai/) voice agents to LangSmith with the Lan
The Pipecat integration requires `langsmith[pipecat]>=0.9.7`.
</Note>

The integration hooks into the spans Pipecat already emits and maps them onto LangSmith's tracing format, so each conversation becomes a single LangSmith trace, with a span per pipeline stage (STT, LLM, TTS).
The integration hooks into the spans Pipecat already emits and maps them onto LangSmith's tracing format, so each conversation becomes a single LangSmith trace, with a span per pipeline stage (STT, LLM, TTS). This covers both the STT/LLM/TTS cascade and speech-to-speech (realtime) models. Realtime models (for example, `OpenAIRealtimeLLMService`) need one extra call to capture the user's transcript. See [When using Pipecat with a realtime model](#when-using-pipecat-with-a-realtime-model).

## Install

Expand Down Expand Up @@ -95,6 +95,59 @@ configure_pipecat()
set_thread_id(conversation_id)
```

## When using Pipecat with a realtime model

With a speech-to-speech (realtime) model there is no separate speech-to-text stage, so the user's transcript is never emitted as an OTel span. Instead it arrives through the user context aggregator's `on_user_turn_message_added` callback, which Pipecat fires once it has the finalized user text. Without wiring it up, the trace shows only the assistant side.

Call `instrument_user_aggregator` once, right after building the context aggregator, so the SDK subscribes to that callback for you and pairs each transcript with its turn. It correlates by the id you pass to `set_thread_id`, so set that first and pass the same id:

<Note>
`instrument_user_aggregator` requires `langsmith[pipecat]>=0.10.6`.
</Note>

```python
from langsmith.integrations.pipecat import configure_pipecat, set_thread_id
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
from pipecat.services.openai.realtime.events import (
AudioConfiguration,
AudioInput,
AudioOutput,
InputAudioTranscription,
SessionProperties,
)
from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService

conversation_id = "..." # any id that identifies the conversation
span_processor = configure_pipecat()
set_thread_id(conversation_id)

llm = OpenAIRealtimeLLMService(
api_key=openai_api_key,
settings=OpenAIRealtimeLLMService.Settings(
model="gpt-realtime",
session_properties=SessionProperties(
# Enable input-audio transcription: OpenAI Realtime sends the model
# raw audio and, by default, produces no user-side text. Without this
# the context aggregator never fires, so the user's turns never reach
# the trace.
audio=AudioConfiguration(
input=AudioInput(transcription=InputAudioTranscription()),
output=AudioOutput(voice="marin"),
),
),
),
)

context = LLMContext(messages=[{"role": "system", "content": "..."}])
context_aggregator = LLMContextAggregatorPair(context, realtime_service_mode=True)
span_processor.instrument_user_aggregator(context_aggregator, conversation_id) # capture the user transcript
```

For OpenAI Realtime, also enable input-audio transcription (`InputAudioTranscription`) on the session; otherwise the model receives raw audio and produces no user-side text, so the aggregator never fires. Other realtime services that surface the user's text through the aggregator themselves (for example, Gemini Live) need only the `instrument_user_aggregator` call, with no extra session configuration.

Only call `instrument_user_aggregator` for realtime models. In the STT/LLM/TTS cascade the transcript is already captured (from the speech-to-text stage), so calling it there would record the user's turns a second time.

## Record the conversation audio

Attach the conversation audio to the trace using Pipecat's [`AudioBufferProcessor`](https://docs.pipecat.ai/server/utilities/audio/audio-recording). Place it after `transport.output()` so it captures what was actually played (after any barge-in truncation), hand it to the integration, and start it once the session is running:
Expand Down
Loading