Next.js + Vercel AI SDK + Langfuse: How to get observationId (responseId) for langfuse scores?

[natterstefan]

Dear vercel/ai-Community,

I need your help please, as I do not know how to obtain the observationId and attach it to a score in langfuse.

Note: originally asked in the langfuse community, but they sent me here with the following hint: "(...) you'd need the SDK to expose the span ids of each of the requests." (src)

---

Context: this is how my experimental_telemetry looks like:

// src: https://langfuse.com/docs/integrations/vercel-ai-sdk#group-multiple-executions-in-one-trace

const parentTraceId = getRandomId()

export async function POST(req: Request) {
	// ... other code

	const trace = langfuse.trace({
	  id: parentTraceId,
	  name: 'cd-chat-messages',
	})

	console.log('trace', trace.observationId) // <<-- this returns `null`
	
	const result = streamText({
		// ....
		experimental_telemetry: {
		    isEnabled: true,
		    functionId: 'cd-chat-message-' + messages.length,
		    metadata: {
		      langfuseTraceId: parentTraceId,
			  // Do not update the parent trace with execution results
		      langfuseUpdateParent: false,
		    },
		  },
	})

	// ... other code
}

The docs state

The id of the observation to which the score should be attached. Pass from backend trace. E.g. {generation, event, span}.id

But how and where do I get the observationId from, taking into account that I use streamText (https://sdk.vercel.ai/docs/reference/ai-sdk-core/stream-text)

// source: https://langfuse.com/docs/sdk/typescript/guide-web#create-scores

// pass traceId and observationId to front end
await langfuseWeb.score({
  traceId: message.traceId,
  observationId: message.observationId, <<<-- where do I get this?
  name: "user-feedback",
  value: 1,
  comment: "I like how personalized the response is",
});

Thank you very much for your time and help!

[Josh-Zirena]

I'm having the same issue as OP. We should be able to tie this into an observation ID that we want a score for. Eg. The observation/span ID from ai.streamText.doStream

[natterstefan]

Haha, what timing, I just checked to see if there was an answer yet. Unfortunately not. Thanks for the support in the sense of ‘Hey, I have that problem too’. Let's see how we can resolve this together and with the help of the community.

[Josh-Zirena]

One possible work-around that I have right now is creating a trace.generation object, taking it's ID and passing that to dataStream.writeMessageAnnotation. Extracting that ID from messages, storing that in my global storage and then using that trace.generation ID in my feedback component so that a score gets attached to it.

It's not a clean work-around, but you'll be able to have feedback on a trace.generation which happens right before the span. So for example, doing it as I described you'd end up with something like this - which is not ideal - but it's better than nothing.

• Trace
  • Generation [the one you created]
  • Span [ai.streamText]
    • Generation [ai.streamText.doStream]

Code would look something like:

export async function POST(req: Request) {
  // ... other code
  const lf = new Langfuse({
    secretKey: 'your secret key',
    baseUrl: 'your base url'
  })
  const trace = lf.trace({id, name: 'chat-session'})
  const gen = trace.generation({name: 'chat-completion', model, input: messages})
  return createDataStreamResponse({
    execute: async (dataStream) => {
      dataStream.writeMessageAnnotation({}
        id: gen.id,
        type: "observationId" // I named it this so I can separate this from my other annotations (it might not be necessary for others)
      )
      const result = streamText({
        model: 'your model',
        system: 'your system prompt'
        // ... other code
        experimental_telemetry: {
          isEnabled: true,
          metadata: { langfuseTraceId: id } // I pulled my id from my request
        }
        onFinish: async () => {
          await gen.end({output: await result.text})
          // ... other code
        }
      })
    }
  })
}

Hope that might be a little helpful for your use case.

[natterstefan]

HI @Josh-Zirena,

I did some more research and analysed the logs. This is what I came up so far:

I've played with the registerOTel settings and found idGenerator (Docs). It's the one generating the ids we later see in langfuse. That's why I though "what happens if I customize them?". Here's the answer:

// instrumentation.ts

registerOTel({
    serviceName: 'your-app',
    idGenerator: {
      // default: random 16 chars
      generateTraceId: () => uuidv4(),
      // default: random 16 chars
      generateSpanId: () => uuidv4(),
    },
    traceExporter: new LangfuseExporter({
    	debug: true
    })
 })

If you then send a message to the API the logs with debugging enabled in LangfuseExporter report something like this (besides all the other context):

[1]     name: 'ai.streamText.doStream',
[1]     _spanContext: {
[1]       traceId: '875566be-5491-40ad-a842-648dcc279578',
[1]       spanId: 'bde24164-3729-4627-b5a5-3e4a00c7641f',
[1]       traceFlags: 1,
[1]       traceState: undefined
[1]     },
[1]     parentSpanId: undefined,

This ids and (some others visible in the debug logs) are then used here

• https://github.com/langfuse/langfuse-js/blob/9bf9b8126b264abc66733052f958427df57ab287/langfuse-vercel/src/LangfuseExporter.ts#L115-L143
• https://github.com/langfuse/langfuse-js/blob/9bf9b8126b264abc66733052f958427df57ab287/langfuse-vercel/src/LangfuseExporter.ts#L144-L157

This id bde24164-3729-4627-b5a5-3e4a00c7641f is then the one used in the langfuse observation:

The problem, as you can see in the screenshot) is that the generation is no child of the span anymore:

Span [cd-chat-message-75]
Generation [ai.streamText.doStream]

instead of

- Span [cd-chat-message-75]
  - Generation [ai.streamText.doStream]

Which is what I still would expect to have based on the given langfuse data model (but it's not a requirement according to the docs).

Besides that the real and still existing problem is how to access the spanId: 'bde24164-3729-4627-b5a5-3e4a00c7641f', and use it in the frontend to associate the score with the generation (aka observationId). I have not yet figured out how to do that.

Using this exposed a different spanId and name. So I assume that's the wrong path (or I miss something).

import { trace, context, type Span } from '@opentelemetry/api'

// e.g. in `onFinish` of `streamText`
const activeSpan: Span | undefined = trace.getSpan(context.active())
const activeSpanContext = activeSpan?.spanContext()
console.log('activeSpanContext', activeSpanContext)

This is what I get

[1] activeSpanContext {
[1]   traceId: '09f31b10-4ecc-4384-92df-4339ea164a73',
[1]   spanId: 'ccbf35e2-0a48-4db9-9c6e-08a417355278',
[1]   traceFlags: 1,
[1]   traceState: undefined
[1] }

Which corresponds to the following debug output of LangfuseExporter:

 name: 'executing api route (app) /(chat)/api/chat/route',
[1]     _spanContext: {
[1]       traceId: '09f31b10-4ecc-4384-92df-4339ea164a73',
[1]       spanId: 'ccbf35e2-0a48-4db9-9c6e-08a417355278',
[1]       traceFlags: 1,
[1]       traceState: undefined
[1]     },
[1]     parentSpanId: undefined,

Maybe I am on the right track but there's something I do not see. 🤔

Further more Gemini pointed out that it is difficult to access all active spans within the application code aka the request were are working with, see also open-telemetry/opentelemetry-js-api#167 and https://stackoverflow.com/questions/72973140/get-all-child-spans-associated-with-opentelemetry-trace/73058566 ...

---

Btw: your solution is similar to mine, but I send the manually created traceId of the trace within the annotations (or data both works) and use that without creating another generation. This is based on the following docs: https://langfuse.com/docs/integrations/vercel-ai-sdk#group-multiple-executions-in-one-trace

[OzzieOrca]

I think I got something to work. See https://github.com/orgs/langfuse/discussions/6210#discussioncomment-14312688.