How to access the raw response from a provider in a stream without reconstructing it from a fullStream?

[Und3rf10w]

I'm trying to get the raw response from the model provider in both streaming and non streaming usecases. An example usecase would be to do something like implementing an API mirror for the provider.

I've tried a few methods using streamObject and streamText, and streamObject's partialObjectStream.

I would like to get a response object that matches whatever the provider's response is.

I COULD manually process the fullStream and make each event match the schema, but I feel like there should be a way to access the original stream/response from the model provider. Personally, I wouldn't even mind if it was a duplex stream or something.

So as an example, in an OpenAI stream, I might get a response like:

{
  "id": "chatcmpl-xxxxxxxxx",
  "object": "chat.completion.chunk",
  "created": 1719518600,
  "model": "gpt-3.5-turbo",
  "system_fingerprint": null,
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "?"
      },
      "finish_reason": null
    }
  ]
}

Using streamText

By doing something like:

const stream = await streamText({
  model: openai(model, {}),
  messages}
})

const webStream = new ReadableStream({
  async start (controller) {
    const reader = stream.fullStream.getReader()
    const encoder = new TextEncoder()
    const decoder = new TextDecoder('utf-8')
    while (true) {
      const { done, value } = await reader.read()
      if (done) {
        controller.enqueue(encoder.encode('data: [DONE]\n\n'))
        break
      }
      controller.enqueue(encoder.encode(`data: ${JSON.stringify(value)}\n\n`))
    }
    controller.close()
  }
})

return new Response(webStream, {
  status: 200,
  headers: {
  ...stream.rawResponse?.headers
  }
})

I can get a response with:

{
  "type": "text-delta",
  "textDelta": "!"
}

Using streamObject

By using something like:

const stream = await streamObject({
  model: openai(model, {}),
  schema: openAIChatResponseSchema,  // manually defined based on existing openAIChatResponseSchema from the ai-sdk provider
  messages,
  mode: 'auto'
  }
})

const webStream = new ReadableStream({
  async start (controller) {
    const reader = stream.partialObjectStream.getReader()
    const encoder = new TextEncoder()
    const decoder = new TextDecoder('utf-8')
    while (true) {
      const { done, value } = await reader.read()
      if (done) {
        controller.enqueue(encoder.encode('data: [DONE]\n\n'))
        break
      }
      if (Object.keys(value).length !== 0 && value.choices?.[0]?.delta?.content) {
              controller.enqueue(encoder.encode(`data: ${JSON.stringify(value)}\n\n`))
      }
    }
    controller.close()
  }
})

return new Response(webStream, {
  status: 200,
  headers: {
  ...stream.rawResponse?.headers
  }
})

I can get a response with:

{
  "choices": [
    {
      "message": {
        "role": "system",
        "content": "Hello"
      }
    }
  ]
}

If you change stream.partialObjectStream.getReader() to stream.fullStream.getReader(), you could pull more values in, but have to parse both responses of "type": "finish" and "type": "text-delta", which as I mentioned above, you'd have to manually implement logic to parse for each provider.

So ultimately, my question is:

Is there a way to access the raw response or stream from a provider using the ai-sdk post 3.0.20 without relying on deprecated functions? (e.g. OpenAIStream). I have something working already which uses OpenAIStream, but I want to migrate away from deprecated functions.

[lgrammel]

One option would be to intercept the fetch call and replaying there. The openai provider allows you to use a custom fetch function.

[Und3rf10w]

I ended up doing something like the below. Unfortunately, I couldn't figure out how to use the fetch to actually access the response:

export async function chatCompletionApi(reqData: ChatCompletionRequest): Promise<Response> {
  const openai = await getLanguageModel(reqData.modelName)

  /** <snip.../> */

  if (reqData.stream) {
    const stream = streamText({
      model: openai,
      /** <snip.../> */
    })
    
    const readable = new ReadableStream({
      async start(controller) {
        try {
          for await (const chunk of stream.textStream) {
            controller.enqueue(chunk)
          }
          controller.close()
        } catch (error) {
          controller.error(error)
        }
      }
    })

    const messageId = generateId()

    // Create transform stream to format as SSE
    const transformer = new TransformStream({
      async transform(chunk, controller) {
        const encoder = new TextEncoder()
        // Create a ChatCompletionChunk object
        const completionChunk: ChatCompletionChunk = {
          id: messageId, // or use a consistent ID across chunks
          choices: [{
            delta: {
              content: chunk,
            },
            finish_reason: null,
            index: 0
          }],
          created: Math.floor(Date.now() / 1000),
          model: reqData.modelName,
          object: 'chat.completion.chunk'
        }

        controller.enqueue(encoder.encode(`data: ${JSON.stringify(completionChunk)}\n\n`))
      },
      async flush(controller) {
        const encoder = new TextEncoder()
        const completionChunk: ChatCompletionChunk = {
          id: messageId, // or use a consistent ID across chunks
          choices: [{
            delta: {
              content: null,
            },
            finish_reason: 'stop',
            index: 0
          }],
          created: Math.floor(Date.now() / 1000),
          model: modelName,
          object: 'chat.completion.chunk'
        }
        controller.enqueue(encoder.encode(`data: ${JSON.stringify(completionChunk)}\n\n`))
        controller.enqueue(encoder.encode('data: [DONE]\n\n'))
      }
    })
    return new Response(readable.pipeThrough(transformer), {
      headers: {
        'Content-Type': 'text/event-stream',
        Connection: 'keep-alive',
        'X-Trace-Id': traceClient.id,
      }
    })
  }
  /** <snip.../> */
}