How to send one message with useChat, but get back distinct message streams from multiple model calls?

[tomsseisums]

I'm looking to make a workflow where user interfaces directly with one chat interface and they get responses back there. But I'd like to get a different response from the same chat session, to be presented in another UI layer.

Idea

Given the UI (CSS Grid Template notation):

main designer
main designer
main product_owner
main product_owner

Where:

• main is the chat assistant interface
• designer is a response from "designer"
• product owner is a response from "product owner"

The idea would be, that the user interfaces strictly with main, sending messages to there.

And in the backend, main essentially is an orchestrator set up to decide what the query is about, and invoking designer and product owner as tools.

designer and product owner themselves should be streamed content.

Attempt

Following I've outlined my attempt, which provides an unexpected output. Most likely I'm doing something wrong.

I've excluded product_owner part here for brevity, as the issue comes up with just getting the one extra stream working properly.

Backend

export async function POST(req: Request) {
    const { messages }: { messages: TeamUIMessage[] } = await req.json();

    return createUIMessageStreamResponse({
       stream: createUIMessageStream({
         originalMessages: messages,
         stopWhen: opts => opts.steps.some(step => step.finishReason === "stop"),

         async execute ({ writer }) {
           const mainResult = streamText({
             /* ... model settings ... */
             messages,
             tools: {
               getDesign: tool({
                 /* ... tool setup ... */
                 execute: async ({ prompt }) {
                   const designerResult = streamText({
                     /* ... model settings ... */
                     prompt
                   })
                   
                   // Add messages for streaming, adding differentiating
                   writer.merge(
                     designerResult.toUIMessageStream({
                       generateMessageId: generateId, // This is a completely new message
                       messageMetadata (options) {
                         return { stream: "designer" }
                       }
                     })
                   )

                   // Return response back to main model
                   return await packagingResult.text
                 }
               })
             }
           })
           
           // Pass the main stream, adding differentiating metadata
           writer.merge(
             mainResult.toUIMessageStream({
               messageMetadata (options) {
                 return { stream: "main" }
               }
             })
           )
         }
       })
    })
}

The main pipelining works, both streams get merged and all is good on that part.

Frontend

The issue seems to be in the Frontend layer.

Any attempts to differentiate the conversations, produce the same unexpected outcome of :

1. designer message starting on the right Conversation column, but after some time it gets moved to the main Conversation column on the left.
2. Parts also get duplicated.

export default function Chat() {
  const [input, setInput] = useState("");
  const { messages, sendMessage, status } = useChat<TeamUIMessage>();

  const handleSubmit = (e: React.FormEvent) => {
    e.preventDefault();
    if (input.trim()) {
      sendMessage({ text: input });
      setInput("");
    }
  };

  return (
    <div className="max-w-4xl mx-auto p-6 relative size-full h-screen">
      <div className="flex flex-col h-full">
        <div className="flex-1 flex flex-row gap-2 items-stretch justify-stretch">
          <Conversation>
            <ConversationContent>
              {messages
                .filter(
                  // Take only user and brain messages
                  (m) => m.role === "user" || m.metadata?.stream === "brain"
                )
                .map((message) => (
                  <Message from={message.role} key={message.id}>
                    <MessageContent>
                      {message.parts.map((part, i) => {
                        switch (part.type) {
                          case "tool-getPackagingDesign":
                            return (
                              <Tool
                                defaultOpen={false}
                                key={`${message.id}-${i}`}
                              >
                                <ToolHeader
                                  type={part.type}
                                  state={part.state}
                                />
                                <ToolContent>
                                  <ToolInput input={part.input} />
                                  <ToolOutput
                                    output={part.output}
                                    errorText={part.errorText}
                                  />
                                </ToolContent>
                              </Tool>
                            );
                          case "text":
                            return (
                              <Response key={`${message.id}-${i}`}>
                                {part.text}
                              </Response>
                            );
                          default:
                            return null;
                        }
                      })}
                    </MessageContent>
                  </Message>
                ))}
              {status === "submitted" && <Loader />}
            </ConversationContent>
            <ConversationScrollButton />
          </Conversation>

          <Conversation>
            <ConversationContent>
              {messages
                .filter(
                  // Take only designer messages
                  (m) => m.metadata?.stream === "designer"
                 )
                .map((message) => (
                  <Message from={message.role} key={message.id}>
                    <MessageContent>
                      {message.parts.map((part, i) => {
                        switch (part.type) {
                          case "text":
                            return (
                              <Response key={`${message.id}-${i}`}>
                                {part.text}
                              </Response>
                            );
                          case "file":
                            if (part.mediaType.startsWith("image/")) {
                              return (
                                <div
                                  className="w-full aspect-square"
                                  key={`${message.id}-${i}`}
                                >
                                  <img
                                    src={part.url}
                                    className="w-full h-full object-contain"
                                  />
                                </div>
                              );
                            }
                            return null;
                          default:
                            return null;
                        }
                      })}
                    </MessageContent>
                  </Message>
                ))}
            </ConversationContent>
            <ConversationScrollButton />
          </Conversation>
        </div>

        <PromptInput
          onSubmit={handleSubmit}
          className="mt-4 w-full max-w-2xl mx-auto relative"
        >
          <PromptInputTextarea
            value={input}
            placeholder="Say something..."
            onChange={(e) => setInput(e.currentTarget.value)}
            className="pr-12"
          />
          <PromptInputSubmit
            status={status === "streaming" ? "streaming" : "ready"}
            disabled={!input.trim()}
            className="absolute bottom-1 right-1"
          />
        </PromptInput>
      </div>
    </div>
  );
}

Issue

So as I outlined above, the issue with this is that messages jump from designer back to main. As well as the parts get duplicated across messages.

The raw response from the backend (Click to Open)

data: {"type":"start","messageMetadata":{"stream":"main"},"messageId":"8rzlmn9GciT8CPIA"}

data: {"type":"start-step"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-start","id":"0","providerMetadata":{"google":{"thoughtSignature":"<redacted>"}}}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":"A great Minecraft character skin often balances several factors:\n\n*   **Originality and Creativity:** Unique designs that stand out from the default or common skins are highly valued.\n*   **Clear Theme or Concept:** A skin that tells a","providerMetadata":{"google":{"thoughtSignature":"<redacted>"}}}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":" story or represents a clear idea (e.g., a futuristic astronaut, a mythical creature, a famous character) is often more engaging.\n*   **Effective Use of Pixels and Shading:** Even with the low resolution, skilled artists"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":" can use subtle shading and pixel placement to create depth, texture, and visual interest.\n*   **Balanced Color Palette:** Harmonious and well-chosen colors make the skin visually appealing and easy on the eyes. Too many clashing colors can"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":" make it look messy.\n*   **Recognizability (if applicable):** If the skin is based on a pop culture character or a real-world figure, it should be instantly recognizable.\n*   **Attention to Detail:** Small"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":" details, like accessories, patterns, or facial expressions, can add a lot of personality to a skin.\n*   **Personal Preference:** Ultimately, what one person considers \"great\" can be subjective and reflect their own style and interests.\n\nNow"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":", here's my best impression of a great Minecraft skin:\n"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"tool-input-start","toolCallId":"7FMJwHlhadYLlfFc","toolName":"getDesign"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"tool-input-delta","toolCallId":"7FMJwHlhadYLlfFc","inputTextDelta":"{\"prompt\":\"A highly creative and original Minecraft character skin. It should depict a wise, ancient forest guardian with intricate leaf and bark patterns for clothing, glowing moss accents, and deep, knowing eyes. The design should showcase excellent pixel art and shading to give depth and texture, utilizing a natural, earthy color palette with subtle magical glows.\"}"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"tool-input-available","toolCallId":"7FMJwHlhadYLlfFc","toolName":"getDesign","input":{"prompt":"A highly creative and original Minecraft character skin. It should depict a wise, ancient forest guardian with intricate leaf and bark patterns for clothing, glowing moss accents, and deep, knowing eyes. The design should showcase excellent pixel art and shading to give depth and texture, utilizing a natural, earthy color palette with subtle magical glows."}}

data: {"type":"start","messageMetadata":{"stream":"designer"},"messageId":"0hRksDDwN7FSkpGK"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-end","id":"0"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"start-step"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-start","id":"0"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-delta","id":"0","delta":"Here's your"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-delta","id":"0","delta":" highly"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-delta","id":"0","delta":" creative and original Minecraft"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-delta","id":"0","delta":" character skin, depicting"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-delta","id":"0","delta":" a wise, ancient"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-delta","id":"0","delta":" forest guardian:\n\n"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"file","mediaType":"image/png","url":"data:image/png;base64,<redacted>"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"text-end","id":"0"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"finish-step"}

data: {"type":"message-metadata","messageMetadata":{"stream":"designer"}}

data: {"type":"finish","messageMetadata":{"stream":"designer"}}

data: {"type":"tool-output-available","toolCallId":"7FMJwHlhadYLlfFc","output":"Here's your highly creative and original Minecraft character skin, depicting a wise, ancient forest guardian:\n\n"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"finish-step"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"start-step"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-start","id":"0"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":"Here's your"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":" highly creative and original Minecraft character skin, depicting a wise, ancient forest guardian:"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":"\n\nA wise, ancient forest guardian with intricate leaf and bark patterns for clothing, glowing moss accents, and deep, knowing eyes. The design showcases excellent pixel art and shading to give depth and texture, utilizing a natural, earthy color palette with subtle magical"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-delta","id":"0","delta":" glows."}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"text-end","id":"0"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"finish-step"}

data: {"type":"message-metadata","messageMetadata":{"stream":"main"}}

data: {"type":"finish","messageMetadata":{"stream":"main"}}

data: [DONE]

Video recording of the issue:

It seems that the parts get somehow merged.

Expectation

I'd expect that the designer message would not be inlined within main message. And the parts also shouldn't be duplicated.

How do I get my expected result?