How does AG-UI include Human In The Loop into their protocol?

[frederikhendrix]

Hey AG-UI,

I was wondering how you recommend adding a Human In The Loop principle to an application. I am not using any Agent framework, but I think I kind of get how to do it.

from ag_ui.core.events import (
    EventType,
    BaseEvent,
    TextMessageStartEvent,
    TextMessageContentEvent,
    TextMessageEndEvent,
    TextMessageChunkEvent,
    ThinkingTextMessageStartEvent,
    ThinkingTextMessageContentEvent,
    ThinkingTextMessageEndEvent,
    ToolCallStartEvent,
    ToolCallArgsEvent,
    ToolCallEndEvent,
    ToolCallChunkEvent,
    ToolCallResultEvent,
    ThinkingStartEvent,
    ThinkingEndEvent,
    StateSnapshotEvent,
    StateDeltaEvent,
    MessagesSnapshotEvent,
    RawEvent,
    CustomEvent,
    RunStartedEvent,
    RunFinishedEvent,
    RunErrorEvent,
    StepStartedEvent,
    StepFinishedEvent,
    Event
)

These are all the events in the latest version. I am thinking that when ToolCallArgsEvent is called you could await and let the user return something which would cancle everything or change certain arguments.

Is this how you guys envision the developer to include Human In The Loop? As far as I have seen MCP doesn't have specific tool decoration for tools you would want a Human In The Loop for.

Kind Regards,

Frederik

[obinnaokechukwu]

My guess is that it's something like this:

• Scenario: A user asks the agent to book a trip. The agent creates a plan (book flight, book hotel, rent car) and presents it to the user as a checklist. The user unchecks "rent car" and approves the rest. The agent then proceeds.

• AG-UI Mechanism: A CUSTOM event with a specific name (e.g., "request_user_approval").

• Producer's Role:

  1. Receive the initial RunAgentInput.
  2. Start the run by emitting RUN_STARTED.
  3. Generate the multi-step plan internally.
  4. Emit a CUSTOM event. The name must be a pre-agreed identifier like "request_user_approval". The value must be the plan object (e.g., an array of steps).
  5. Emit a TOOL_CALL_START event for a frontend tool that requires user interaction (e.g., name: "user_approval_tool"). The tool call will not end until the user responds.
  6. The Producer's logic now waits. The event stream is paused.
  7. When the Consumer sends the user's response, it arrives as a TOOL_CALL_RESULT event in a new RunAgentInput.
  8. The Producer processes the result from the tool call (e.g., the list of approved steps).
  9. The Producer continues its work, potentially emitting more events.
  10. Finally, the Producer emits RUN_FINISHED.

• Consumer's Role:

  1. Send the initial RunAgentInput.
  2. Listen for a TOOL_CALL_START event for a tool that the frontend must handle (e.g., user_approval_tool).
  3. When this event is received, use the arguments from the tool call to render a custom UI (e.g., a checklist of tasks with a "Confirm" button).
  4. When the user interacts with the UI and clicks "Confirm," capture the result (e.g., the list of checked items).
  5. Create a TOOL_CALL_RESULT event containing the user's decision.
  6. Send this result back to the Producer to resume the run.

[frederikhendrix]

My guess is that it's something like this:

• Scenario: A user asks the agent to book a trip. The agent creates a plan (book flight, book hotel, rent car) and presents it to the user as a checklist. The user unchecks "rent car" and approves the rest. The agent then proceeds.

• AG-UI Mechanism: A CUSTOM event with a specific name (e.g., "request_user_approval").

• Producer's Role:

  1. Receive the initial RunAgentInput.
  2. Start the run by emitting RUN_STARTED.
  3. Generate the multi-step plan internally.
  4. Emit a CUSTOM event. The name must be a pre-agreed identifier like "request_user_approval". The value must be the plan object (e.g., an array of steps).
  5. Emit a TOOL_CALL_START event for a frontend tool that requires user interaction (e.g., name: "user_approval_tool"). The tool call will not end until the user responds.
  6. The Producer's logic now waits. The event stream is paused.
  7. When the Consumer sends the user's response, it arrives as a TOOL_CALL_RESULT event in a new RunAgentInput.
  8. The Producer processes the result from the tool call (e.g., the list of approved steps).
  9. The Producer continues its work, potentially emitting more events.
  10. Finally, the Producer emits RUN_FINISHED.

• Consumer's Role:

  1. Send the initial RunAgentInput.
  2. Listen for a TOOL_CALL_START event for a tool that the frontend must handle (e.g., user_approval_tool).
  3. When this event is received, use the arguments from the tool call to render a custom UI (e.g., a checklist of tasks with a "Confirm" button).
  4. When the user interacts with the UI and clicks "Confirm," capture the result (e.g., the list of checked items).
  5. Create a TOOL_CALL_RESULT event containing the user's decision.
  6. Send this result back to the Producer to resume the run.

The agent creates a plan (book flight, book hotel, rent car) and presents it to the user as a checklist. The user unchecks "rent car" and approves the rest. The agent then proceeds.

This is also a tool call already right??

4. Emit a CUSTOM event. The name must be a pre-agreed identifier like "request_user_approval". The value must be the plan object (e.g., an array of steps).

Yes, I also think so but I wouldn't like this, this feels like you cannot create a UI for each possible tool that is being called. This feels like in your UI you really have to check what TOOL_START_EVENT will return as name and based on that specific name render the frontend a certain way.

Still thank you very much for your response!

[obinnaokechukwu]

The agent creates a plan (book flight, book hotel, rent car) and presents it to the user as a checklist. The user unchecks "rent car" and approves the rest. The agent then proceeds.

This is also a tool call already right??

This will be part of the tool call. The implementation can be a helper function that yields to the main loop and resumes when an appropriate Ack event is received.

And yes, I kind of agree with your sentiment that this feels like something that could be handled more cleanly, perhaps as a first-class citizen, but I can also see why the AG-UI designers chose this approach (I suspect because it's agent framework agnostic).