create_response and cancel_response APIs for MultimodalAgent

[longcw]

Created a push-to-talk example for the manual VAD of OAI realtime api.

A follow up of #1347

[changeset-bot]

⚠️ No Changeset found

Latest commit: ee14d48

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[lukasIO]

just my 2c, but would be nice if this was using @bcherry's RPC feature via LiveKit, instead of a custom WebSocket ?

[longcw]

just my 2c, but would be nice if this was using @bcherry's RPC feature via LiveKit, instead of a custom WebSocket ?

The frontend in the example is just a simple button without a full livekit client. What is the simplest way to incorporate the RPC in a single web page or do we want to add a full client in the example. Or mention to use RPC for a real production in readme and comments?

[longcw]

@lukasIO @davidzhao updated to use RPC with frontend example livekit-examples/voice-assistant-frontend#23

[davidzhao]

@longcw since the frontend will take awhile, can we push the multimodal_agent changes in first, without the PTT example?

[longcw]

@longcw since the frontend will take awhile, can we push the multimodal_agent changes in first, without the PTT example?

sounds good. how about we keep the example but just mention how should it work in readme without a link to the frontend.