New resumability approach: durablefetch

[remorses]

I have been thinking about new simpler way to add resumability to any existing chat interface:

• replace API endpoint with a reverse proxy that keeps the request open
• when user leaves the page, the request to the server keeps going
• when user reconnects check if the request is still in progress, if true resume it, simply calling the same endpoint
• this approach relies on the ApI endpoint having an unique url for each chat

I implemented this idea using Cloudflare Durable Objects and put the code here: durablefetch

I also implemented a demo on github here

You can already try the approach in this demo website

Here is how a simple AI chat is implemented with this approach:

import { DefaultChatTransport } from 'ai'
import { DurableFetchClient } from 'durablefetch'

const df = new DurableFetchClient()

export function Chat({ chatId }) {
    const api = `/api/chat?chatId=${chatId}`

    useEffect(() => {
        // if in progress, send message and resume stream with durablefetch
        df.isInProgress(api).then(({ inProgress }) => {
            if (inProgress) {
                const text = localStorage.getItem('lastMessage') || ''
                return sendMessage({
                    text,
                })
            }
        })
    }, [])

    const { messages, sendMessage, error } = useChat({
        transport: new DefaultChatTransport({ api, fetch: df.fetch }),
        id: chatId,
    })

    return (
        <div className='flex flex-col w-full h-dvh py-8 stretch'>
            <div className='space-y-4 flex-grow overflow-y-auto'>
                {messages.map((m) => {
                    if (!m.parts?.length) return
                    return (
                        <div key={m.id} className='max-w-xl mx-auto'>
                            <div className='font-bold'>{m.role}</div>
                            <div className='space-y-4'>
                                {m.parts.map((p, i) => {
                                    switch (p.type) {
                                        case 'text':
                                            return (
                                                <div key={i}>
                                                    <p>{p.text}</p>
                                                </div>
                                            )
                                    }
                                })}
                            </div>
                        </div>
                    )
                })}
            </div>

            <form
                onSubmit={(e) => {
                    e.preventDefault()
                    const message = new FormData(e.currentTarget).get('message')
                    sendMessage({ text: message })
                    localStorage.setItem('lastMessage', message)
                    e.currentTarget.reset()
                }}
                className='w-full max-w-xl mx-auto'
            >
                <input
                    autoFocus
                    className='w-full p-2 border border-gray-300 rounded shadow-xl'
                    name='message'
                    placeholder='Say something...'
                />
            </form>
        </div>
    )
}

[remorses]

@lgrammel what do you think of this approach? this removes the need to host Redis for the user, you could even add this capability directly in the ai package itself to provide resumability out of the box