Skip to content

Need help running fastagent on Cloudflare Workers #574

@volkanunsal

Description

@volkanunsal

Hi. I'm working on a proof of concept to run fastagent on Cloudflare workers with Workers AI. This uses a Worker binding to invoke the models instead of a url:

env.AI.run("@cf/meta/llama-3.1-8b-instruct", { "messages": [...] });

I'm curious if there is a straightforward way to create a custom model provider to handle this use case. I saw there is a "generic" model API, but it probably wouldn't work for me because it appears to be a singleton. Maybe the new-api in the examples folder would fit my use-case, but I don't see a simple way of extending FastAgentLLM to add a new model definition -- is there a guide for extending it?

Is there anything I'm missing? What are my options?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions