Need for Open Source AI Agent Workflow Framework

[howagain]

Seeking to develop an open-source, extensible framework for AI agent workflows, compatible with JavaScript and Python, to facilitate easy and efficient automation. Current tools have limitations, underscoring the need for a more versatile solution.

Existing Solutions & Their Limitations:

• 🌐 Agenthub Templates:
  • Pros: Excellent visual builder, rapid goal achievement with prebuilt blocks.
  • Cons: Closed source, limited free tier, lacks portability and future-proofing.
• 🎨 ComfyUI (Popular among Unity developers for image generation):
  • Pros: Flow-based design, favorable for Unity devs.
  • Cons: Lacks easy code integration, needs prebuilt pipelines, and extensions for Python/Node.js async functions.

Key Objectives for New Framework:

1. User-Friendly: Accessible to both JS and Python users.
2. Extensibility: Adaptable for various automation tasks.
3. Open Source: Open to community contributions and collaboration.

Eager for ideas and partnerships to build a more robust, open-source framework.

[howagain]

Did some research tonight on this.

Think that testing even a manual process is important and could help clarify our thinking.

This was a good kick off for that: https://towardsdatascience.com/testing-large-language-models-like-we-test-software-92745d28a359

Seems like yes no questions with some light JS is important.

Then did a search and found: https://github.com/confident-ai/deepeval

Seems like a good framework to test out.

It also pointed me to several other frameworks for LLMS (as we should have a testing and orchestration framework for more complex tools to get better speed and best practices).

https://www.guardrailsai.com/ - the RAIL format is interesting and I like the schema. Not a huge fan of the XML generated as it's hard to read without syntax highlighting, maybe VSCode would be helpful. Worried about "enterprise" use case, as we're looking more for fast and easy to prototype at this stage.

https://www.llamaindex.ai/ - have a good aesthetic to their site which I like and helps with adoption. Think it's a good supporter of the open source LLM framework and has use cases for unstructured data like they have in the site. Probably a good foundation to use.

https://python.langchain.com/docs/expression_language/get_started - seems like they've had some developments since I last looked and now have new product offerings and the Langchain Expression Language.

Also found tools like https://www.gradio.app/guides/key-features that seem like a nice E2E solution. As it gives us UI elements, custom building blocks, and is all in python. I like that there's a front end and back end interface, I can build my own components, and have composable blocks. I could see this being a nice integration with autogen or some other app. It would also allow us to train speciality models for specific data tasks.

Since it's svelte too we could always swap the front end out for astro as well. I think that we can have a better interface with astro and have more of a documentation site with helpful blog posts and sections to be able to quickly find infromation and then with astro we could have powerful models inside of it.

Not sure the best place to start with our use case and these tools but wanted to report on the research so far.

[howagain]

I also think that storing all of these tools in a vector database and managing the ingestion with LlamaIndex is looking like a good strategy as it integrates with Langchain for more complex agent workflows but is specifically designed to query data. It leans into an existing community rather than doing our own thing. I also think that using astro with MDX is a nice option to give people inline playground tools. Could also check out Repl.it as I think they just had a LLM powered hackathon.e

This was a nice overview of the different options

And this was a nice overview of LlamaIndex

[howagain]

Checked out Repl.it and I think that right now it's more of an alternative to github copilot rather than a framework for our use case

[howagain]

Did a bit more research and there's some companies. Would like to support an open source solution that allows companies to be built on top of it with a self hosting option. Most are in beta now as AI agents are still a new field. But saw Abacus.AI and https://www.lindy.ai/ both of which are in beta. Will keep eyes out for twitter. But think that a strong code first solution is better and more powerful than a no-code tool from the ground up. Transformers were first introduced theoretically and then the apps flowed from it. Tech shouldn't be built for the lowest common denominator from the start. Instead the main point is similar to customGPTs, can you power custom tasks and have a nice playground to build stuff easily. Let the extra features like no-code workflows and batteries included automations come later.