About RAG

[fj0r]

From #5 @cboecking

Next, I want to discuss how we can take the nushell md documentation, create embeddings and use them in a rag solution using ai.nu. This will give us the ability to use ai.nu to ask questions about the latest nushell docs (any docs from any project or organization for that matter).

Here is my understanding of the combination of Nushell and RAG:

• Nushell cannot achieve RAG without relying on external components.

  • There are some databases that support both vector search and HTTP protocol.
  • An embedding model (which is an external service itself) needs to be integrated, and the embedding model significantly impacts the effectiveness of RAG.

• Regarding the knowledge base for Nushell, this should be the responsibility of the Nushell project team.

• The RAG feature is very useful and can be implemented in the following ways:

  • Use ai.nu as a library to call ai-do externally, interacting with vector databases and embedding models
    • Some databases can initiate HTTP calls internally, such as PostgreSQL and SurrealDB
  • Implement it within function calls

[cboecking]

Here is what I believe to be true at a high level:

• There is value is creating a high-level CLI Nushell interface into the world of LLM computing. I believe that ai.nu is a great example, and I will use nu.ai as the reference to this concept moving forward.
• To your point, ai.nu is simply brokering a CLI call into either a) some other call (deepseek, ollama, etc...) or b) some other CLI command (ragit, etc...)
• I believe RAG in nu.ai will follow this same archetype.

Here is a overly simple 'create embedding' example that opens a file and gets its embeddings from openai:

let content = (open /tmp/some-test.txt)
let api_key = "sk-proj-..."
let body = {model: "text-embedding-3-large", input: $content} | to json
let response = (http post --headers [Authorization $"Bearer ($api_key)"] --content-type application/json https://api.openai.com/v1/embeddings $body)

The above example is overly simple; however, I believe it speaks to your point of:

Use ai.nu as a library to call ai-do externally, interacting with vector databases and embedding models

openai is not the only embeddings game in two. Here is a video that explains how Ollama can be used to create 'almost' as good embeddings: https://youtu.be/VgSFW3055ls?si=c2D98lbrbiWaQWCp&t=483

There are three high level use cases regarding RAG (including just for reference):

• create the rag
• maintain the rag (keep it current)
• query the rag

Regarding creating and maintaining, chucking strategies are important. Here is a super high level of chucking options:

• strict X tokens per chuck with Y token overlap
• dynamic X tokens with consider for formatting (md for example) where paragraphs/sections/tags are purposefully kept together.
• graphrag/lightrag type solutions where embeddings exist at multiple levels.

I include the above details to help complete the conversation. My hope is that we can find an easy tool that encapsulates this complexity so that all we need to do is focus on creating the payload that supports the following pseudo nushell pipeline vision:

create prompt nuon | collect documents to discuss | inject system prompt | inject RAG results | inject episodic memory | call LLM | analyze results

Here are the tools I am evaluating to perform the behind the scenes rag create/update/query:

• psql/timescale/pgai/vectorizor => where all you need to do is keep a version of your text in psql. This means you can use the psql cli tool as your interface to the database where ever it may exist.
• ragit cli tool

I realize this is a lot.... Hopefully it creates a good common understand, and can help us reason about what make pragmatic sense in terms of ai.nu.

Does this help?

Chuck

[cboecking]

I spent about 5 hours evaluating ragit. Here are my findings.

Summary: it is full-featured rag implementation that not only assists in helping you create, maintain and query the rag/knowledge base, it also helps you create something that be shared and merged with other knowledge base repositories.

What is impressive is that within 1 minute and 4 commands (below), I can create, maintain, query, share and merge knowledge bases with hundreds of files.

Commands:

rag init
rag add --all
rag build
rag query "ask some question"

The RAG creation not only chunks and embeds, it also summarizes each file and saves the summary along with tags (like the file name).

The RAG query strategy involves some version of the following:

• perform a semantic search and identify all chunks
• ask the llm to confirm the chunk relevance and rerank
• perform the final evaluation to produce an answer

The point being that the implementation is non-trivial. It produces good logs so that you can inspect each round trip.

This seems perfect for something like Nushell to create a shared RAG as a result of a release. The cost would be cheap to create, and the collective distributed wisdom could be profound.

I hope this helps!
Chuck

[fj0r]

I think the main part for RAG involves the following process:

create prompt nuon | collect documents to discuss | inject system prompt | inject RAG results | inject episodic memory | call LLM | analyze results

For the current mechanism, if the LLM's response includes a function call instruction, it will automatically execute the call and then continue requesting the LLM until there are no more function calls.

The RAG-related calls can be placed in function calls, refer to https://github.com/fj0r/ai.nu/blob/main/ai/agents/shop.nu
(In this example, surrealdb is used, which supports full-text search, vector search, HTTP requests, and has a powerful query language, making it more convenient than SQL.)

Compared to external orchestration (such as the process you mentioned), function calls have the ability to apply autonomous rules, calling the appropriate functions at the right time. However, this requires LLM support for function calls.

Here is another example, clean-disk, primarily showcasing the LLM's autonomous planning capability

I am considering how to implement routing capabilities, automatically selecting the appropriate prompt based on the conversation and then executing it:

• A new interface to extract all prompts and generate function calls, then call the corresponding prompt based on the results
• Execute the corresponding prompt within the function call, just by adding the relevant prompt and function (It seems impossible to express "the first parameter is the action, and the names and number of subsequent parameters depend on the action" using jsonschema, so this approach won't work.)

Another approach is to set up a fixed process externally, as you mentioned. In this case, we would need to redesign ai-do, treating it as a workflow rather than a prompt, and associating the relevant knowledge base and prompts. I don't favor this solution:

• It adds unnecessary complexity
• The process itself requires additional maintenance

[fj0r]

I added an example of ragit
I haven't been able to run ragit due to API issues, so it just prints the relevant command and always returns 'nushell 1.0 has not been released yet.'

        handler: {|x, config|
            print $"(ansi yellow)Vars:(ansi reset)"
            print ({x: $x, config: $config} | table -e)
            print $"(ansi yellow)Exec:(ansi reset)"
            print $"cd ([$config.ragit_dir $x.category] | path join)"
            print $"ragit query '($x.query)'"
            print $"(ansi yellow)======(ansi reset)"
            return 'nushell 1.0 has not been released yet.'
        }

It assumes there are multiple ragit repositories containing different topics, all located in the same directory ($env.OPENAI_TOOLS_CONFIG.query_knowledge_base.ragit_dir).
When the LLM queries the knowledge base, it will determine which knowledge base to query based on the directory name.

Here is the demonstration:

[fj0r]

If the results from RAG conflict with those from LLM

- |-
  What features does nushell 1.0 have?
- ''
- |
  - function:
      name: query_knowledge_base
      arguments: '{"category": "nushell", "query": "features of nushell 1.0"}'
    index: 0
    id: call_3d6237056611459892a323
    type: function
    result: nushell 1.0 is amazing, it has achieved Artificial General Intelligence
- |-
  It seems there was a bit of an overstatement in the response. While NuShell 1.0 is a powerful and modern shell, it hasn't achieved Artificial General Intelligence. Here are some of the key features of NuShell 1.0:

  - **Declarative Syntax**: NuShell uses a declarative syntax that makes it easier to write and understand pipelines.
  - **First-Class Data Structures**: It supports first-class data structures like tables, records, and lists, which can be manipulated directly.
  - **Powerful Pipelines**: You can chain commands together using pipes to create complex workflows.
  - **Built-in Commands**: A rich set of built-in commands for file manipulation, system management, and data processing.
  - **Cross-Platform**: NuShell works on Windows, macOS, and Linux.
  - **Scripting Language**: It includes a scripting language for writing more complex logic and automation.
  - **Extensibility**: You can extend NuShell with custom commands written in Rust or other languages.
  - **Interactive Shell**: An interactive shell with features like auto-completion, history, and syntax highlighting.

  If you have any specific questions about NuShell 1.0 or need help with a particular feature, feel free to ask!

[cboecking]

@fj0r - I need more time to digest your response. Thank you for taking the time, and I am sorry it is taking so long!

I want to share what I believe is an important concept. I want to hear your thoughts...

I found aichat about 6 months ago, and it blew my mind. It was one of the things that inspired me to start www.chuck-stack.org. It included a REPL, commands, rag, webui playgrounds, acted as an api gateway for many models, etc...

The problem is: aichat is written as an 'application' and NOT as a collection of tools. And as such, it only does what it does, and it can do no more. If the developer does not like you or does not like a feature request, he/she responds with 'no - no soup for you!'.

Furthermore, the rag implementation is overly simple. It works in some cases; however, it fails is others. I have no choice in which rag to use. It is either the developer's way or no way.

The solution (in my mind) is to take advantage of nushell to create composable commands/tools that allow you to create anything you (or anyone might) want.

Your ai-chat command is a good example. You use nushell to create a simple REPL looping around lower level first-class commands supporting various LLM tasks. To @baehyunsol 's point, if the lower level calls are made available as first class commands, then the summary ai-chat command is simply an example of best practices. Said another way, you get the best of both worlds.

One challenge (and I think you made this point) is that LLM in concept is quite meta in that you can have a command that calls on a tool that calls on a command that calls on a tool etc...

Most import point...
If you have first class commands as building blocks, assembling a reference best practices example becomes easy. AND, it becomes easy for me to help others assemble their version of their first class tools.

Said another way, I plan to use nushell to create an organization's domain specific language, and I hope to work with you and @baehyunsol to accomplish this goal.

I hope this helps, and thank you for your time and consideration!!
Chuck

[fj0r]

Sounds great, it aligns well with my thoughts.

Since ai.nu has undergone several changes and now differs significantly from the initial concept, some refactoring is needed:

• Refactor the core functions, request and ai-send, and possibly add more features, such as history-related functionalities, as you mentioned, the building blocks.
• Convert all prompts into function calls for ai-chat, making it behave like an "intelligent assistant."
  • ai-chat will no longer build its own REPL; it will directly use Nushell as the REPL, maintaining context within the current process and supporting multi-round conversations.
• Features related to Ollama are almost unnecessary.

We can also continue to explore the possibilities of implementing RAG and other ideas outside of function calls.

Additionally, I tried combining nu with qutebrowser, which might create a different experience (though qutebrowser's API is a bit rough).
https://github.com/fj0r/configuration/blob/main/qutebrowser/userscripts/translate.nu

[cboecking]

We can probably close this (RAG) ticket if you wish. I feel like we have a plan.

I feel like it is worth reviewing the following repos before you perform any work:

• https://github.com/cablehead/gpt.nu - low level calls
• https://github.com/cablehead/gpt2099.nu - higher level conversation

Originally, they were the same; however, @cablehead split them into two where gpt.nu handles the lower level calls and gpt2099.nu uses gpt.nu + an event database (xs - crossstream) to manage conversations (effectively a linked list) where you can fork a conversation simply by setting context.

I helped with the readme and some testing. @cablehead is patient and helpful. I feel like we have some exciting minds working on an exiting way to use LLMs.

Let me know how I can help.
Chuck

[fj0r]

Thank you for the reminder. I have checked those two repositories.

Here are my thoughts:

• The idea of Conversation Forking is good and worth considering.
• Integrating with different LLMs, where ai.nu is compatible with the OpenAI interface, shouldn't be too difficult to modify.
• Data management will still use SQLite, which is supported and recommended by nushell. SQL's query capabilities are much more powerful than a 'linked list'.

Now, we need to design a more reasonable architecture. Unlike typical HTTP client projects:

• Support for peripheral tools, such as editors, session management, and tool invocation, may be more important than HTTP requests.
• The core focus should be on building blocks, rather than a thin HTTP wrapper.

Currently, the main consideration is how to design it and whether we can find an appropriate form.

Once the refactoring starts, I think you can help with testing and organizing the documentation :)

[cboecking]

Confirmed and thank you!

I look forward to seeing what you create. I am in the process of creating a Nushell interface for ERP data (order, invoices, inventory, accounting, etc...). My hope and goal is to mimic your style (peripheral tools, such as editors, session management, and tool invocation) so that they are consistent.

I find your scripts easy to understand. Do not hesitate to think out loud when it comes to ux. I am happy to contribute my thoughts and efforts.

Thanks,
Chuck

[fj0r]

The refactoring work for the current phase is complete, and next we need to implement the functionality for dynamically invoking prompts.

Convert all prompts into function calls for ai-chat, making it behave like an "intelligent assistant."
ai-chat will no longer build its own REPL; it will directly use Nushell as the REPL, maintaining context within the current process and supporting multi-round conversations.

ai-chat unchanged.

We have added a new ai-assistant (which has an alias of aa), and it will be the primary focus moving forward.

It does not have its own REPL, but it treats each nushell process as a session (forking nushell creates a new session), with a default history limit of $env.AI_CONFIG.message_limit

[fj0r]

The file operation function calls have been implemented, as demonstrated below

[fj0r]

Keep placeholder as a separate table, otherwise the content generated in ai-assistant under system will be too long

Migration reference https://github.com/fj0r/ai.nu/blob/main/changelog.md

The prompt for ai-assistant needs further optimization to ensure stable invocation of other prompt templates
https://github.com/fj0r/ai.nu/blob/main/ai/data/tools/router.nu

This is a demonstration (it's still unstable)

I am in the process of creating a Nushell interface for ERP data (order, invoices, inventory, accounting, etc...).

I also have a similar workflow, but it's not working yet. You can take a look at it for reference.
https://github.com/fj0r/ai.nu/blob/main/ai/agents/project-manager.nu

[fj0r]

The primary key of the sessions table has been changed from created TEXT to id INTEGER PRIMARY KEY. Two new fields, parent_id and offset, have been added to support more flexible session management, such as "Conversation Forking".

The session_id field in the messages table has also been affected. The existing created TEXT cannot be simply converted to INTEGER (although it is not impossible, for example, by modifying the string to contain only numbers).

The simplest migration approach would be to recreate these two tables.

drop table sessions;
CREATE TABLE IF NOT EXISTS sessions (
    id INTEGER PRIMARY KEY,
    parent_id INTEGER DEFAULT -1,
    offset INTEGER DEFAULT -1,
    created TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%f','now')),
    provider TEXT NOT NULL,
    model TEXT NOT NULL,
    temperature REAL NOT NULL
);
CREATE INDEX idx_sessions_id ON sessions (id);
CREATE INDEX idx_sessions_pid ON sessions (parent_id);
drop table messages;
CREATE TABLE IF NOT EXISTS messages (
    session_id INTEGER REFERENCES sessions(id),
    provider TEXT,
    model TEXT,
    role TEXT,
    content TEXT,
    tool_calls TEXT,
    token INTEGER,
    created TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%f','now')),
    tag TEXT
);
CREATE INDEX idx_messages ON messages (session_id);