Can I store Vector_index which belongs to pandas df_nodes?

[mraguth]

Most of the time I query in memory index. However, I am wondering if I store the same 'df_nodes' suggested in the below exapmple for the documents.

import os
from llama_index.core import (
    StorageContext,
    VectorStoreIndex,
    load_index_from_storage,
)

if not os.path.exists("storage"):
    index = VectorStoreIndex.from_documents(docs)
    # save index to disk
    index.set_index_id("vector_index")
    index.storage_context.persist("./storage")
else:
    # rebuild storage context
    storage_context = StorageContext.from_defaults(persist_dir="storage")
    # load index
    index = load_index_from_storage(storage_context, index_id="vector_index")

The way I create df_nodes are like below and my goal is store that df_nodes into vector database like above example:

table_dfs = []
table_dfs.append(df)
first_dataframe = table_dfs[0]
df_query_engines = [
    PandasQueryEngine(table_df, llm=llm) for table_df in table_dfs
]


summaries = [
    ("""
    1. This dataset contains Sales data by process and pathways.
      """
     ),
]

df_nodes = [
    IndexNode(text=summary, index_id=f"pandas{idx}")
    for idx, summary in enumerate(summaries)
]

df_id_query_engine_mapping = {
    f"pandas{idx}": df_query_engine
    for idx, df_query_engine in enumerate(df_query_engines)
}

# construct top-level vector index + query engine
vector_index = VectorStoreIndex(df_nodes)

[dosubot[bot]]

To store df_nodes into a vector database, you can use the add method from the MongoDBAtlasVectorSearch class. Here is an example of how you can do it:

# Assuming df_nodes is a list of BaseNode objects
df_nodes = [...]  # Your list of BaseNode objects

# Set up MongoDB connection
import pymongo
import os
import getpass

mongo_uri = getpass.getpass("MONGO_URI:")
if not mongo_uri:
    print("MONGO_URI not set")

def get_mongo_client(mongo_uri):
    """Establish connection to the MongoDB."""
    try:
        client = pymongo.MongoClient(mongo_uri)
        print("Connection to MongoDB successful")
        return client
    except pymongo.errors.ConnectionFailure as e:
        print(f"Connection failed: {e}")
        return None

mongo_client = get_mongo_client(mongo_uri)

DB_NAME = "your_database"
COLLECTION_NAME = "your_collection"

db = mongo_client[DB_NAME]
collection = db[COLLECTION_NAME]

# To ensure we are working with a fresh collection
collection.delete_many({})

# Initialize the MongoDBAtlasVectorSearch
from llama_index.vector_stores.mongodb import MongoDBAtlasVectorSearch

vector_store = MongoDBAtlasVectorSearch(
    mongo_client,
    db_name=DB_NAME,
    collection_name=COLLECTION_NAME,
    index_name="vector_index",
)

# Add nodes to the vector store
vector_store.add(df_nodes)

print("Successfully added nodes to the vector store")

This code initializes the MongoDBAtlasVectorSearch with the necessary parameters and then adds the df_nodes to the vector store using the add method. Make sure you have set up your MongoDB Atlas cluster, database, and collection as described in the provided documentation ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu ....I am not able to get the data out from pg.duckdb. That means, it never store... thoughts? Erros is below:

CatalogException: Catalog Error: Table with name documents does not exist!
Did you mean "information_schema.columns"?

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[13], [line 1](vscode-notebook-cell:?execution_count=13&line=1)
----> [1](vscode-notebook-cell:?execution_count=13&line=1) vector_store = DuckDBVectorStore.from_local(
      [2](vscode-notebook-cell:?execution_count=13&line=2)     "C:/Users/GIT/opensource/storage/pg.duckdb")

File c:\ProgramData\anaconda3\envs\llama2\Lib\site-packages\llama_index\vector_stores\duckdb\base.py:177, in DuckDBVectorStore.from_local(cls, database_path, table_name, embed_dim, text_search_config, **kwargs)
    [175](file:///C:/ProgramData/anaconda3/envs/llama2/Lib/site-packages/llama_index/vector_stores/duckdb/base.py:175)     _table_info = _conn.execute(f"SHOW {table_name};").fetchall()
    [176](file:///C:/ProgramData/anaconda3/envs/llama2/Lib/site-packages/llama_index/vector_stores/duckdb/base.py:176) except Exception as e:
--> [177](file:///C:/ProgramData/anaconda3/envs/llama2/Lib/site-packages/llama_index/vector_stores/duckdb/base.py:177)     raise ValueError(f"Index table {table_name} not found in the database.")
    [179](file:///C:/ProgramData/anaconda3/envs/llama2/Lib/site-packages/llama_index/vector_stores/duckdb/base.py:179) # Not testing for the column type similarity only testing for the column names.
    [180](file:///C:/ProgramData/anaconda3/envs/llama2/Lib/site-packages/llama_index/vector_stores/duckdb/base.py:180) _std = {"text", "node_id", "embedding", "metadata_"}

ValueError: Index table documents not found in the database.

[dosubot[bot]]

The error you're encountering indicates that the documents table does not exist in your DuckDB database. This can happen if the table was not created or if the data was not properly persisted. Here’s a complete example to ensure that the data is correctly stored and retrievable from the DuckDB vector database:

1. Store the data: Use the add method of the DuckDBVectorStore to store the TextNode objects.
2. Persist the data: Ensure the data is saved to disk.
3. Retrieve the data: Use the query method of the DuckDBVectorStore to retrieve the stored data.

Here is the complete code:

import pandas as pd
from llama_index.vector_stores.duckdb import DuckDBVectorStore
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.core.schema import TextNode, NodeRelationship, RelatedNodeInfo
from llama_index.query_engines.pandas import PandasQueryEngine
from llama_index.vector_stores.types import VectorStoreQuery

# Sample pandas DataFrame
data = {
    'column1': ['data1', 'data2'],
    'column2': ['data3', 'data4']
}
df = pd.DataFrame(data)

# Create a list of DataFrames
table_dfs = [df]
first_dataframe = table_dfs[0]

# Create query engines for each DataFrame
df_query_engines = [
    PandasQueryEngine(table_df, llm=None) for table_df in table_dfs
]

# Summaries for the nodes
summaries = [
    "1. This dataset contains Patient level data by process and pathways."
]

# Create nodes using TextNode
df_nodes = [
    TextNode(
        text=summary,
        id_=f"pandas{idx}",
        relationships={NodeRelationship.SOURCE: RelatedNodeInfo(node_id=f"pandas{idx}")},
        metadata={"source": "pandas"},
        embedding=[0.1, 0.2, 0.3]  # Example embedding
    )
    for idx, summary in enumerate(summaries)
]

# Initialize the DuckDB vector store and persist to disk
vector_store = DuckDBVectorStore("pg.duckdb", persist_dir="./persist/", embed_dim=3)

# Add nodes to the vector store
vector_store.add(df_nodes)

# Create storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Create index from documents
vector_index = VectorStoreIndex(df_nodes, storage_context=storage_context)

# Persist the storage context to disk
storage_context.persist()

print("Successfully added nodes to the vector store")

# Load the vector store from disk
vector_store = DuckDBVectorStore.from_local("./persist/pg.duckdb")

# Query the vector store to verify data storage
query_embedding = [0.1, 0.2, 0.3]  # Example query embedding
res = vector_store.query(VectorStoreQuery(query_embedding=query_embedding, similarity_top_k=1))

# Check the results
for node in res.nodes:
    print(f"Node ID: {node.node_id}")
    print(f"Content: {node.get_content()}")
    print(f"Metadata: {node.metadata}")

Key Points:

1. Initialization and Persistence: The DuckDBVectorStore is initialized with a database name and a persist directory to ensure data is saved to disk.
2. Adding Nodes: Nodes are added to the vector store using the add method.
3. Persisting Data: The storage_context.persist() method ensures that the data is saved to disk.
4. Loading from Disk: The DuckDBVectorStore.from_local method is used to load the vector store from the persisted database.
5. Querying Data: The query method is used to retrieve and verify the stored data.

By following these steps, you can ensure that the data is correctly stored and retrievable from the DuckDB vector database ^[1][2].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu... great. It did work. How I retrieve pandas dataframe back from duckdb? can you give me a sample code please?

[dosubot[bot]]

To retrieve the original pandas DataFrame from the DuckDB vector database, you can use the duckdb package to execute a query and fetch the results into a DataFrame. Here is an example of how you can do this:

import duckdb
import pandas as pd

# Connect to the DuckDB database
conn = duckdb.connect('path_to_your_database.duckdb')

# Execute a query to retrieve the data from the table
df = conn.execute('SELECT * FROM documents').fetchdf()

# Now df is a pandas DataFrame containing the data from the DuckDB table
print(df)

Replace 'path_to_your_database.duckdb' with the path to your DuckDB database file. This code will connect to the DuckDB database, execute a query to select all data from the documents table, and fetch the results into a pandas DataFrame ^[1].

To continue talking to Dosu, mention @dosu.