Need help in converting json to nodes and store in FAISS vector database

[mraguth]

I have json file and I like to read, embed, store in to FAISS vector db. Hoever, I am getting error. Please help. Code I am using...

# Load JSON file
file_path = "c:/parquet/json/data.json"
with open(file_path, 'r') as f:
    data = json.load(f)

# Convert JSON data to a list of Document objects
docs = [Document(page_content=json.dumps(item)) for item in data["Data"]]
embed_model = embed_model

splitter = SemanticSplitterNodeParser(
    buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model
)

nodes = splitter.get_nodes_from_documents(docs)

vector_index = VectorStoreIndex(nodes)
query_engine_split = vector_index.as_query_engine()

[dosubot[bot]]

To convert a JSON file to nodes and store it in a FAISS vector database, you can follow these steps. Here's a corrected version of your code:

1. Load the JSON file.
2. Convert JSON data to a list of Document objects.
3. Use SemanticSplitterNodeParser to split documents into nodes.
4. Embed the nodes and store them in a FAISS vector database.

Here's the complete code:

import json
import faiss
import numpy as np
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.node_parser.text.semantic_splitter import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding  # or your chosen embedding model

# Load JSON file
file_path = "c:/parquet/json/data.json"
with open(file_path, 'r') as f:
    data = json.load(f)

# Convert JSON data to a list of Document objects
docs = [Document(text=json.dumps(item)) for item in data["Data"]]

# Initialize the embedding model
embed_model = OpenAIEmbedding()  # Replace with your actual embedding model

# Initialize the SemanticSplitterNodeParser
splitter = SemanticSplitterNodeParser(
    buffer_size=1, 
    breakpoint_percentile_threshold=95, 
    embed_model=embed_model
)

# Split documents into nodes
nodes = splitter.get_nodes_from_documents(docs)

# Convert nodes to embeddings
embeddings = [node.get_embedding() for node in nodes]
embeddings = np.array(embeddings, dtype='float32')

# Initialize FAISS index
dimension = embeddings.shape[1]
faiss_index = faiss.IndexFlatL2(dimension)
faiss_index.add(embeddings)

# Save the FAISS index to disk
faiss.write_index(faiss_index, 'faiss_index.bin')

print("FAISS index created and saved successfully.")

Key Points:

• Ensure you have the correct embedding model initialized (OpenAIEmbedding in this example).
• The SemanticSplitterNodeParser is used to split documents into nodes.
• Nodes are converted to embeddings and stored in a FAISS vector database.

This should help you convert your JSON data to nodes and store them in a FAISS vector database without errors ^[1][2].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu I am getting Value error "ValueError: embedding not set." Also I like store FAISS vector databse locally or presisit local drive like "c:/faiss".... please help here.

[dosubot[bot]]

To resolve the "ValueError: embedding not set" error and store the FAISS vector database locally, you need to ensure that the embeddings are properly set before adding them to the FAISS index. Additionally, you can use the persist method to save the FAISS index to a specified path.

Here's an updated version of your code:

import json
import faiss
import numpy as np
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.node_parser.text.semantic_splitter import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding  # or your chosen embedding model
from llama_index.legacy.vector_stores.faiss import FaissVectorStore
from llama_index.legacy.schema import BaseNode

# Load JSON file
file_path = "c:/parquet/json/data.json"
with open(file_path, 'r') as f:
    data = json.load(f)

# Convert JSON data to a list of Document objects
docs = [Document(text=json.dumps(item)) for item in data["Data"]]

# Initialize the embedding model
embed_model = OpenAIEmbedding()  # Replace with your actual embedding model

# Initialize the SemanticSplitterNodeParser
splitter = SemanticSplitterNodeParser(
    buffer_size=1, 
    breakpoint_percentile_threshold=95, 
    embed_model=embed_model
)

# Split documents into nodes
nodes = splitter.get_nodes_from_documents(docs)

# Ensure embeddings are set for each node
for node in nodes:
    node.embedding = embed_model.get_text_embedding(node.get_content())

# Initialize FAISS index
dimension = len(nodes[0].embedding)
faiss_index = faiss.IndexFlatL2(dimension)

# Initialize the FaissVectorStore
vector_store = FaissVectorStore(faiss_index=faiss_index)

# Add nodes to the FAISS index
vector_store.add(nodes)

# Persist the FAISS index to "c:/faiss"
persist_path = "c:/faiss/faiss_index"
vector_store.persist(persist_path=persist_path)

print("FAISS index created and saved successfully.")

Key Points:

1. Ensure Embeddings are Set: Before adding nodes to the FAISS index, ensure that embeddings are set for each node.
2. Persist FAISS Index: Use the persist method to save the FAISS index to the specified path "c:/faiss/faiss_index".

This should resolve the "ValueError: embedding not set" error and store the FAISS vector database locally ^[1].

To continue talking to Dosu, mention @dosu.