Simple_Load_And_Split

The SimpleLoadAndSplit function uses the PyPDF2 library to extract text file types and split them into chunks. By default, it uses the “semantic_text_splitter” method from the semantic_text_splitter library and uses "bert-base-uncased" tokenizer frim HuggingFace.

def SimpleLoadAndSplit(file_path: str,
                       remove_sword: bool = False,
                       max_chunk_size: int = 500,
                             )

Hyperparameters

• file_path (str): The path to the file to be processed. Various file types are supported.
• remove_sword (bool): If True, remove stop words during the chunking process. Default is False.
• max_chunk_size (int): The maximum size of each chunk in characters. Default is 500.

Usage

To use the SimpleLoadAndSplit function, follow the steps below:

Import necessary libraries and load environment variables:

import os
from dotenv import load_dotenv

load_dotenv()
HUGGINGFACE_API_KEY = os.environ['HUGGINGFACE_API_KEY']

Initialize Mistral and QA models:

from indox.llms import Mistral
from indox.embeddings import HuggingFaceEmbedding

mistral_qa = Mistral(api_key=HUGGINGFACE_API_KEY)
embed = HuggingFaceEmbedding(model="multi-qa-mpnet-base-cos-v1")

Perform the Simple_load_and_split on the file:

file_path = "path/to/your/file.pdf"  # Specify the file path
simple_load_split = SimpleLoadAndSplit(file_path=file_path,
                                       remove_sword=False,
                                       max_chunk_size=200)

docs = simple_load_split.load_and_chunk()

Example Code

Here’s a complete example of using the SimpleLoadAndSplit function in a Jupyter notebook:

import os
from dotenv import load_dotenv

load_dotenv()
HUGGINGFACE_API_KEY = os.environ['HUGGINGFACE_API_KEY']

Initialize Mistral and QA models:

from indox.llms import Mistral
from indox.embeddings import HuggingFaceEmbedding

mistral_qa = Mistral(api_key=HUGGINGFACE_API_KEY)
embed = HuggingFaceEmbedding(model="multi-qa-mpnet-base-cos-v1")

file_path = "path/to/your/file.pdf"  # Specify the file path
simple_load_split = SimpleLoadAndSplit(file_path=file_path,
                                       remove_sword=False,
                                       max_chunk_size=200)

docs = simple_load_split.load_and_chunk()

---

Previous: Unstructured Load and Split | Next: Embedding Models