diff --git a/comps/retrievers/multimodal_langchain/redis/README.md b/comps/retrievers/multimodal_langchain/redis/README.md new file mode 100644 index 0000000000..1b7fa72584 --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/README.md @@ -0,0 +1,121 @@ +# Retriever Microservice + +This retriever microservice is a highly efficient search service designed for handling and retrieving embedding vectors from multimodal data. It operates by receiving an embedding vector as input and conducting a similarity search against vectors stored in a VectorDB database. Users must specify the VectorDB's URL and the index name, and the service searches within that index to find documents with the highest similarity to the input vector. + +The service primarily utilizes similarity measures in vector space to rapidly retrieve contentually similar documents. The vector-based retrieval approach is particularly suited for handling large datasets, offering fast and accurate search results that significantly enhance the efficiency and quality of information retrieval. + +Overall, this microservice provides robust backend support for applications requiring efficient similarity searches, playing a vital role in scenarios such as recommendation systems, information retrieval, or any other context where precise measurement of document similarity is crucial. + +## 🚀1. Start Microservice with Python (Option 1) + +To start the retriever microservice, you must first install the required python packages. + +### 1.1 Install Requirements + +```bash +pip install -r requirements.txt +``` +### 1.2 Setup VectorDB Service + +You need to setup your own VectorDB service (Redis in this example), and ingest your knowledge documents into the vector database. + +As for Redis, you could start a docker container using the following commands. +Remember to ingest data into it manually. + +```bash +docker run -d --name="redis-vector-db" -p 6379:6379 -p 8001:8001 redis/redis-stack:7.2.0-v9 +``` +### 1.3 Ingest images or video + +Upload a video or images using the dataprep microservice, instructions can be found [here](https://github.com/opea-project/GenAIComps/tree/main/comps/dataprep/redis/multimodal_langchain/README.md). + +### 1.4 Start Retriever Service + +```bash +python langchain_multimodal/retriever_redis.py +``` + +## 🚀2. Start Microservice with Docker (Option 2) + +### 2.1 Setup Environment Variables + +```bash +export your_ip=$(hostname -I | awk '{print $1}') +export REDIS_URL="redis://${your_ip}:6379" +export INDEX_NAME=${your_index_name} +``` + +### 2.2 Build Docker Image + +```bash +cd ../../../../ +docker build -t opea/multimodal-retriever-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/multimodal_langchain/redis/docker/Dockerfile . +``` + +To start a docker container, you have two options: + +- A. Run Docker with CLI +- B. Run Docker with Docker Compose + +You can choose one as needed. + +### 2.3 Run Docker with CLI (Option A) + +```bash +docker run -d --name="multimodal-retriever-redis-server" -p 7000:7000 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e REDIS_URL=$REDIS_URL -e INDEX_NAME=$INDEX_NAME opea/multimodal-retriever-redis:latest +``` + +### 2.4 Run Docker with Docker Compose (Option B) + +```bash +cd docker +docker compose -f docker_compose_retriever.yaml up -d +``` + +## 🚀3. Consume Retriever Service + +### 3.1 Consume Embedding Service + +To consume the Retriever Microservice, you can generate a mock embedding vector of length 512 with Python. + +```bash +your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)") +curl http://${your_ip}:7000/v1/multimodal_retrieval \ + -X POST \ + -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding}}" \ + -H 'Content-Type: application/json' +``` + +You can set the parameters for the retriever. + +```bash +your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)") +curl http://localhost:7000/v1/multimodal_retrieval \ + -X POST \ + -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity\", \"k\":4}" \ + -H 'Content-Type: application/json' +``` + +```bash +your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)") +curl http://localhost:7000/v1/multimodal_retrieval \ + -X POST \ + -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity_distance_threshold\", \"k\":4, \"distance_threshold\":1.0}" \ + -H 'Content-Type: application/json' +``` + +```bash +your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)") +curl http://localhost:7000/v1/multimodal_retrieval \ + -X POST \ + -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity_score_threshold\", \"k\":4, \"score_threshold\":0.2}" \ + -H 'Content-Type: application/json' +``` + +```bash +your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)") +curl http://localhost:7000/v1/multimodal_retrieval \ + -X POST \ + -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"mmr\", \"k\":4, \"fetch_k\":20, \"lambda_mult\":0.5}" \ + -H 'Content-Type: application/json' +``` diff --git a/comps/retrievers/multimodal_langchain/redis/__init__.py b/comps/retrievers/multimodal_langchain/redis/__init__.py new file mode 100644 index 0000000000..916f3a44b2 --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/__init__.py @@ -0,0 +1,2 @@ +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 diff --git a/comps/retrievers/multimodal_langchain/redis/docker/Dockerfile b/comps/retrievers/multimodal_langchain/redis/docker/Dockerfile new file mode 100644 index 0000000000..33077c3037 --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/docker/Dockerfile @@ -0,0 +1,29 @@ +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +FROM langchain/langchain:latest + +ARG ARCH="cpu" + +RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \ + libgl1-mesa-glx \ + libjemalloc-dev \ + vim + +RUN useradd -m -s /bin/bash user && \ + mkdir -p /home/user && \ + chown -R user /home/user/ + +COPY comps /home/user/comps + +USER user + +RUN pip install --no-cache-dir --upgrade pip && \ + if [ ${ARCH} = "cpu" ]; then pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu; fi && \ + pip install --no-cache-dir -r /home/user/comps/retrievers/multimodal_langchain/redis/requirements.txt + +ENV PYTHONPATH=$PYTHONPATH:/home/user + +WORKDIR /home/user/comps/retrievers/multimodal_langchain/redis + +ENTRYPOINT ["python", "retriever_redis.py"] diff --git a/comps/retrievers/multimodal_langchain/redis/docker/docker_compose_retriever.yaml b/comps/retrievers/multimodal_langchain/redis/docker/docker_compose_retriever.yaml new file mode 100644 index 0000000000..efba29a4e1 --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/docker/docker_compose_retriever.yaml @@ -0,0 +1,23 @@ +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +version: "1.0" + +services: + retriever: + image: opea/multimodal-retriever-redis:latest + container_name: multimodal-retriever-redis-server + ports: + - "7000:7000" + ipc: host + environment: + no_proxy: ${no_proxy} + http_proxy: ${http_proxy} + https_proxy: ${https_proxy} + REDIS_URL: ${REDIS_URL} + INDEX_NAME: ${INDEX_NAME} + restart: unless-stopped + +networks: + default: + driver: bridge diff --git a/comps/retrievers/multimodal_langchain/redis/multimodal_config.py b/comps/retrievers/multimodal_langchain/redis/multimodal_config.py new file mode 100644 index 0000000000..f92d5755d7 --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/multimodal_config.py @@ -0,0 +1,80 @@ +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +import os + +current_file_path = os.path.abspath(__file__) +parent_dir = os.path.dirname(current_file_path) + +def get_boolean_env_var(var_name, default_value=False): + """Retrieve the boolean value of an environment variable. + Args: + var_name (str): The name of the environment variable to retrieve. + default_value (bool): The default value to return if the variable + is not found. + Returns: + bool: The value of the environment variable, interpreted as a boolean. + """ + true_values = {"true", "1", "t", "y", "yes"} + false_values = {"false", "0", "f", "n", "no"} + + # Retrieve the environment variable's value + value = os.getenv(var_name, "").lower() + + # Decide the boolean value based on the content of the string + if value in true_values: + return True + elif value in false_values: + return False + else: + return default_value + + +# Check for openai API key +#if "OPENAI_API_KEY" not in os.environ: +# raise Exception("Must provide an OPENAI_API_KEY as an env var.") + + +# Whether or not to enable langchain debugging +DEBUG = get_boolean_env_var("DEBUG", False) +# Set DEBUG env var to "true" if you wish to enable LC debugging module +if DEBUG: + import langchain + + langchain.debug = True + + +# Embedding model +EMBED_MODEL = os.getenv("EMBED_MODEL", "BridgeTower/bridgetower-large-itm-mlm-itc") + +# Redis Connection Information +REDIS_HOST = os.getenv("REDIS_HOST", "localhost") +REDIS_PORT = int(os.getenv("REDIS_PORT", 6379)) + + +def format_redis_conn_from_env(): + redis_url = os.getenv("REDIS_URL", None) + if redis_url: + return redis_url + else: + using_ssl = get_boolean_env_var("REDIS_SSL", False) + start = "rediss://" if using_ssl else "redis://" + + # if using RBAC + password = os.getenv("REDIS_PASSWORD", None) + username = os.getenv("REDIS_USERNAME", "default") + if password is not None: + start += f"{username}:{password}@" + + return start + f"{REDIS_HOST}:{REDIS_PORT}" + + +REDIS_URL = format_redis_conn_from_env() + +# Vector Index Configuration +INDEX_NAME = os.getenv("INDEX_NAME", "test-index") + +REDIS_SCHEMA = os.getenv("REDIS_SCHEMA", "redis_schema.yml") +schema_path = os.path.join(parent_dir, REDIS_SCHEMA) +INDEX_SCHEMA = schema_path +NUM_RETRIEVED_RESULTS = int(os.getenv("NUM_RETRIEVED_RESULTS", 1)) \ No newline at end of file diff --git a/comps/retrievers/multimodal_langchain/redis/redis_schema.yml b/comps/retrievers/multimodal_langchain/redis/redis_schema.yml new file mode 100644 index 0000000000..32f4a79ae4 --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/redis_schema.yml @@ -0,0 +1,19 @@ +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +text: + - name: content + - name: b64_img_str + - name: video_id + - name: source_video + - name: embedding_type + - name: title + - name: transcript_for_inference +numeric: + - name: time_of_frame_ms +vector: + - name: content_vector + algorithm: HNSW + datatype: FLOAT32 + dims: 512 + distance_metric: COSINE diff --git a/comps/retrievers/multimodal_langchain/redis/requirements.txt b/comps/retrievers/multimodal_langchain/redis/requirements.txt new file mode 100644 index 0000000000..e6ceddd4ef --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/requirements.txt @@ -0,0 +1,11 @@ +docarray[full] +fastapi +langchain_community +opentelemetry-api +opentelemetry-exporter-otlp +opentelemetry-sdk +prometheus-fastapi-instrumentator +redis +shortuuid +uvicorn +transformers \ No newline at end of file diff --git a/comps/retrievers/multimodal_langchain/redis/retriever_redis.py b/comps/retrievers/multimodal_langchain/redis/retriever_redis.py new file mode 100644 index 0000000000..50d567f821 --- /dev/null +++ b/comps/retrievers/multimodal_langchain/redis/retriever_redis.py @@ -0,0 +1,93 @@ +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +import time +from typing import Union + +from comps.embeddings.multimodal_embeddings.bridgetower import BridgeTowerEmbedding +from langchain_community.vectorstores import Redis +from multimodal_config import INDEX_NAME, REDIS_URL, REDIS_SCHEMA + +from comps import ( + EmbedMultimodalDoc, + SearchedMultimodalDoc, + ServiceType, + TextDoc, + opea_microservices, + register_microservice, + register_statistics, + statistics_dict, +) +from comps.cores.proto.api_protocol import ( + ChatCompletionRequest, + RetrievalRequest, + RetrievalResponse, + RetrievalResponseData, +) + +@register_microservice( + name="opea_service@multimodal_retriever_redis", + service_type=ServiceType.RETRIEVER, + endpoint="/v1/multimodal_retrieval", + host="0.0.0.0", + port=7000, +) + +@register_statistics(names=["opea_service@multimodal_retriever_redis"]) +def retrieve( + input: Union[EmbedMultimodalDoc, RetrievalRequest, ChatCompletionRequest] +) -> Union[SearchedMultimodalDoc, RetrievalResponse, ChatCompletionRequest]: + + start = time.time() + # check if the Redis index has data + if vector_db.client.keys() == []: + search_res = [] + else: + # if the Redis index has data, perform the search + if input.search_type == "similarity": + search_res = vector_db.similarity_search_by_vector(embedding=input.embedding, k=input.k) + elif input.search_type == "similarity_distance_threshold": + if input.distance_threshold is None: + raise ValueError("distance_threshold must be provided for " + "similarity_distance_threshold retriever") + search_res = vector_db.similarity_search_by_vector( + embedding=input.embedding, k=input.k, distance_threshold=input.distance_threshold + ) + elif input.search_type == "similarity_score_threshold": + docs_and_similarities = vector_db.similarity_search_with_relevance_scores( + query=input.text, k=input.k, score_threshold=input.score_threshold + ) + search_res = [doc for doc, _ in docs_and_similarities] + elif input.search_type == "mmr": + search_res = vector_db.max_marginal_relevance_search( + query=input.text, k=input.k, fetch_k=input.fetch_k, lambda_mult=input.lambda_mult + ) + else: + raise ValueError(f"{input.search_type} not valid") + + # return different response format + retrieved_docs = [] + if isinstance(input, EmbedMultimodalDoc): + metadata_list = [] + for r in search_res: + metadata_list.append(r.metadata) + retrieved_docs.append(TextDoc(text=r.page_content)) + result = SearchedMultimodalDoc(retrieved_docs=retrieved_docs, initial_query=input.text, metadata=metadata_list) + else: + for r in search_res: + retrieved_docs.append(RetrievalResponseData(text=r.page_content, metadata=r.metadata)) + if isinstance(input, RetrievalRequest): + result = RetrievalResponse(retrieved_docs=retrieved_docs) + elif isinstance(input, ChatCompletionRequest): + input.retrieved_docs = retrieved_docs + input.documents = [doc.text for doc in retrieved_docs] + result = input + + statistics_dict["opea_service@multimodal_retriever_redis"].append_latency(time.time() - start, None) + return result + + +if __name__ == "__main__": + + embeddings = BridgeTowerEmbedding() + vector_db = Redis.from_existing_index(embedding=embeddings, schema=REDIS_SCHEMA, index_name=INDEX_NAME, redis_url=REDIS_URL) + opea_microservices["opea_service@multimodal_retriever_redis"].start() diff --git a/tests/test_retrievers_multimodal_langchain_redis.sh b/tests/test_retrievers_multimodal_langchain_redis.sh new file mode 100644 index 0000000000..3d3c37ac30 --- /dev/null +++ b/tests/test_retrievers_multimodal_langchain_redis.sh @@ -0,0 +1,84 @@ +#!/bin/bash +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +set -x + +WORKPATH=$(dirname "$PWD") +LOG_PATH="$WORKPATH/tests" +ip_address=$(hostname -I | awk '{print $1}') + +function build_docker_images() { + cd $WORKPATH + docker build --no-cache -t opea/multimodal-retriever-redis:comps --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/multimodal_langchain/redis/docker/Dockerfile . + if [ $? -ne 0 ]; then + echo "opea/multimodal-retriever-redis built fail" + exit 1 + else + echo "opea/multimodal-retriever-redis built successful" + fi +} + +function start_service() { + # redis + docker run -d --name test-comps-multimodal-retriever-redis-vector-db -p 5010:6379 -p 5011:8001 -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy redis/redis-stack:7.2.0-v9 + sleep 10s + + # redis retriever + export REDIS_URL="redis://${ip_address}:5010" + export INDEX_NAME="rag-redis" + retriever_port=5009 + unset http_proxy + docker run -d --name="test-comps-multimodal-retriever-redis-server" -p ${retriever_port}:7000 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e REDIS_URL=$REDIS_URL -e INDEX_NAME=$INDEX_NAME opea/multimodal-retriever-redis:comps + + sleep 3m +} + +function validate_microservice() { + retriever_port=5009 + export PATH="${HOME}/miniforge3/bin:$PATH" + source activate + URL="http://${ip_address}:$retriever_port/v1/multimodal_retrieval" + test_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)") + + HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST -d "{\"text\":\"test\",\"embedding\":${test_embedding}}" -H 'Content-Type: application/json' "$URL") + if [ "$HTTP_STATUS" -eq 200 ]; then + echo "[ retriever ] HTTP status is 200. Checking content..." + local CONTENT=$(curl -s -X POST -d "{\"text\":\"test\",\"embedding\":${test_embedding}}" -H 'Content-Type: application/json' "$URL" | tee ${LOG_PATH}/retriever.log) + + if echo "$CONTENT" | grep -q "retrieved_docs"; then + echo "[ retriever ] Content is as expected." + else + echo "[ retriever ] Content does not match the expected result: $CONTENT" + docker logs test-comps-multimodal-retriever-redis-server >> ${LOG_PATH}/retriever.log + exit 1 + fi + else + echo "[ retriever ] HTTP status is not 200. Received status was $HTTP_STATUS" + docker logs test-comps-multimodal-retriever-redis-server >> ${LOG_PATH}/retriever.log + exit 1 + fi +} + +function stop_docker() { + cid_retrievers=$(docker ps -aq --filter "name=test-comps-multimodal-retriever*") + if [[ ! -z "$cid_retrievers" ]]; then + docker stop $cid_retrievers && docker rm $cid_retrievers && sleep 1s + fi +} + +function main() { + + stop_docker + + build_docker_images + start_service + + validate_microservice + + stop_docker + # echo y | docker system prune + +} + +main