Vectordbpublic 0.1

.vscode/settings.json

@@ -11,6 +11,5 @@
         "nofooter",
         "ONTAP",
         "Vectorstores"
-    ],
-    "asciidoc.antora.enableAntoraSupport": false
+    ]
 }

ai/_index.yml

@@ -39,6 +39,8 @@ indexpage:
           url: tr-4915.html
         - title: "Hybrid Cloud AI with Data Caching"
           url: hcaios_executive_summary.html
+        - title: "Vector Database solution with NetApp"
+          url: vector-database-solution-with-netapp.html
 #
     - title: "Orchestation and Management"
       links:

ai/sidebar.yml

@@ -203,6 +203,46 @@ entries:
         url: https://github.com/NetApp/netapp-dataops-toolkit
       - title: "Data Movement with E-Series and BeeGFS for AI and Analytics Workflows"
         url: "/ai/tr-4915.html"
+      - title: "Vector Database solution with NetApp"
+        url: /ai/vector-database-solution-with-netapp.html
+        entries:
+          - title: "Introduction"
+            url: /ai/vector-database-introduction.html
+          - title: "Solution Overview"
+            url: /ai/vector-database-solution-overview.html
+          - title: "Vector Database"
+            url: /ai/vector-database-vector-database.html
+          - title: "Technology requirement"
+            url: /ai/vector-database-technology-requirement.html
+          - title: "Deployment Procedure"
+            url: /ai/vector-database-deployment-procedure.html
+          - title: "Solution Verification Overview"
+            url: /ai/vector-database-solution-verification-overview.html
+            entries:
+              - title: "Milvus cluster setup with Kubernetes in on-premises"
+                url: /ai/vector-database-milvus-cluster-setup.html
+              - title: "Milvus with Amazon FSxN for NetApp ONTAP – file and object duality"
+                url: /ai/vector-database-milvus-with-Amazon-FSxN-for-NetApp-ONTAP.html
+              - title: "Vector database protection using NetApp SnapCenter"
+                url: /ai/vector-database-protection-using-snapcenter.html
+              - title: "Disaster Recovery using NetApp SnapMirror"
+                url: /ai/vector-database-disaster-recovery-using-netapp-snapmirror.html
+              - title: "Performance validation"
+                url: /ai/vector-database-performance-validation.html
+          - title: "Vector database with Instaclustr using PostGreSQL: pgvector"
+            url: /ai/vector-database-instaclustr-with-pgvector.html
+          - title: "Vector Database Use Cases"
+            url: /ai/vector-database-use-cases.html
+          - title: "Conclusion"
+            url: /ai/vector-database-conclusion.html
+          - title: "Appendix A: values.yaml"
+            url: /ai/vector-database-values-yaml.html
+          - title: "Appendix B: prepare_data_netapp_new_py"
+            url: /ai/vector-database-prepare-data-netapp-new-py.html
+          - title: "Appendix C: verify_data_netapp.py"
+            url: /ai/vector-database-verify-data-netapp-py.html
+          - title: "Appendix D: docker_compose.yml"
+            url: /ai/vector-database-docker-compose-xml.html
 
   - title: "Use Cases"
     entries:

ai/vector-database-conclusion.adoc

@@ -0,0 +1,50 @@
+---
+sidebar: sidebar
+permalink: ai/vector-database-conclusion.html
+keywords: vector database, conclusion
+summary: conclusion - vector database solution for netapp
+---
+
+= Conclusion
+:hardbreaks:
+:nofooter:
+:icons: font
+:linkattrs:
+:imagesdir: ./../media/
+
+[.lead]
+
+== Conclusion
+
+In conclusion, this document provides a comprehensive overview of deploying and managing vector databases, such as Milvus and pgvector, on NetApp storage solutions. We discussed the infrastructure guidelines for leveraging NetApp ONTAP and StorageGRID object storage and validated the Milvus database in AWS FSX for NetApp ONTAP through file and object store. 
+
+We explored NetApp's file-object duality, demonstrating its utility not only for data in vector databases but also for other applications. We also highlighted how SnapCenter, NetApp's enterprise management product, offers backup, restore, and clone functionalities for vector database data, ensuring data integrity and availability. 
+
+The document also delves into how NetApp's Hybrid Cloud solution offers data replication and protection across on-premises and cloud environments, providing a seamless and secure data management experience. We provided insights into the performance validation of vector databases like Milvus and pgvecto on NetApp ONTAP, offering valuable information on their efficiency and scalability.
+
+Finally, we discussed two generative AI use cases: RAG with LLM and the NetApp’s internal ChatAI. These practical examples underscore the real-world applications and benefits of the concepts and practices outlined in this document. Overall, this document serves as a comprehensive guide for anyone looking to leverage NetApp's powerful storage solutions for managing vector databases.
+
+=== Acknowledgments
+The author like to heartfelt thanks to the below contributors, others who provided their feedback and comments to make this paper valuable to NetApp customers and NetApp fields.   
+
+. Sathish Thyagarajan, Technical Marketing Engineer, ONTAP AI & Analytics, NetApp
+. Mike Oglesby, Technical Marketing Engineer, NetApp
+. AJ Mahajan, Senior Director, NetApp
+. Joe Scott, Manager, Workload Performance Engineering, NetApp
+. Puneet Dhawan, Senior Director, Product Management Fsx, NetApp
+. Yuval Kalderon, Senior Product Manager, FSx Product Team, NetApp
+
+=== Where to find additional information
+To learn more about the information that is described in this document, review the following documents and/or websites:
+
+* Milvus documentation - https://milvus.io/docs/overview.md
+* Milvus standalone documentation - https://milvus.io/docs/v2.0.x/install_standalone-docker.md
+* NetApp Product Documentation
+https://www.netapp.com/support-and-training/documentation/
+* instaclustr - link:https://www.instaclustr.com/support/documentation/?_bt=&_bk=&_bm=&_bn=x&_bg=&utm_term=&utm_campaign=&utm_source=adwords&utm_medium=ppc&hsa_acc=1467100120&hsa_cam=20766399079&hsa_grp=&hsa_ad=&hsa_src=x&hsa_tgt=&hsa_kw=&hsa_mt=&hsa_net=adwords&hsa_ver=3&gad_source=1&gclid=CjwKCAjw26KxBhBDEiwAu6KXtzOZhN0dl0H1smOMcj9nsC0qBQphdMqFR7IrVQqeG2Y4aHWydUMj2BoCdFwQAvD_BwE[instalclustr documentation]
+
+=== Version history
+|===
+|Version	|Date	|Document version history
+|Version 1.0	|April 2024	|Initial release
+|===

ai/vector-database-deployment-procedure.adoc

@@ -0,0 +1,52 @@
+---
+sidebar: sidebar
+permalink: ai/vector-database-deployment-procedure.html
+keywords: vector database, vector-database-deployment-procedure
+summary: vector-database-deployment-procedure - vector database solution for netapp
+---
+
+= Deployment Procedure
+:hardbreaks:
+:nofooter:
+:icons: font
+:linkattrs:
+:imagesdir: ./../media/
+
+[.lead]
+
+== Deployment procedure
+
+In this deployment section, we used milvus vector database with Kubernetes for the lab setup as below. 
+
+image:Deployment_architecture.png[Error: Missing Graphic Image] 
+
+The netapp storage provides the storage for the cluster to keep customers data and milvus cluster data. 
+
+=== NetApp storage setup – ONTAP
+
+* Storage system initialization 
+* Storage virtual machine (SVM) creation 
+* Assignment of logical network interfaces
+* NFS, S3 configuration and licensing 
+
+Please follow the steps below for NFS (Network File System):
+
+1. 	Create a FlexGroup volume for NFSv4. In our set up for this validation, we have used 48 SSDs, 1 SSD dedicated for the controller’s root volume and 47 SSDs spread across for NFSv4]].Verify that the NFS export policy for the FlexGroup volume has read/write permissions for the Kubernetes (K8s) nodes network. If these permissions are not in place, grant read/write (rw) permissions for the K8s nodes network.
+2.	On all K8s nodes, create a folder and mount the FlexGroup volume onto this folder through a Logical Interface (LIF) on each K8s nodes.
+
+Please follow the steps below for NAS S3 (Network Attached Storage Simple Storage Service):
+
+1.	Create a FlexGroup volume for NFS.
+2.	Set up an object-store-server with HTTP enabled and the admin status set to 'up' using the "vserver object-store-server create" command. You have the option to enable HTTPS and set a custom listener port.
+3.	Create an object-store-server user using the "vserver object-store-server user create -user <username>" command.
+4.	To obtain the access key and secret key, you can run the following command: "set diag; vserver object-store-server user show -user <username>". However, moving forward, these keys will be supplied during the user creation process or can be retrieved using REST API calls. 
+5.	Establish an object-store-server group using the user created in step 2 and grant access. In this example, we have provided "FullAccess".
+6.	Create a NAS bucket by setting its type to "nas" and supplying the path to the NFSv3 volume. It's also possible to utilize an S3 bucket for this purpose.
+
+=== NetApp storage setup – StorageGRID
+
+1.	Install the storageGRID software.
+2.	Create a tenant and bucket.
+3.	Create user with required permission. 
+
+Please check more details in https://docs.netapp.com/us-en/storagegrid-116/primer/index.html

ai/vector-database-disaster-recovery-using-netapp-snapmirror.adoc

@@ -0,0 +1,30 @@
+---
+sidebar: sidebar
+permalink: ai/vector-database-disaster-recovery-using-netapp-snapmirror.html
+keywords: vector database
+summary: Disaster Recovery using NetApp SnapMirror - vector database solution for netapp
+---
+
+= Disaster Recovery using NetApp SnapMirror
+:hardbreaks:
+:nofooter:
+:icons: font
+:linkattrs:
+:imagesdir: ./../media/
+
+[.lead]
+== Disaster Recovery using NetApp SnapMirror
+
+image:vector_database_dr_fsxn_gcnv.png[Error: Missing Graphic Image]
+
+Disaster recovery is crucial for maintaining the integrity and availability of a vector database, especially given its role in managing high-dimensional data and executing complex similarity searches. A well-planned and implemented disaster recovery strategy ensures that data is not lost or compromised in the event of unforeseen incidents, such as hardware failures, natural disasters, or cyber-attacks. This is particularly significant for applications relying on vector databases, where the loss or corruption of data could lead to significant operational disruptions and financial losses. Moreover, a robust disaster recovery plan also ensures business continuity by minimizing downtime and allowing for the quick restoration of services. This is achieved through NetApp data replication product SnapMirrror across different geographical locations, regular backups, and failover mechanisms. Therefore, disaster recovery is not just a protective measure, but a critical component of responsible and efficient vector database management.
+
+NetApp's SnapMirror provides data replication from one NetApp ONTAP storage controller to another, primarily used for disaster recovery (DR) and hybrid solutions. In the context of a vector database, this tool facilitates the smooth transition of data between on-premises and cloud environments. This transition is achieved without necessitating any data conversions or application refactoring, thereby enhancing the efficiency and flexibility of data management across multiple platforms.
+
+NetApp Hybrid solution in a vector database scenario can bring about more advantages:
+
+. Scalability: NetApp's hybrid cloud solution offers the ability to scale your resources as per your requirements. You can utilize on-premises resources for regular, predictable workloads and cloud resources such as Amazon FSxN for NetApp ONTAP and Google Cloud NetApp Volume (GCNV) for peak times or unexpected loads.
+. Cost Efficiency: NetApp's hybrid cloud model allows you to optimize your costs by using on-premises resources for regular workloads and only paying for cloud resources when you need them. This pay-as-you-go model can be quite cost-effective with a NetApp instaclustr service offering. For on-prem and major cloud service providers, instaclustr provids support and consultation.  
+. Flexibility: NetApp's hybrid cloud gives you the flexibility to choose where to process your data. For example, you might choose to perform complex vector operations on-premises where you have more powerful hardware, and less intensive operations in the cloud. 
+. Business Continuity: In the event of a disaster, having your data in a NetApp hybrid cloud can ensure business continuity. You can quickly switch to the cloud if your on-premises resources are affected. We can leverage NetApp SnapMirror to move the data from on-prem to cloud and vice versa.
+. Innovation: NetApp's hybrid cloud solutions can also enable faster innovation by providing access to cutting-edge cloud services and technologies. NetApp innovations in cloud such as Amazon FSxN for NetApp ONTAP, Azure NetApp Files and Google Cloud NetApp Volumes are cloud service providers innovative products and preferred NAS. 

ai/vector-database-docker-compose-xml.adoc

@@ -0,0 +1,86 @@
+---
+sidebar: sidebar
+permalink: ai/vector-database-docker-compose-xml.html
+keywords: vector database
+summary: docker-compose.xml - vector database solution for netapp
+---
+
+= Appendix D: docker-compose.yml
+:hardbreaks:
+:nofooter:
+:icons: font
+:linkattrs:
+:imagesdir: ./../media/
+
+[.lead]
+
+== Appendix D: docker-compose.yml
+
+[source,yml]
+....
+version: '3.5'
+
+services:
+  etcd:
+    container_name: milvus-etcd
+    image: quay.io/coreos/etcd:v3.5.5
+    environment:
+      - ETCD_AUTO_COMPACTION_MODE=revision
+      - ETCD_AUTO_COMPACTION_RETENTION=1000
+      - ETCD_QUOTA_BACKEND_BYTES=4294967296
+      - ETCD_SNAPSHOT_COUNT=50000
+    volumes:
+      - /home/ubuntu/milvusvectordb/volumes/etcd:/etcd
+    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
+    healthcheck:
+      test: ["CMD", "etcdctl", "endpoint", "health"]
+      interval: 30s
+      timeout: 20s
+      retries: 3
+
+  minio:
+    container_name: milvus-minio
+    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
+    environment:
+      MINIO_ACCESS_KEY: minioadmin
+      MINIO_SECRET_KEY: minioadmin
+    ports:
+      - "9001:9001"
+      - "9000:9000"
+    volumes:
+      - /home/ubuntu/milvusvectordb/volumes/minio:/minio_data
+    command: minio server /minio_data --console-address ":9001"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
+      interval: 30s
+      timeout: 20s
+      retries: 3
+
+  standalone:
+    container_name: milvus-standalone
+    image: milvusdb/milvus:v2.4.0-rc.1
+    command: ["milvus", "run", "standalone"]
+    security_opt:
+    - seccomp:unconfined
+    environment:
+      ETCD_ENDPOINTS: etcd:2379
+      MINIO_ADDRESS: minio:9000
+    volumes:
+      - /home/ubuntu/milvusvectordb/volumes/milvus:/var/lib/milvus
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
+      interval: 30s
+      start_period: 90s
+      timeout: 20s
+      retries: 3
+    ports:
+      - "19530:19530"
+      - "9091:9091"
+    depends_on:
+      - "etcd"
+      - "minio"
+
+networks:
+  default:
+    name: milvus
+....

ai/vector-database-instaclustr-with-pgvector.adoc

@@ -0,0 +1,18 @@
+---
+sidebar: sidebar
+permalink: ai/vector-database-instaclustr-with-pgvector.html
+keywords: vector database
+summary: instaclustr with pgvector - vector database solution for netapp
+---
+
+= Vector Database with Instaclustr using PostgreSQL: pgvector
+:hardbreaks:
+:nofooter:
+:icons: font
+:linkattrs:
+:imagesdir: ./../media/
+
+[.lead]
+== Vector Database with Instaclustr using PostgreSQL: pgvector
+
+In this section, we delve into the specifics of how instaclustr product integrates with postgreSQL on pgvector fuctionality. We have an example of “How To Improve Your LLM Accuracy and Performance With PGVector and PostgreSQL®: Introduction to Embeddings and the Role of PGVector”. Please check the link:https://www.instaclustr.com/blog/how-to-improve-your-llm-accuracy-and-performance-with-pgvector-and-postgresql-introduction-to-embeddings-and-the-role-of-pgvector/[blog] to get more information.

ai/vector-database-introduction.adoc

@@ -0,0 +1,30 @@
+---
+sidebar: sidebar
+permalink: ai/vector-database-introduction.html
+keywords: vector database
+summary: Introduction to vector database solution for netapp
+---
+
+= Introduction
+:hardbreaks:
+:nofooter:
+:icons: font
+:linkattrs:
+:imagesdir: ./../media/
+
+[.lead]
+== Introduction
+
+Vector databases effectively address the challenges that are designed to handle the complexities of semantic search in Large Language Models (LLMs) and generative Artificial Intelligence (AI). Unlike traditional data management systems, vector databases are capable of processing and searching through various types of data, including images, videos, text, audio, and other forms of unstructured data, by using the content of the data itself rather than labels or tags.
+
+The limitations of Relational Database Management Systems (RDBMS) are well-documented, particularly their struggles with high-dimensional data representations and unstructured data common in AI applications. RDBMS often necessitate a time-consuming and error-prone process of flattening data into more manageable structures, leading to delays and inefficiencies in searches. Vector databases, however, are designed to circumvent these issues, offering a more efficient and accurate solution for managing and searching through complex and high-dimensional data, thus facilitating the advancement of AI applications.
+
+This document serves as a comprehensive guide for customers who are currently using or planning to use vector databases, detailing the best practices for utilizing vector databases on platforms such as NetApp ONTAP, NetApp StorageGRID, Amazon FSxN for NetApp ONTAP, and SnapCenter. The content provided herein covers a range of topics:
+
+* Infrastructure guidelines for vector databases, like Milvus, provided by NetApp storage through NetApp ONTAP and StorageGRID object storage. 
+* Validation of the Milvus database in AWS FSX for NetApp ONTAP through file and object store.
+* Delves into NetApp’s file-object duality, demonstrating its utility for data in vector databases as well as other applications. 
+* How NetApp’s Data Protection Management product, SnapCenter, offers backup and restore functionalities for vector database data.
+* How NetApp’s Hybrid Cloud offers data replication and protection across on-premises and cloud environments. 
+* Provides insights into the performance validation of vector databases like Milvus and pgvector on NetApp ONTAP.
+* Two specific use cases: Retrieval Augmented Generation (RAG) with Large Language Models(LLM) and the NetApp IT team’s ChatAI, thereby offering practical examples of the concepts and practices outlined.