LLM on NERC

In this repo you will find resources, examples, recipes to setup and work with LLMs on NERC OpenShift and NERC RHOAI.

Content

LLM Inference Servers

The following LLM Inference Servers for LLMs can be deployed standalone on the NERC OpenShift:

• Ollama: how to deploy Ollama.

• llama.cpp: how to deploy llama.cpp.

Vector Databases

The following Databases can be used as a Vector Store for Retrieval Augmented Generation (RAG) applications:

• Qdrant: Full recipe to deploy the Qdrant, create a suitable Database for a Vector Store.

• Milvus: Full recipe to deploy the Milvus vector store, in standalone mode.

• PostgreSQL+pgvector: Full recipe to create an instance of PostgreSQL with the pgvector extension, making it usable as a vector store.

Inference and application examples

• Langchain examples: Various notebooks demonstrating how to work with Langchain. Examples are provided for different types of LLM servers (standalone or using the Model Serving stack of NERC OpenShift AI) and different vector databases.

• UI examples: Various examples on how to create and deploy a UI to interact with your LLM.

LLM clients

• AnythingLLM: AnythingLLM is an all-in-one AI application that supports any LLM, any document, and any agent while ensuring full privacy. Here, it is implemented as a NERC RHOAI custom workbench, allowing seamless integration with your LLM and other resources.

setup

This folder contains YAML scripts for setting up local S3 storage (MinIO) based on your requirements - whether you need a basic setup without pre-configured buckets or a setup with buckets and data connections in your NERC RHOAI environment.