Enhancement: backup docs for LibreChat.ai install

[kurtseifried]

What features would you like to see added?

I'd like to write some backup and restore docs and script snippet examples for librechat (independent of how it is installed, e.g. podman/docker/local/whatever), can I suggest under "User Guides" we add a backup section?

e.g. it's actually pretty easy to grab all the data from mongo, if using docker for example:

mongodump -h MongoDB_server -d LibreChat -o [BACKUP DIR]

and you can get MongoDB server depending on how you installed it, e.g. docker:

docker ps | grep "chat-mongodb$" | cut -d" " -f1 | xargs docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}'

the mongodump gets you all your mongo data, and you can use bsondump to inspect the bson files easily and convert them to text suitable for checking into a git repo for example.

That and your librechat.yaml and .env file, what else is needed to backup and restore a system?

Also need to grab vector, meilisearch, the logs, etc.

What is the correct way to go about doing this, e.g. discuss it, submit a PR, something else?

More details

I'd like to write this document since I need to backup my installs anyways and it's good to share. I suspect a lot more people will want to use librechat in production like myself which means backups, exporting/importing data and so on.

Which components are impacted by your request?

General

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Sure, PRs are welcome on the docs repo: https://github.com/LibreChat-AI/librechat.ai

[danny-avila]

Was running out of the house when I first saw this, I can answer some of the questions specifically

That and your librechat.yaml and .env file, what else is needed to backup and restore a system?

What you listed is pretty much it but I think we would need to backup the vectors from postgresql, not sure what the process would be. Technically, meilisearch could be backed up too, but the indices get rebuilt rather quickly when you migrate so it's not really necessary.

What is the correct way to go about doing this, e.g. discuss it, submit a PR, something else?

Feel free to submit a PR, this specific link goes over how to get started. You can basically run the website locally and even preview how it would look like:

https://www.librechat.ai/docs/documentation

[kurtseifried]

Ok some initial notes:

This needs an option to backup EVERYTHING, and to also remove API keys/SECRET/etc.

Files

Config files

LibreChat/.env - needs an option to remove the API keys
LibreChat/librechat.yaml - needs an options to remove the supportedIds and so on

Log files

Docker: LibreChat/logs/

MongoDB

https://www.mongodb.com/docs/database-tools/mongodump/

Dump the entire MongoDB:

mongodump -h [MONGODBIP] -d [DATABASE]

Find the database name

Search .env:

MONGO_URI=mongodb://127.0.0.1:27017/LibreChat

Find the IP

If MONGO_URI is NOT 127.0.0.1 then we can probably use that IP.

Docker:

`docker ps | grep "mongo" | cut -d" " -f1 | xargs docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}'`

Meili search

https://www.meilisearch.com/docs/learn/advanced/snapshots_vs_dumps

Snapshots are highly efficient, but not portable between different versions of Meilisearch. Use snapshots for periodic data backups.

Dumps are portable between different Meilisearch versions, but not very efficient. Use dumps when updating to a new Meilisearch release.

Vector database

using pgvector, so PostgreSQL https://www.postgresql.org/docs/current/app-pgdump.html

rag.yml has the default configs for it.

Find the IP:

`docker ps | grep "ankane/pgvector:latest" | cut -d" " -f1 | xargs docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}'`

PGPASSWORD=mypassword pg_dump -U myuser -d mydatabase -f [OUTPUT_FILE] -h [HOSTNAME]

[kurtseifried]

Meilisearch backup:

MEILIESEARCH_HOST=`docker ps | grep "getmeili/meilisearch:v1.7.3" | cut -d" " -f1 | xargs docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}'`

# Get MEILI_MASTER_KEY from .env
MEILI_MASTER_KEY=$(grep -m 1 -E '^MEILI_MASTER_KEY=' .env | sed 's/^MEILI_MASTER_KEY=//')

curl -s -X POST "http://$MEILIESEARCH_HOST:7700/dumps" -H "Authorization: Bearer $MEILI_MASTER_KEY"

# Which will return something like:

# {"taskUid":85,"indexUid":null,"status":"enqueued","type":"dumpCreation","enqueuedAt":"2024-06-10T19:34:47.774316365Z"}

# To check on the status of a backup:
# curl -s -X GET "http://$MEILIESEARCH_HOST:7700/tasks/86" -H "Authorization: Bearer $MEILI_MASTER_KEY"
# and if "status": "succeeded",

# The data will end up in meili_data_v1.7/dumps/ or equivalent with a file like 20240610-193753475.dump that is a gzip compressed tarball with the files:

#./experimental-features.json
#./indexes/convos/documents.jsonl
#./indexes/convos/metadata.json
#./indexes/convos/settings.json
#./indexes/messages/documents.jsonl
#./indexes/messages/metadata.json
#./indexes/messages/settings.json
#./keys.jsonl
#./metadata.json
#./tasks/queue.jsonl

Please note that the export dir is already protected in .gitignore: meili_data*