Skip to content

DEPLOY.md

Latest commit

 

History

History
320 lines (243 loc) · 10.8 KB

DEPLOY.md

File metadata and controls

320 lines (243 loc) · 10.8 KB
Raw

Deployment guide (production, /agent0 gateway)

This walks through deploying the stack behind an existing nginx reverse proxy at https://<host>/agent0. Assumes nginx runs as a Docker container on a network you can share with the app (same pattern as the /kitabu block in your autogarage.conf).

1. Prepare the env file

On the server, from the project root:

cp .env.prod.example .env.prod

Fill in real values. Minimum you must change:

  • MYSQL_ROOT_PASSWORD — strong random password
  • OTP_SECRET — openssl rand -hex 32
  • PUBLIC_BASE_URL — must be the full public URL including the /agent0 prefix, e.g. https://158.220.86.151/agent0
  • TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, TWILIO_WHATSAPP_NUMBER

.env.prod is gitignored — never commit it.

2. Boot the stack

# Build images and start api + bot + db, detached
docker compose -f docker-compose.prod.yaml up -d --build

# Watch logs
docker compose -f docker-compose.prod.yaml logs -f api bot

Compose creates a private network agent0_default shared by api, bot, and db. Nothing is published to the host — external traffic has to come through nginx.

3. Attach nginx to agent0's network

nginx needs to resolve api and bot by name, so connect its container once:

NGX=$(docker ps --format '{{.Names}}' | grep -i nginx | head -1)
docker network connect agent0_default "$NGX"

# Verify
docker exec "$NGX" getent hosts api     # should print an IP
docker exec "$NGX" getent hosts bot     # should print an IP

This stays in effect across docker compose restart. If you ever recreate the nginx container (e.g. docker compose up --force-recreate in its own project), rerun the network connect command.

Health check from the host via nginx (after step 4):

curl https://158.220.86.151/agent0/health

4. Add the nginx location blocks

In the server block of autogarage.conf, replace the empty location = /agent0/ { … } with two prefix-matched locations. Nginx picks the longest-matching prefix, so /agent0/webhooks/whatsapp/ wins over /agent0/ for Twilio traffic.

# Canonicalise trailing slash
location = /agent0 {
    return 302 /agent0/;
}

# Twilio inbound webhooks → bot service
location /agent0/webhooks/whatsapp/ {
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    client_max_body_size               20M;

    # trailing slash strips /agent0
    proxy_pass http://bot:3004/webhooks/whatsapp/;
}

# Everything else (dashboard, API, extract, reconcile…) → api service
location /agent0/ {
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout                 120s;
    client_max_body_size               60M;

    # trailing slash strips /agent0 before forwarding
    proxy_pass http://api:3003/;
}

Notes:

  • Keep the = modifier only on the canonical-redirect block. The other two must be prefix matches (no =), otherwise /agent0/bot-test-ui/… will fall through to a 404.
  • client_max_body_size 60M on the main block covers the 50 MB PDF upload cap (MAX_UPLOAD_BYTES=52428800) plus multipart overhead.
  • proxy_read_timeout 120s gives tabula-java enough room to parse larger statements.

Reload nginx:

docker exec <nginx-container> nginx -t && docker exec <nginx-container> nginx -s reload

5. Point Twilio at the webhook

In the Twilio console (Programmable Messaging → WhatsApp sandbox, or the production number's messaging config), set the inbound webhook URL to:

https://158.220.86.151/agent0/webhooks/whatsapp/payment

Method: HTTP POST, x-www-form-urlencoded.

6. Verify end-to-end

# Dashboard loads
open https://158.220.86.151/agent0/

# Health + DB connection
curl -s https://158.220.86.151/agent0/health

# Inbound webhook smoke test (simulates Twilio)
curl -s -X POST https://158.220.86.151/agent0/webhooks/whatsapp/payment \
  -d 'From=whatsapp:+254712345678' \
  -d 'Body=MPESA TBC1A2B3C4' \
  -d 'NumMedia=0'

Day-to-day commands

# Update code → rebuild + restart without dropping the DB volume
git pull
docker compose -f docker-compose.prod.yaml up -d --build

# Restart one service (e.g. after .env.prod change)
docker compose -f docker-compose.prod.yaml restart api
docker compose -f docker-compose.prod.yaml restart bot

# Tail logs
docker compose -f docker-compose.prod.yaml logs -f api
docker compose -f docker-compose.prod.yaml logs -f bot

# Run ad-hoc SQL
docker compose -f docker-compose.prod.yaml exec db \
  mysql -uroot -p"$MYSQL_ROOT_PASSWORD" statement_service

# Backup the DB
docker compose -f docker-compose.prod.yaml exec db \
  mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" statement_service \
  > backups/statement_service-$(date +%F).sql

# Stop everything (keeps volumes)
docker compose -f docker-compose.prod.yaml down

# DANGER: stop and WIPE the DB + uploads volumes
docker compose -f docker-compose.prod.yaml down -v

Hourly backups (DB + uploads volume)

scripts/backup.sh snapshots the MySQL database and the uploads named volume into a timestamped directory. Install it once:

# On the server, from the project directory:
sudo mkdir -p /var/backups/agent0 /var/log
sudo chown "$USER" /var/backups/agent0

# Verify manually before committing to cron:
./scripts/backup.sh
ls /var/backups/agent0/

Then add to the user's crontab (crontab -e):

# Agent Zero — hourly DB + uploads backup
0 * * * * cd /path/to/agent0 && ./scripts/backup.sh >> /var/log/agent0-backup.log 2>&1

Layout per snapshot:

/var/backups/agent0/YYYY-MM-DD_HHMM/
  db.sql.gz       # mysqldump --single-transaction, gzipped
  uploads.tar.gz  # full tar of the uploads volume contents

Retention: backup.sh deletes snapshots older than 48 hours by default. Override with BACKUP_RETENTION_HOURS=168 ./scripts/backup.sh (7 days) or set the env in the cron line. Snapshots are local disk — for true disaster recovery, rsync /var/backups/agent0/ to a second host or an S3-compatible bucket on a daily schedule.

Restore from a snapshot:

./scripts/restore.sh /var/backups/agent0/2026-04-18_1400
# Prompts for "RESTORE" to confirm. Restores both db and uploads.
# --db-only or --uploads-only limit the scope.

Continuous deployment (push to main → server)

.github/workflows/deploy.yml runs on every push to main and on manual workflow_dispatch. It:

  1. Runs npm test on a GitHub-hosted Ubuntu runner.
  2. If tests pass, SSHes to the server and runs git reset --hard origin/main + docker compose -f docker-compose.prod.yaml up -d --build.
  3. Polls http://localhost:3003/health inside the api container until it returns 200. Tails logs and fails the job if it doesn't come up in 60s.
  4. Prunes dangling images to keep disk usage in check.

One-time bootstrap on the server

SSH in as the user you'll deploy as (not root; prefer a dedicated deploy user who's in the docker group). Then:

# Clone the repo to the path you'll put in DEPLOY_PATH
sudo mkdir -p /opt/agent0 && sudo chown "$USER" /opt/agent0
cd /opt/agent0
git clone https://github.com/Ngugi1/agent0.git .

# Create the prod env on the server
cp .env.prod.example .env.prod
$EDITOR .env.prod                 # fill in secrets

# First boot (subsequent deploys are handled by CI)
docker compose -f docker-compose.prod.yaml up -d --build

# Attach nginx to agent0's network so it can resolve `api` and `bot`
NGX=$(docker ps --format '{{.Names}}' | grep -i nginx | head -1)
docker network connect agent0_default "$NGX"

The deploy user's shell must be able to run docker compose without sudo — add them to the docker group (sudo usermod -aG docker deploy) and re-login.

Required GitHub repo secrets

In the repo's Settings → Secrets and variables → Actions → New repository secret, add:

Secret Value
SSH_HOST 158.220.86.151
SSH_USER deploy user on the server (e.g. deploy)
SSH_PORT 22 (optional; omit if default)
SSH_PRIVATE_KEY the full contents of an SSH private key whose public half is in ~deploy/.ssh/authorized_keys
DEPLOY_PATH server path where the repo lives (e.g. /opt/agent0)

Generate a dedicated deploy key, don't reuse a personal one:

ssh-keygen -t ed25519 -f agent0_deploy -C "agent0 ci deploy"
# copy agent0_deploy.pub to the server (append to ~deploy/.ssh/authorized_keys)
# paste the contents of agent0_deploy (the private key) into the
# SSH_PRIVATE_KEY secret

Manual deploys

From the Actions tab, run the Deploy to production workflow via Run workflow on main. Useful for rerunning after a config change that didn't touch code.

What stays on the server vs. what deploys

The CI job only touches the git working tree and docker state. It does not overwrite .env.prod, the uploads volume, or the db_data volume. That means secrets, accumulated statements, and the MySQL data directory all survive a deploy.

Troubleshooting

  • Cannot POST /properties from nginx → the /agent0/ block is missing the trailing slash on proxy_pass http://api:3003/;. Without it the /agent0 prefix is forwarded intact and the API doesn't have that route.
  • api container can't resolve bot → both services must be on the internal network; check docker network inspect agent0_internal.
  • nginx can't resolve api/bot → the nginx container isn't on the agent0_default network. Run docker network connect agent0_default <nginx-container> and reload nginx. If nginx gets recreated (not just restarted), you must rerun this.
  • Twilio 63016 outside messaging window → the sandbox only permits freeform outbound within 24h of the tenant's last inbound. Ask them to reply once, or use an approved template.
  • Invoice PDFs blank → shouldn't happen after alpha-0, but if it does check uploads volume is mounted and the API log for PDFKit errors.
  • CI deploy fails at SSH step → double-check SSH_PRIVATE_KEY has the full PEM including -----BEGIN… / -----END… lines, and that the public half is in ~<deploy-user>/.ssh/authorized_keys with mode 600 on the file and 700 on .ssh. Test with ssh -i agent0_deploy deploy@<host> from your laptop.
  • CI deploy fails at docker compose → the deploy user likely isn't in the docker group. sudo usermod -aG docker <user> and re-login (or newgrp docker).
  • Health probe times out → most often the DB container is still initialising on first boot. Rerun the workflow; subsequent runs reuse the db_data volume and are fast.