LabsCrypt · Smartdevs17 · May 27, 2026
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
@@ -1322,3 +1322,7 @@ graph TB
 ---
 
 For implementation details, see the code in respective directories. For contribution guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md).
+
+### Runbooks
+
+Operational runbooks for on-call engineers are available in [docs/runbooks/](docs/runbooks/). See the [Indexer Recovery Runbook](docs/runbooks/indexer-recovery.md) for procedures on handling indexer lag, RPC outages, and quarantined events.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -11,6 +11,7 @@ This document provides a set of guidelines for contributing to RemitLend and its
 - [Branching Strategy](#branching-strategy)
 - [Commit Message Guidelines](#commit-message-guidelines)
 - [Pull Request Standards](#pull-request-standards)
+- [Environment Variables](#environment-variables)
 - [Testing Requirements](#testing-requirements)
 - [Style Guides](#style-guides)
 
@@ -91,6 +92,10 @@ When opening a PR, ensure your description includes:
     - [ ] Documentation has been updated.
     - [ ] Commit messages follow standards.
 
+## Environment Variables
+
+Before setting up the project locally, review the full environment variable reference in [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md). Each `.env.example` file contains a pointer to this canonical reference. If you add a new environment variable, update both the relevant `.env.example` and the table in `ENVIRONMENT.md`.
+
 ## Testing Requirements
 
 Before submitting, verify your changes by running:

diff --git a/README.md b/README.md
@@ -192,6 +192,10 @@ The repository is organized as a monorepo containing three core packages:
 
 We welcome contributions from developers of all skill levels! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines on how to get started.
 
+### Environment Variables
+
+See [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) for a full reference of all environment variables across backend, frontend, and scripts. Each `.env.example` file also links to this document.
+
 ### Quick Contribution Guide
 
 1. Fork the repository.

diff --git a/backend/.env.example b/backend/.env.example
@@ -1,3 +1,4 @@
+# See docs/ENVIRONMENT.md for full reference
 CORS_ALLOWED_ORIGINS=http://localhost:3000,http://localhost:3001
 FRONTEND_URL=http://localhost:3000
 

diff --git a/backend/src/routes/loanRoutes.ts b/backend/src/routes/loanRoutes.ts
@@ -12,6 +12,7 @@ import {
   repayLoan,
   submitTransaction,
 } from "../controllers/loanController.js";
+import { getLoanEvents } from "../controllers/indexerController.js";
 import {
   requireJwtAuth,
   requireScopes,
@@ -213,6 +214,48 @@ router.get(
   getLoanAmortizationSchedule,
 );
 
+/**
+ * @swagger
+ * /loans/{loanId}/events:
+ *   get:
+ *     summary: Get events for a specific loan
+ *     description: >
+ *       Returns chronological loan events for the authenticated borrower.
+ *     tags: [Loans]
+ *     security:
+ *       - BearerAuth: []
+ *     parameters:
+ *       - in: path
+ *         name: loanId
+ *         required: true
+ *         schema:
+ *           type: integer
+ *         description: Loan ID
+ *       - in: query
+ *         name: limit
+ *         schema:
+ *           type: integer
+ *           default: 50
+ *       - in: query
+ *         name: cursor
+ *         schema:
+ *           type: string
+ *     responses:
+ *       200:
+ *         description: Loan events retrieved successfully
+ *       401:
+ *         description: Missing or invalid Bearer token
+ *       404:
+ *         description: Loan not found or not accessible
+ */
+router.get(
+  "/:loanId/events",
+  requireJwtAuth,
+  requireScopes("read:loans"),
+  requireLoanBorrowerAccess,
+  getLoanEvents,
+);
+
 /**
  * @swagger
  * /loans/request:

diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
@@ -0,0 +1,100 @@
+# Environment Variable Reference
+
+This document lists every environment variable used by the RemitLend platform. Each table covers one package.
+
+---
+
+## Backend (`backend/`)
+
+| Variable | Dev | Staging | Prod | Default | Description | Source |
+|---|---|---|---|---|---|---|
+| `CORS_ALLOWED_ORIGINS` | ✓ | ✓ | ✓ | `http://localhost:3000,http://localhost:3001` | Comma-separated origins allowed by CORS | `backend/src/config/index.ts` |
+| `FRONTEND_URL` | ✓ | ✓ | ✓ | `http://localhost:3000` | Frontend base URL used for links | `backend/src/config/index.ts` |
+| `DATABASE_URL` | ✓ | ✓ | ✓ | `postgres://postgres:postgres@db:5432/remitlend` | PostgreSQL connection string | `backend/src/db/connection.js` |
+| `REDIS_URL` | ✓ | ✓ | ✓ | `redis://redis:6379` | Redis connection string | `backend/src/services/cacheService.ts` |
+| `STELLAR_NETWORK` | ✓ | ✓ | ✓ | `testnet` | Stellar network name (`testnet`, `pubnet`, `sandbox`) | `backend/src/config/stellar.ts` |
+| `STELLAR_RPC_URL` | ✓ | ✓ | ✓ | `https://soroban-testnet.stellar.org` | Soroban RPC endpoint | `backend/src/config/stellar.ts` |
+| `STELLAR_NETWORK_PASSPHRASE` | ✓ | ✓ | ✓ | `Test SDF Network ; September 2015` | Network passphrase for transaction signing | `backend/src/config/stellar.ts` |
+| `LOAN_MANAGER_CONTRACT_ID` | ✓ | ✓ | ✓ | — | Deployed loan manager contract address | `backend/src/config/stellar.ts` |
+| `REMITTANCE_NFT_CONTRACT_ID` | — | ✓ | ✓ | — | Deployed remittance NFT contract address | `backend/src/config/contracts.ts` |
+| `LENDING_POOL_CONTRACT_ID` | ✓ | ✓ | ✓ | — | Deployed lending pool contract address | `backend/src/config/stellar.ts` |
+| `MULTISIG_GOVERNANCE_CONTRACT_ID` | — | ✓ | ✓ | — | Deployed multisig governance contract address | `backend/src/config/contracts.ts` |
+| `POOL_TOKEN_ADDRESS` | ✓ | ✓ | ✓ | — | Pool token contract address | `backend/src/config/stellar.ts` |
+| `STELLAR_USDC_ISSUER` | — | ✓ | ✓ | — | USDC asset issuer address | `backend/src/config/stellar.ts` |
+| `STELLAR_EURC_ISSUER` | — | ✓ | ✓ | — | EURC asset issuer address | `backend/src/config/stellar.ts` |
+| `STELLAR_PHP_ISSUER` | — | ✓ | ✓ | — | PHP asset issuer address | `backend/src/config/stellar.ts` |
+| `LOAN_MANAGER_ADMIN_SECRET` | ✓ | ✓ | ✓ | — | Admin secret key for loan manager operations | `backend/src/config/stellar.ts` |
+| `SCORE_RECONCILIATION_SOURCE_SECRET` | — | ✓ | ✓ | — | Secret key for score reconciliation operations | `backend/src/services/scoreService.ts` |
+| `LOAN_MIN_SCORE` | ✓ | ✓ | ✓ | `500` | Minimum credit score to request a loan | `backend/src/config/loans.ts` |
+| `LOAN_MAX_AMOUNT` | ✓ | ✓ | ✓ | `50000` | Maximum loan amount in USD | `backend/src/config/loans.ts` |
+| `LOAN_INTEREST_RATE_PERCENT` | ✓ | ✓ | ✓ | `12` | Annual interest rate percentage | `backend/src/config/loans.ts` |
+| `CREDIT_SCORE_THRESHOLD` | ✓ | ✓ | ✓ | `600` | Threshold for loan approval score | `backend/src/config/loans.ts` |
+| `SCORE_DELTA_REPAY` | ✓ | ✓ | ✓ | `15` | Points added to score on timely repayment | `backend/src/config/scores.ts` |
+| `SCORE_DELTA_DEFAULT` | ✓ | ✓ | ✓ | `50` | Points deducted on default | `backend/src/config/scores.ts` |
+| `SCORE_DELTA_LATE` | ✓ | ✓ | ✓ | `5` | Points deducted on late payment | `backend/src/config/scores.ts` |
+| `INDEXER_POLL_INTERVAL_MS` | ✓ | ✓ | ✓ | `30000` | Event indexer poll interval in milliseconds | `backend/src/config/indexer.ts` |
+| `INDEXER_BATCH_SIZE` | ✓ | ✓ | ✓ | `100` | Events fetched per poll cycle | `backend/src/config/indexer.ts` |
+| `DEFAULT_CHECK_INTERVAL_MS` | ✓ | ✓ | ✓ | `1800000` | Default checker interval (30 min) | `backend/src/services/defaultChecker.ts` |
+| `DEFAULT_CHECK_MAX_LOANS_PER_RUN` | ✓ | ✓ | ✓ | `500` | Max loans processed per default check run | `backend/src/services/defaultChecker.ts` |
+| `DEFAULT_CHECK_BATCH_SIZE` | ✓ | ✓ | ✓ | `25` | Loans per batch during default check | `backend/src/services/defaultChecker.ts` |
+| `DEFAULT_CHECK_BATCH_TIMEOUT_MS` | ✓ | ✓ | ✓ | `300000` | Timeout per batch (5 min) | `backend/src/services/defaultChecker.ts` |
+| `DEFAULT_CHECK_CONCURRENCY` | ✓ | ✓ | ✓ | `3` | Concurrent check workers | `backend/src/services/defaultChecker.ts` |
+| `DEFAULT_CHECK_POLL_ATTEMPTS` | ✓ | ✓ | ✓ | `30` | Max poll attempts per check | `backend/src/services/defaultChecker.ts` |
+| `DEFAULT_CHECK_POLL_SLEEP_MS` | ✓ | ✓ | ✓ | `1000` | Sleep between poll attempts | `backend/src/services/defaultChecker.ts` |
+| `LOAN_TERM_LEDGERS` | ✓ | ✓ | ✓ | `17280` | Default loan term in ledgers (~30 days) | `backend/src/config/loans.ts` |
+| `SCORE_RECONCILIATION_INTERVAL_MS` | ✓ | ✓ | ✓ | `3600000` | Score reconciliation interval | `backend/src/config/scores.ts` |
+| `SCORE_RECONCILIATION_MAX_BORROWERS_PER_RUN` | ✓ | ✓ | ✓ | `500` | Max borrowers per reconciliation run | `backend/src/config/scores.ts` |
+| `SCORE_RECONCILIATION_BATCH_SIZE` | ✓ | ✓ | ✓ | `25` | Borrowers per batch in reconciliation | `backend/src/config/scores.ts` |
+| `SCORE_RECONCILIATION_AUTOCORRECT_ENABLED` | ✓ | ✓ | ✓ | `false` | Enable automatic score correction | `backend/src/config/scores.ts` |
+| `SCORE_RECONCILIATION_AUTOCORRECT_THRESHOLD` | ✓ | ✓ | ✓ | `50` | Max points auto-corrected per run | `backend/src/config/scores.ts` |
+| `JWT_SECRET` | ✓ | ✓ | ✓ | `your-super-secret-jwt-key-change-in-production` | JWT signing/verification secret | `backend/src/middleware/jwtAuth.ts` |
+| `INTERNAL_API_KEY` | ✓ | ✓ | ✓ | `change-me` | API key for internal endpoints | `backend/src/middleware/auth.ts` |
+| `WEBHOOK_REQUEST_TIMEOUT_MS` | ✓ | ✓ | ✓ | `30000` | Outgoing webhook request timeout | `backend/src/services/webhookService.ts` |
+| `SENTRY_DSN` | — | ✓ | ✓ | — | Sentry DSN for backend error tracking | `backend/src/app.ts` |
+| `NOTIFICATION_RETENTION_DAYS` | ✓ | ✓ | ✓ | `90` | Days to keep unread notifications | `backend/src/services/notificationService.ts` |
+| `READ_NOTIFICATION_RETENTION_DAYS` | ✓ | ✓ | ✓ | `30` | Days to keep read notifications | `backend/src/services/notificationService.ts` |
+| `SENDGRID_API_KEY` | — | ✓ | ✓ | — | SendGrid API key for email | `backend/src/services/emailService.ts` |
+| `FROM_EMAIL` | — | ✓ | ✓ | — | Sender email address | `backend/src/services/emailService.ts` |
+| `ADMIN_EMAIL` | — | ✓ | ✓ | — | Admin notification email | `backend/src/services/notificationService.ts` |
+| `ADMIN_WEBHOOK_URL` | — | ✓ | ✓ | — | Admin notification webhook URL | `backend/src/services/notificationService.ts` |
+| `TWILIO_ACCOUNT_SID` | — | ✓ | ✓ | — | Twilio account SID for SMS | `backend/src/services/smsService.ts` |
+| `TWILIO_AUTH_TOKEN` | — | ✓ | ✓ | — | Twilio auth token | `backend/src/services/smsService.ts` |
+| `TWILIO_PHONE_NUMBER` | — | ✓ | ✓ | — | Twilio sender phone number | `backend/src/services/smsService.ts` |
+
+---
+
+## Frontend (`frontend/`)
+
+| Variable | Dev | Staging | Prod | Default | Description | Source |
+|---|---|---|---|---|---|---|
+| `NEXT_PUBLIC_API_URL` | ✓ | ✓ | ✓ | `http://localhost:3001` | Backend API base URL | `frontend/src/app/hooks/useApi.ts` |
+| `NEXT_PUBLIC_SENTRY_DSN` | — | ✓ | ✓ | — | Sentry DSN for frontend error tracking | `frontend/src/sentry.client.config.ts` |
+| `SENTRY_DSN` | — | ✓ | ✓ | — | Sentry DSN server-side | `frontend/src/sentry.server.config.ts` |
+| `SENTRY_ORG` | — | ✓ | ✓ | — | Sentry organization slug | `frontend/sentry.client.config.ts` |
+| `SENTRY_PROJECT` | — | ✓ | ✓ | — | Sentry project slug | `frontend/sentry.client.config.ts` |
+| `SENTRY_AUTH_TOKEN` | — | ✓ | ✓ | — | Sentry auth token for source maps | `frontend/next.config.ts` |
+| `NODE_ENV` | ✓ | ✓ | ✓ | `development` | Node environment (`development`, `test`, `production`) | `next.config.ts` |
+| `NEXT_PUBLIC_STELLAR_EXPLORER_URL` | ✓ | ✓ | ✓ | `https://stellar.expert/explorer/testnet` | Stellar explorer base URL for transaction links | `frontend/src/components/ui/TxHashLink.tsx` |
+
+---
+
+## Contracts / Scripts (`contracts/`, `scripts/`)
+
+| Variable | Dev | Staging | Prod | Default | Description | Source |
+|---|---|---|---|---|---|---|
+| `SOROBAN_RPC_URL` | ✓ | ✓ | ✓ | `https://soroban-testnet.stellar.org` | RPC URL for contract deployment | `scripts/deploy.ts` |
+| `SOROBAN_NETWORK_PASSPHRASE` | ✓ | ✓ | ✓ | `Test SDF Network ; September 2015` | Network passphrase for contract operations | `scripts/deploy.ts` |
+| `SOROBAN_ACCOUNT` | ✓ | ✓ | ✓ | — | Deployer account secret key | `scripts/deploy.ts` |
+| `DEPLOY_CONFIG_PATH` | — | ✓ | ✓ | `scripts/deploy-config.json` | Path to deploy configuration | `scripts/deploy.ts` |
+
+---
+
+## `.env.example` vs `ENVIRONMENT.md` Drift
+
+A CI job (`env-docs-check`) runs on every PR to ensure the keys listed in `.env.example` files are present in this document. The check performs a sorted diff and fails if any key is missing from either side.
+
+To update this document after adding a new environment variable:
+
+1. Add the variable to the relevant `.env.example` file.
+2. Add a row to the table above with all columns filled.
+3. The CI job will pass automatically.
diff --git a/docs/runbooks/README.md b/docs/runbooks/README.md
@@ -0,0 +1,11 @@
+# Runbooks
+
+Operational runbooks for on-call engineers working on the RemitLend platform.
+
+## Index
+
+- [Indexer Recovery](indexer-recovery.md) — Responding to indexer lag, RPC outages, and quarantined events.
+
+## Purpose
+
+These runbooks provide step-by-step procedures for diagnosing and resolving common production incidents. They are meant to be followed in order during an incident, with clear escalation points at each stage.
diff --git a/docs/runbooks/indexer-recovery.md b/docs/runbooks/indexer-recovery.md
@@ -0,0 +1,178 @@
+# Indexer Recovery Runbook
+
+When the event indexer falls behind, crashes, or encounters an RPC outage, use this runbook to restore normal operation.
+
+---
+
+## 1. Detecting Indexer Lag
+
+### Via Health Endpoint
+
+```bash
+curl /api/indexer/status
+```
+
+Check the `last_indexed_ledger` field against the current Stellar ledger sequence (available from the RPC `getLatestLedger` method). A gap larger than `INDEXER_POLL_INTERVAL_MS × 2` indicates lag.
+
+### Via Prometheus Metrics (future)
+
+Once Prometheus is deployed, alert on:
+
+- `indexer_lag_ledgers > 100`
+- `indexer_last_indexed_timestamp > 5 minutes ago`
+
+### Via Database Query
+
+```sql
+SELECT
+  (SELECT MAX(ledger) FROM contract_events) AS last_indexed_ledger,
+  (SELECT MAX(ledger_closed_at) FROM contract_events) AS last_indexed_timestamp;
+```
+
+Compare the timestamp to `NOW()`. A difference > 5 minutes suggests the indexer is stuck or has crashed.
+
+---
+
+## 2. Safe Pause / Resume
+
+### Pause the Indexer
+
+If the indexer is consuming too many resources or if you need to investigate:
+
+```bash
+curl -X POST /api/admin/indexer/pause \
+  -H "x-api-key: ${INTERNAL_API_KEY}"
+```
+
+The indexer will finish its current poll cycle and then stop. In-flight events are not lost because the indexer tracks the last indexed ledger in the database.
+
+### Verify Paused State
+
+```bash
+curl /api/indexer/status
+```
+
+Look for `status: "paused"`.
+
+### Resume the Indexer
+
+```bash
+curl -X POST /api/admin/indexer/resume \
+  -H "x-api-key: ${INTERNAL_API_KEY}"
+```
+
+The indexer resumes from the last indexed ledger stored in `indexer_state`.
+
+---
+
+## 3. Using `reindex-ledger-range`
+
+When events are missing or corrupted for a specific ledger range, you can re-index a range of ledgers.
+
+### Syntax
+
+```bash
+curl -X POST /api/admin/indexer/reindex \
+  -H "Content-Type: application/json" \
+  -H "x-api-key: ${INTERNAL_API_KEY}" \
+  -d '{
+    "startLedger": 123456,
+    "endLedger": 123500,
+    "contractIds": ["CA...", "CB..."]
+  }'
+```
+
+### When to Use
+
+- A batch of events was mis-decoded due to a schema upgrade.
+- The `contract_events` table has gaps in a known ledger range.
+- An RPC node returned incomplete results and you need to retry.
+
+### Behaviour
+
+1. The indexer resets its `last_indexed_ledger` to `startLedger - 1` for the affected contracts.
+2. Events in the range are re-fetched and upserted (`ON CONFLICT (event_id) DO NOTHING`).
+3. The indexer resumes normal polling from `endLedger + 1`.
+
+---
+
+## 4. Inspecting and Reprocessing Quarantined Events
+
+Events that fail decoding or validation are moved to a quarantine table.
+
+### View Quarantined Events
+
+```bash
+curl /api/admin/indexer/quarantine \
+  -H "x-api-key: ${INTERNAL_API_KEY}"
+```
+
+Response includes:
+
+- `event_id` — original Soroban event ID
+- `ledger` — ledger sequence
+- `error` — the error message that caused quarantine
+- `raw_payload` — raw XDR base64 for debugging
+
+### Reprocess a Quarantined Event
+
+Once the issue is resolved (e.g., a decoding bug is fixed):
+
+```bash
+curl -X POST /api/admin/indexer/quarantine/{event_id}/reprocess \
+  -H "x-api-key: ${INTERNAL_API_KEY}"
+```
+
+The event is re-decoded and inserted into `contract_events`. If it fails again, it stays in quarantine.
+
+### Bulk Reprocess All
+
+```bash
+curl -X POST /api/admin/indexer/quarantine/reprocess-all \
+  -H "x-api-key: ${INTERNAL_API_KEY}"
+```
+
+Use this after a hotfix deployment to clear the quarantine backlog.
+
+---
+
+## 5. Handling an RPC Outage
+
+### Symptoms
+
+- `/api/indexer/status` returns `rpc_status: "unreachable"`.
+- Backend logs show repeated `ERR_HTTP_REQUEST` or timeout errors from the Soroban RPC.
+
+### Steps
+
+1. **Verify RPC availability** from a separate host:
+   ```bash
+   curl -X POST <STELLAR_RPC_URL> \
+     -H "Content-Type: application/json" \
+     -d '{"jsonrpc":"2.0","id":1,"method":"getHealth"}'
+   ```
+
+2. **Failover to a secondary RPC** if available:
+   - Update the `STELLAR_RPC_URL` environment variable.
+   - Restart the indexer via pause/resume or container restart.
+
+3. **If no secondary RPC is available:**
+   - Pause the indexer (see Section 2).
+   - Monitor the RPC provider status page.
+   - Once RPC is restored, resume the indexer. The gap will be caught up automatically as the indexer polls from the last indexed ledger.
+
+4. **If the outage exceeds 1 hour:**
+   - Consider running the `reindex-ledger-range` command (Section 3) after the RPC is restored to ensure no events were missed during the outage window.
+
+---
+
+## 6. Escalation Contacts
+
+For incidents that cannot be resolved with the steps above, escalate via the [contributor Telegram group](https://t.me/+DOylgFv1jyJlNzM0).
+
+When escalating, include:
+
+- Ledger range of the gap
+- Indexer status JSON output
+- Relevant backend log excerpts (redact any secrets)
+- Steps already attempted
diff --git a/docs/wiki/indexer-sync-flow.md b/docs/wiki/indexer-sync-flow.md
@@ -62,3 +62,6 @@ The indexer runs as a background process (`indexerManager.ts`) that initiates pe
 - `backend/src/services/eventIndexer.ts`: Core logic for polling and processing.
 - `backend/src/services/indexerManager.ts`: Lifecycle management for the indexer.
 - `backend/src/db/connection.js`: Database connection and query execution.
+
+## Related Documentation
+- [Indexer Recovery Runbook](../runbooks/indexer-recovery.md) — Procedures for handling indexer lag, RPC outages, and quarantined events.