Added a full guarded notification for all finacial transaction notifi…

Backend/DOCUMENTATION/outbox-changes.md

@@ -0,0 +1,122 @@
+# Transactional Outbox Implementation — Changes Documentation
+
+## Overview
+
+This document explains the changes made to implement a transactional outbox pattern for reliable financial and notification job processing in the Vaulty backend.
+
+## Problem Statement
+
+Previously, the backend used a direct BullMQ approach where database writes and queue submissions were separate operations. This created a race condition where:
+
+- A database write could succeed while queue submission failed, causing lost verification emails, duplicate payment processing, stale streak calculations, or notifications referring to records that do not exist yet.
+- A job could run before its related database transaction committed, leading to inconsistent state.
+
+## Solution: Transactional Outbox Pattern
+
+The transactional outbox pattern ensures that domain changes and outbox events are written atomically in the same Prisma transaction. A dedicated outbox processor then safely publishes unpublished events to BullMQ queues.
+
+## Files Changed
+
+### New Files
+
+| File | Purpose |
+|---|---|
+| `Backend/src/repositories/outbox.repository.ts` | CRUD operations for `OutboxEvent` records — create, find pending, mark published/failed/dead-letter, reset failed events |
+| `Backend/src/services/notification.service.ts` | Notification service with transactional outbox integration — creates notifications and outbox events in the same Prisma transaction |
+| `Backend/src/jobs/outbox.processor.ts` | Dedicated processor that polls pending outbox events and publishes them to the correct BullMQ queue with idempotency and retry support |
+| `Backend/tests/unit/outbox.repository.test.ts` | Unit tests for the outbox repository — mocking Prisma and verifying CRUD operations |
+| `Backend/tests/integration/outbox.integration.test.ts` | Integration tests verifying transactional atomicity, failed queue publication does not lose events, idempotency, and dead-letter transitions |
+| `Backend/DOCUMENTATION/outbox-changes.md` | This file — explains all changes made |
+
+### Modified Files
+
+| File | Changes |
+|---|---|
+| `Backend/prisma/schema.prisma` | Added `OutboxEvent` model with fields: `id`, `eventType`, `aggregateId`, `aggregateType`, `payload`, `attemptCount`, `maxAttempts`, `status`, `nextRetryAt`, `publishedAt`, `failedAt`, `deadLetterReason`, `createdAt`, `updatedAt`. Added `OutboxEventStatus` and `OutboxEventType` enums. Added database indexes on `(status, nextRetryAt)`, `(aggregateId, aggregateType)`, and `(eventType)`. |
+| `Backend/src/queues/index.ts` | Added `OUTBOX_PROCESSOR` queue name and `getOutboxProcessorQueue()` getter |
+| `Backend/src/jobs/index.ts` | Added `initializeOutboxProcessor()` and `stopOutboxProcessorSafe()` functions; integrated outbox processor lifecycle with server startup/shutdown |
+| `Backend/src/services/auth.service.ts` | Replaced direct `queueVerificationEmail`/`queuePasswordResetEmail` calls with Prisma `$transaction` blocks that write outbox events atomically with domain state |
+| `Backend/src/services/vault.service.ts` | Wrapped vault creation, deposit, withdrawal, lock, unlock, and close operations in Prisma `$transaction` blocks that write outbox events alongside domain data |
+| `Backend/src/services/payment.service.ts` | Wrapped payment initiation (deposit/withdrawal) and instruction requests in Prisma `$transaction` blocks that write outbox events alongside domain data |
+| `Backend/src/services/notification.service.ts` | New service — creates notifications and outbox events in the same Prisma transaction |
+| `Backend/src/repositories/user.repository.ts` | Added `createOutboxEvent()` helper method for convenience |
+| `Backend/src/server.ts` | Added `stopOutboxProcessorSafe()` call in shutdown sequence; imported outbox processor lifecycle functions |
+
+## Architecture
+
+### Write Path
+
+```
+Business Service
+  └─ prisma.$transaction(async (tx) => {
+       ├─ tx.domainModel.create(...)   // e.g., user, payment, vault transaction
+       └─ tx.outboxEvent.create(...)   // same transaction, atomic
+     })
+```
+
+### Publish Path
+
+```
+Outbox Processor (continuous loop)
+  ├─ Find pending events (status=PENDING, nextRetryAt <= now, attemptCount < 5)
+  ├─ For each event:
+  │   ├─ Determine target queue from eventType
+  │   ├─ Publish to BullMQ queue
+  │   ├─ On success: mark PUBLISHED
+  │   ├─ On transient failure: mark FAILED with exponential backoff
+  │   └─ On max retries: mark DEAD_LETTER
+  └─ Sleep 1s when no pending events, then repeat
+```
+
+### Shutdown Path
+
+```
+SIGTERM / SIGINT
+  ├─ stopOutboxProcessorSafe()   // stop polling for new events
+  ├─ disconnectPrisma()          // close DB connections
+  ├─ disconnectRedis()           // close Redis connections
+  └─ closeQueueConnections()     // close BullMQ queues
+```
+
+## Idempotency
+
+Each outbox event is keyed by `(aggregateId, eventType)`. The `findByIdempotencyKey` method checks for existing pending or published events before creating a new one, preventing duplicate queue submissions on retries.
+
+## Retry Behavior
+
+- **Max attempts**: 5
+- **Backoff**: Exponential starting at 5 seconds, doubling each attempt, capped at 300 seconds
+- **Scheduling**: Failed events get a `nextRetryAt` timestamp; the processor only picks up events whose retry time has arrived
+- **Terminal state**: After 5 failed attempts, the event is moved to `DEAD_LETTER` with a `deadLetterReason`
+
+## Event Types
+
+| Event Type | Queue | Aggregate |
+|---|---|---|
+| `EMAIL_VERIFICATION` | `email` | User |
+| `PASSWORD_RESET` | `email` | User |
+| `EMAIL_RESEND` | `email` | User |
+| `PAYMENT_INITIATED` | `payment-processing` | Payment |
+| `PAYMENT_INSTRUCTIONS` | `payment-processing` | Payment |
+| `PAYMENT_STATUS_UPDATE` | `payment-processing` | Payment |
+| `VAULT_DEPOSIT` | `vault-reconciliation` | VaultTransaction |
+| `VAULT_WITHDRAWAL` | `vault-reconciliation` | VaultTransaction |
+| `VAULT_LOCK` | `vault-reconciliation` | SavingsVault |
+| `VAULT_UNLOCK` | `vault-reconciliation` | SavingsVault |
+| `VAULT_CLOSE` | `vault-reconciliation` | SavingsVault |
+| `NOTIFICATION` | `notifications` | Notification |
+| `STREAK_UPDATE` | `streak-calculation` | User |
+| `RECONCILIATION` | `stellar-confirmation` | Transaction |
+
+## Monitoring
+
+- **Outbox lag**: Count of `PENDING` events with `nextRetryAt <= now`
+- **Dead-letter queue**: Count of `DEAD_LETTER` events (should be zero)
+- **Publish failure rate**: Ratio of `FAILED` events to total published events
+- **Event age**: `PENDING` events older than 5 minutes indicate processor issues
+
+## Recovery
+
+- **Automatic**: Failed events are retried automatically with exponential backoff
+- **Manual**: Dead-letter events can be inspected via SQL and re-published by resetting their status to `PENDING`
+- **Reset script**: A one-off script can reset `FAILED` events past their retry time back to `PENDING` status

Backend/README.md

@@ -961,6 +961,93 @@ Redis is used for:
 * Queue management
 * Temporary data
 * Rate limiting
+* Outbox event publishing
+
+---
+
+# Transactional Outbox Pattern
+
+The backend implements a transactional outbox pattern to ensure reliable delivery of financial and notification jobs. This pattern guarantees that database state changes and queue enqueuing are atomic, preventing lost emails, duplicate payments, stale streak calculations, or notifications referring to non-existent records.
+
+## How It Works
+
+1. **Write Phase**: Business services write domain changes and an `OutboxEvent` record inside the same Prisma `$transaction`. This ensures the event is persisted atomically with the domain data.
+2. **Publish Phase**: A dedicated outbox processor (`outbox.processor.ts`) polls for pending events and publishes them to the correct BullMQ queue.
+3. **Idempotency**: Each outbox event is keyed by `aggregateId` + `eventType`, preventing duplicate queue submissions on retries.
+4. **Retry with Backoff**: Failed publish attempts are recorded with exponential backoff (`5s`, `10s`, `20s`, `40s`, `80s`), up to a maximum of 5 attempts.
+5. **Dead Letter**: Events exceeding the retry limit are moved to a `DEAD_LETTER` terminal state for manual inspection and recovery.
+
+## Event Types
+
+| Event Type | Queue | Description |
+|---|---|---|
+| `EMAIL_VERIFICATION` | `email` | Sends a verification email on user registration |
+| `PASSWORD_RESET` | `email` | Sends a password reset email |
+| `EMAIL_RESEND` | `email` | Resends a verification email |
+| `PAYMENT_INITIATED` | `payment-processing` | Triggers payment polling after deposit/withdrawal initiation |
+| `PAYMENT_INSTRUCTIONS` | `payment-processing` | Triggers payment processing after provider instructions are requested |
+| `PAYMENT_STATUS_UPDATE` | `payment-processing` | Triggers status update reconciliation |
+| `VAULT_DEPOSIT` | `vault-reconciliation` | Triggers on-chain deposit reconciliation |
+| `VAULT_WITHDRAWAL` | `vault-reconciliation` | Triggers on-chain withdrawal reconciliation |
+| `VAULT_LOCK` | `vault-reconciliation` | Triggers vault lock reconciliation |
+| `VAULT_UNLOCK` | `vault-reconciliation` | Triggers vault unlock reconciliation |
+| `VAULT_CLOSE` | `vault-reconciliation` | Triggers vault close reconciliation |
+| `NOTIFICATION` | `notifications` | Sends a user notification |
+| `STREAK_UPDATE` | `streak-calculation` | Triggers streak recalculation |
+| `RECONCILIATION` | `stellar-confirmation` | Triggers Stellar transaction reconciliation |
+
+## Retry Behavior
+
+- **Max attempts**: 5
+- **Backoff strategy**: Exponential, starting at 5 seconds and doubling each attempt (capped at 300 seconds)
+- **Retry scheduling**: Failed events have a `nextRetryAt` timestamp; the processor only picks up events whose retry time has arrived
+- **Terminal failure**: After 5 failed attempts, the event is moved to `DEAD_LETTER` status with a `deadLetterReason`
+
+## Operational Recovery
+
+### Processor Startup
+
+The outbox processor starts automatically when the server boots (in both API and worker modes). It runs as a continuous loop that:
+
+1. Queries for pending events with `nextRetryAt <= now` and `attemptCount < 5`
+2. Publishes each event to the appropriate BullMQ queue
+3. Marks the event as `PUBLISHED` on success
+4. Marks the event as `FAILED` with a retry timestamp on transient failure
+5. Marks the event as `DEAD_LETTER` when `attemptCount >= 5`
+
+### Processor Shutdown
+
+On `SIGTERM` or `SIGINT`, the server gracefully shuts down the outbox processor before disconnecting Prisma and Redis. This prevents in-flight publishes from being lost.
+
+### Manual Recovery
+
+To recover stuck events:
+
+```bash
+# Reset failed events that are past their retry time back to PENDING
+# (run via Prisma Studio or a one-off script)
+npx prisma db execute --file prisma/sql/reset_outbox.sql
+```
+
+To inspect dead-letter events:
+
+```sql
+SELECT id, eventType, aggregateId, attemptCount, deadLetterReason, createdAt
+FROM outbox_events
+WHERE status = 'DEAD_LETTER'
+ORDER BY createdAt DESC;
+```
+
+## Monitoring Expectations
+
+- **Outbox lag**: Monitor the count of `PENDING` events with `nextRetryAt <= now`. A growing lag indicates the processor is falling behind.
+- **Dead-letter queue**: Monitor the count of `DEAD_LETTER` events. Any non-zero count requires manual investigation.
+- **Publish failure rate**: Track the ratio of `FAILED` events to total published events. A spike indicates a downstream queue or connectivity issue.
+- **Event age**: Alert on `PENDING` events older than 5 minutes (indicating the processor may be stuck).
+
+## Testing
+
+Unit tests for the outbox repository are in `tests/unit/outbox.repository.test.ts`. Integration tests verifying transactional atomicity and retry behavior are in `tests/integration/outbox.integration.test.ts`.
 
 ---