Smart-Route

Smart-Route is a self-healing, production-grade Canary Deployment infrastructure orchestrated inside a Monorepo. It features a dynamic traffic-splitting Nginx load balancer and an intelligent, automated Python Watchdog-Agent capable of monitoring system stability and triggering seamless automated rollbacks (Failover) without human intervention.

---

Architecture Design & Components

The system architecture is designed for High Availability (HA) and consists of 4 tightly integrated components:

1. api-v1 (The Stable Core)

• Role: Handles 90% of the general baseline production traffic.
• Endpoints:
  • GET /health — Used for active Docker service healthchecking.
  • GET /api/version — Returns stabilization telemetry ("stability": "stable").
  • GET /api/items — Serves baseline mockup telemetry data.

2. api-v2 (The Canary Deployment)

• Role: Holds the new features under evaluation, isolating risk to 10% of total user traffic.
• Endpoints: Parallel to v1 but exposes debugging tools:
  • V2_FAIL_RATE (Env Variable) — Dynamically controls simulated fault probability (e.g., 0.5 triggers HTTP 500 errors on 50% of the traffic).
  • GET /debug/trigger-error — Explicit diagnostic endpoint to manually test the failover mechanics.

3. nginx (The Heart of Routing)

• Strategy: Implements a stateful least_conn (Least Connections) distribution mechanism which outclasses simple Round-Robin by routing incoming client transactions to nodes with less active session memory.
• Upstream Topology:

  upstream smart_route_backend {
      least_conn;
      server api-v1:8000 weight=9 max_fails=3 fail_timeout=10s;
      server api-v2:8000 weight=1 max_fails=3 fail_timeout=10s;
  }

• Configuration Decoupling: To avoid immutable file issues, the routing matrices are not hardcoded inside the proxy image. They reside inside a shared Docker Volume (nginx-config) allowing programmatic mutation at runtime.

4. watchdog-agent (The Self-Healing Engine)

An automated Python orchestration runtime daemon executing three continuous jobs:

• Real-Time Log Ingestion: Spawns a streaming pointer (container.logs(stream=True, follow=True, tail=50)) attached to the api-v2 container runtime daemon, actively searching for HTTP 500 response markers.
• Algorithmic Error Counting (Sliding Window): Implements a time-decay queue window. If 5 or more HTTP 500 exceptions are intercepted within a rolling 60-second horizon, a failover event sequence triggers immediately. Stale error histories outside the timeline decay gracefully.
• Dynamic Mitigation & Discord Alerting: On deployment failure, the agent writes a single-upstream router schema (v1-only) to the shared volume, issues a safe configuration syntax-validation check (nginx -t), reloads the core server state (nginx -s reload), and logs a rich embed report directly into Discord.

---

Ports Mapping & Network Exposure

To simulate a real-world enterprise gateway, individual backend services are hidden inside the isolated Docker internal network and are not exposed directly to the host machine. Only the Nginx Gateway is exposed to handle all public requests:

Service Name	Internal Port	External (Host) Port	Accessibility
nginx	80	8080	Public Gateway (All traffic hits http://localhost:8080)
api-v1	8000	None	Internal Only (Accessible only via Nginx Upstream)
api-v2	8000	None	Internal Only (Accessible only via Nginx Upstream)
watchdog-agent	None	None	Internal Daemon (No listening ports)

---

Monorepo Repository Topology

smart-route/
├── apps/
│   ├── fastapi-v1/         # Production stable API application (Internal Port: 8000)
│   └── fastapi-v2/         # Testing Canary unstable API application (Internal Port: 8000)
├── infrastructure/
│   └── nginx/              # Custom proxy server infrastructure
├── tools/
│   └── watchdog-agent/     # Intelligent automated intervention python script
└── docker-compose.yml      # Master declaration binding orchestration hooks together

---

Shared Storage Layer (Volume Bridge)

Instead of relying on insecure or brittle cross-container orchestration, communication between the watchdog-agent and nginx occurs through an isolated shared storage path:

# nginx container volume mapping
- nginx-config:/etc/nginx/conf.d

# watchdog container volume mapping
- nginx-config:/nginx-config

Container Runtime	Mount Reference Path	Operational Intent
nginx	/etc/nginx/conf.d/default.conf	Read-Only Core Routing Execution
watchdog	/nginx-config/default.conf	Write-Access Dynamic Failover Adjustment

---

System Initialization Lifecycle

To ensure race conditions do not crash the infrastructure during a fresh install, Docker Compose controls order of dependencies utilizing micro-healthchecks:

1. [api-v1]  ──┐
               ├── [ Healthchecks Pass successfully ]
2. [api-v2]  ──┘
               ↓
3. [nginx]   ── [ Launches only when Upstream backends are completely stable ]
               ↓
4. [watchdog]── [ Begins monitoring when Nginx and Canary are live ]

---

How to Run & Execute (Step-by-Step)

Follow these exact steps to spin up the entire ecosystem and verify its state:

1. Configure the Environment

Open the docker-compose.yml file and supply your target Discord Webhook URL inside the watchdog-agent service environment section:

environment:
  - DISCORD_WEBHOOK_URL=https://discord.com

2. Build and Launch the Infrastructure

Execute the following standard deployment command from the repository root directory:

docker-compose up --build -d

The --build flag guarantees all localized changes inside the sub-apps are re-compiled, and -d detaches the process to run smoothly in the background.

3. Verify Container Infrastructure States

Check if all services passed their healthchecks and are operational:

docker ps

You should see 4 active healthy containers running.

---

Live Failover Demonstration Walkthrough

Step 1: Validate Weighted Routing Performance

Send multiple parallel curl connections. You will observe traffic distributed in a stable 9:1 partition structure.

curl http://localhost:8080/api/version

Step 2: Inject Chaos (Simulating 5xx Outage)

Hit the simulated failure target path on the canary server 5 consecutive times within one minute to hit the threshold window:

curl http://localhost:8080/debug/trigger-error

Step 3: Trace the Intervention Matrix

1. The Watchdog-Agent captures the five consecutive telemetry failures from the log stream.
2. The agent re-generates default.conf within /nginx-config/.
3. The agent sends a hot-reload sequence inside the proxy container (nginx -s reload).
4. A warning dashboard webhook is blasted to the target Discord engineering channel.
5. All traffic instantly scales 100% back to api-v1, implementing an instantaneous network-level recovery pattern without generating explicit service disruptions.

---

Why a Monorepo?

• Single Unified Codebase: Maintains all isolated system runtimes (v1, v2, nginx, watchdog) in a clean consolidated tree.
• Atomic Deployments: Changes to an api schema and its corresponding watchdog log parser pattern can occur inside a single Pull Request.
• Streamlined CI/CD: Provides a centralized workspace layout, reducing deployment pipelines to a single script command execution layer.