Self-Healing Infrastructure Bot

A production-grade Linux daemon that autonomously monitors system health, detects anomalies, and executes targeted remediations - logging every action to a structured audit trail and exposing real-time metrics to Prometheus and Grafana.

Built as a portfolio project to demonstrate Linux systems engineering, Python automation, observability stack deployment, and systemd service management.

---

Architecture

Linux System (psutil)
  CPU · Memory · Disk · Services
         │
         ▼
   monitor.py  ──── polls every 60s
         │
    ┌────┴────┐
    ▼         ▼
remediator  exporter.py
  .py       (HTTP :8000)
    │              │
    ▼              ▼
audit_log    Prometheus
(SQLite)     (scrapes :8000)
                   │
                   ▼
                Grafana
             (dashboards)

The bot runs as a systemd service — starts on boot, restarts automatically on crash (Restart=on-failure), and logs to journalctl.

---

What it monitors and remediates

Anomaly	Threshold	Remediation
CPU spike	> 80% warning / > 90% critical	Identifies top CPU processes, logs PIDs, fires alert
High memory	> 80% warning / > 90% critical	Drops Linux page cache via /proc/sys/vm/drop_caches
Disk full	> 75% warning / > 85% critical	Purges /tmp files older than 7 days
Service down	Any watched service stops	Attempts systemctl restart, escalates after 3 failures

All remediations are safe by design — no process killing, no destructive filesystem operations outside /tmp.

---

Tech stack

• Python 3.12 — core daemon logic
• psutil — Linux kernel metric collection (CPU, memory, disk, processes)
• prometheus-client — custom metrics exporter on :8000
• Prometheus — time-series metric storage (Docker)
• Grafana — live dashboards with threshold alerting (Docker)
• Node Exporter — OS-level metrics (Docker)
• SQLite — structured audit log (no server required)
• systemd — process supervision, auto-restart, boot persistence
• Docker Compose — observability stack orchestration
• pytest — 13 unit tests with mocking (0 real system calls in tests)

---

Project structure

self-healing-bot/
├── bot/
│   ├── __init__.py
│   ├── config.py          # all thresholds and settings
│   ├── monitor.py         # main poll loop and orchestration
│   ├── remediator.py      # one function per anomaly type
│   ├── exporter.py        # Prometheus metrics HTTP server
│   └── audit_log.py       # SQLite structured event logger
├── infra/
│   ├── docker-compose.yml # Prometheus + Grafana + Node Exporter
│   ├── prometheus.yml     # scrape targets configuration
│   └── grafana/
│       ├── datasource.yml # auto-provisions Prometheus data source
│       └── dashboard.json # pre-built 10-panel monitoring dashboard
├── systemd/
│   └── self-healing-bot.service  # systemd unit file
├── tests/
│   └── test_remediator.py # 13 pytest unit tests
├── requirements.txt
└── README.md

---

Running locally

Prerequisites: Python 3.10+, Docker, Docker Compose

# Clone and set up
git clone https://github.com/Akindu27/self-healing-bot.git
cd self-healing-bot
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Start the observability stack
cd infra && docker compose up -d && cd ..

# Run the bot
python3 -m bot.monitor

Open http://localhost:3000 (Grafana, admin/admin) and import infra/grafana/dashboard.json.

---

Installing as a systemd service

sudo cp systemd/self-healing-bot.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable self-healing-bot   # start on boot
sudo systemctl start self-healing-bot    # start now
sudo systemctl status self-healing-bot   # verify running

View live logs:

journalctl -u self-healing-bot -f

Test self-healing (bot restarts automatically after crash):

sudo kill -9 $(pgrep -f "bot.monitor")
sleep 6
sudo systemctl status self-healing-bot  # new PID, running again

---

Prometheus metrics exposed

Metric	Type	Description
bot_cpu_usage_percent	Gauge	Current CPU usage %
bot_memory_usage_percent	Gauge	Current memory usage %
bot_disk_usage_percent{mount}	Gauge	Disk usage % per mount point
bot_remediations_total{type,outcome}	Counter	Total remediations by type and outcome
bot_anomalies_detected_total{type,severity}	Counter	Total anomalies detected
bot_last_poll_timestamp_seconds	Gauge	Unix timestamp of last poll (heartbeat)
bot_info	Info	Bot version and author metadata

---

Audit log

Every anomaly and remediation is written to audit.db (SQLite):

sqlite3 audit.db "SELECT timestamp, anomaly_type, severity, action_taken, outcome FROM events ORDER BY timestamp DESC LIMIT 10;"

Example output:

2026-05-26T08:02:52+00:00 | disk | critical | Purged 12 files from /tmp (234MB freed) | resolved
2026-05-26T07:15:01+00:00 | service | critical | Restarted cron successfully | resolved
2026-05-26T06:44:23+00:00 | memory | warning | Dropped page cache, freed 180MB | resolved

---

Running tests

python3 -m pytest tests/ -v

13 passed in 0.36s

Tests use unittest.mock to patch all system calls — no real filesystem, process, or systemd interaction during testing.

---

Grafana dashboard

10-panel dashboard covering:

• CPU, Memory, Disk gauges with colour-coded thresholds
• Time-series graphs for CPU and memory trends
• Bot heartbeat panel (alerts if polling stops)
• Remediation counter bar chart by anomaly type and outcome
• Anomaly event rate over time
• System-level CPU and memory from Node Exporter

Import from infra/grafana/dashboard.json via Grafana → Dashboards → Import.

---

What I learned

• How Linux exposes system metrics through /proc and how psutil reads them
• The difference between Prometheus Gauges and Counters, and why pull-based scraping is more reliable than push
• How systemd manages process lifecycle — Restart=on-failure, SIGTERM handling, journalctl logging
• Why sustained-check patterns prevent false positives in production monitoring
• How to use unittest.mock.patch to write fast, deterministic tests that don't touch real system resources
• SQLite as a lightweight audit store — structured, queryable, no server required

---

Built by Akindu Gunarathna — github.com/Akindu27/self-healing-bot