[Feature Request]: Add Structured Logging with JSON Output and a Log Aggregation View in the Dashboard

[divyanshim27]

Summary

Arachnode is a multi-service system running 6 distinct microservices simultaneously (crawler, scraper, aggregator, contact-discovery, email-generator, gateway, scheduler). Debugging failures across these services currently requires running docker compose logs <service> and manually scanning unstructured text output. The scheduler/logger.py file exists and describes JSON log formatting, but the logs are not surfaced in the dashboard or aggregated in a queryable format. Adding structured JSON logging across all services and a log viewer in the dashboard would make the system significantly more debuggable.

Problem

• When a crawl fails or a contact discovery request times out, there is no centralized place to see what went wrong across which service.
• Log output from docker compose logs is interleaved from all services with no correlation IDs, making it impossible to trace a single job through the full pipeline.
• The dashboard has no log viewer — users must context-switch to the terminal to diagnose issues.
• There is no structured log format enforced across services — some services use print(), others use Python's logging module with varying formats.

Impact

• Contributors and users debugging failed workflows spend disproportionate time in terminal log output.
• Without correlation IDs, it is impossible to trace which log lines in the aggregator correspond to which crawl job from the crawler.
• The system's "run in background" nature (APScheduler cron jobs) makes silent failures invisible without structured logging.

Proposed Solution

I would like to standardize logging across all services using Python's structlog library and surface a log tail view in the dashboard:

Shared logging config (shared/logger.py):

import structlog
import logging

def configure_logging(service_name: str):
    structlog.configure(
        processors=[
            structlog.stdlib.add_log_level,
            structlog.stdlib.add_logger_name,
            structlog.processors.TimeStamper(fmt="iso"),
            structlog.processors.JSONRenderer()
        ],
        wrapper_class=structlog.stdlib.BoundLogger,
        logger_factory=structlog.stdlib.LoggerFactory(),
    )
    return structlog.get_logger(service=service_name)

Usage in each service:

logger = configure_logging("aggregator")
logger.info("job_inserted", job_id=job_id, company=company, source=source)
logger.error("db_write_failed", job_id=job_id, error=str(e))

Correlation ID middleware (FastAPI):

from uuid import uuid4
from starlette.middleware.base import BaseHTTPMiddleware

class CorrelationIDMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        correlation_id = request.headers.get("X-Correlation-ID", str(uuid4()))
        with structlog.contextvars.bind_contextvars(correlation_id=correlation_id):
            response = await call_next(request)
        response.headers["X-Correlation-ID"] = correlation_id
        return response

Dashboard log tail (via Gateway SSE endpoint):

# gateway/main.py
@app.get("/api/logs/stream")
async def stream_logs():
    # Tail docker logs from all services via subprocess
    async def generate():
        proc = await asyncio.create_subprocess_exec(
            "docker", "compose", "logs", "--follow", "--tail=50",
            stdout=asyncio.subprocess.PIPE
        )
        while True:
            line = await proc.stdout.readline()
            if line:
                yield f"data: {line.decode()}\n\n"
    return StreamingResponse(generate(), media_type="text/event-stream")

Deliverables

• shared/logger.py — standardized structlog configuration.
• All 6 services updated to use structured logging (replacing print() and inconsistent logging calls).
• CorrelationIDMiddleware added to all FastAPI services.
• GET /api/logs/stream SSE endpoint in the gateway.
• Log tail panel in dashboard.html with service filter and severity filter (INFO/WARNING/ERROR).
• structlog added to all requirements.txt files.

Labels: enhancement, observability, developer-experience, GSSoC 2026

Could you assign this issue to me?

[divyanshim27]

Hey @sharmavaibhav31,
Please assign to me under GSSoC 2026.
/claim /assign /claim-assignment

[anshul23102]

Hi @sharmavaibhav31, important observability feature! Adding structured logging with JSON output and log aggregation view would improve debugging and monitoring capabilities.

Could this be assigned to me? I'm ready to implement structured logging infrastructure.

Thanks for the opportunity! 📊