LogStash

A scalable, real-time log ingestion and monitoring platform built from the ground up. This project is not just a simple web app, but an end-to-end data pipeline designed to handle structured logs efficiently, inspired by professional tools like DataDog and Splunk.

Data Flow

Data Flow (Ingestion & Real-time):

[Client] --POST--> [Log Server] --(index)--> [Elasticsearch] | '--(broadcast via Socket.IO)--> [Live Dashboard Clients]

Search Flow:

[Client] --GET--> [Log Server] --(search)--> [Elasticsearch]

🛠️ Tech Stack & Tools

Backend: Node.js, Express.js

Database: Elasticsearch (for high-speed, indexed text search)

Real-time Communication: WebSockets (via Socket.IO)

Orchestration: Docker, Docker Compose

API Testing: Postman

Features

• Log Ingestion: A robust endpoint to receive structured logs (POST /log).
• Indexed Storage: Logs are stored in Elasticsearch, making them instantly searchable.
• Powerful Search: A dedicated search endpoint (GET /search) to query logs by matching keywords in the message.
• Live Log Tailing: A real-time, WebSocket-based feed that instantly broadcasts new logs to all connected clients, simulating a live tail -f command.
• Fully Containerized: The entire application (Node.js server + Elasticsearch) is managed by Docker Compose, making the setup and deployment incredibly simple and reliable.

HOW TO RUN

Getting the system up and running is as easy as one command.

• Prerequisites: Docker Desktop installed and running.

• Steps:

1. Clone this repository to your local machine.
2. Navigate to the root directory of the project.
3. Run the magic command:

docker-compose up --build

• The factory is now live! The server is running on http://localhost:4000.

API ENDPOINTS

Ingest a Log Endpoint: /log Method: POST Body (raw/json): code JSON { "level": "error", "message": "User login failed due to invalid password.", "service": "auth-service" }

Search Logs Endpoint: /search Method: GET Query Parameter: q Example: http://localhost:4000/search?q=failed

Live Log Feed (WebSocket) Endpoint: http://localhost:4000 Protocol: Use a Socket.IO client to connect. Event to Listen for: 'new-log'