Container based hadoop distributed system log analyzer
Mammoth is a container based hadoop distributed system log analyzer.
It monitors the Hadoop system as a whole and plays a role in easily delivering meaningful data to users.
Manage your Hadoop system with ease!
Do not worry and use it in any OS!
Easy installation!
Just copy & Paste!
-
Hadoop Overview
- Status monitoring of all nodes
- Disk Usage/Quota monitoring with visualiztion
- vCores/Memory monitoring
- Status monitoring of all applications
-
Resources(Yarn monitoring)
- vCores monitoring with graph visualization
- Memory monitoring with graph visualization
-
HDFS(Hadoop File System monitoring)
- Disk quota monitoring
- Disk usage monitoring
- Disk status monitoring with graph visualization
- File browsing service
-
Applications(MapReduce applications monitoring)
- Status monitoring of all applications
- Detailed application information monitoring
- Mammoth monitor
Python is installed on most systems these days.
The monitor gets the Hadoop log directly from where the Hadoop system is installed and sends data to the collector server.
Since most systems these days have Python installed, we could implement a monitor program in Python easily and conveniently.
Here is README.md of mammoth monitor!
- Mammoth collector
Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic).
Collector server is responsible for processing data received from numerous monitors appropriately and passing the data to influxdb.
Since it is a server that has to handle a lot of traffic, we configured a collector server by balancing the load in our service.
The framework used for Collector Server is a framework called FastAPI.
This framework has been proven to be as fast as Node.js or Go , and we used it because it is a Python-based framework.
Here is README.md of mammoth collector!
- Mammoth GUI
Flutter is Google's UI toolkit for building beautiful, natively compiled applications for mobile, web, desktop, and embedded devices from a single codebase.
There is only one reason I used flutter in the GUI.
Because it is a cross platform framework.
All apps can be compiled from one source to Windows , Mac , Linux , iPadOS , and Android tablet.
We do not plan to support the web environment and mobile phone environment.
Just because I want users to use it on the big screen.
Here is README.md of mammoth GUI!
- Download the Mammoth and register as a member.
- Windows, OSX, Linux, or iPad, you can use this app anywhere you can think of.
- Install mammoth monitor in your hadoop system.
- Run Mammoth monitor
-
Download the Mammoth and register as a member.
- Windows, OSX, Linux, or iPad, you can use this app anywhere you can think of.
-
Install mammoth monitor in your hadoop system.
-
Install mammoth collector in your own server.
-
Install influxdb and mongodb in your own server.
-
Change
conf.py
in your collector -
Change
conf.py
in your monitor
-
Collector(Load balanced)
- mammoth-collector1 :
- CPU : 4 cores
- Memory : 8GB
- mammoth-collector2 :
- CPU : 4 cores
- Memory : 8GB
- mammoth-collector3 :
- CPU : 4 cores
- Memory : 8GB
- mammoth-collector4 :
- CPU : 4 cores
- Memory : 8GB
- mammoth-collector5 :
- CPU : 4 cores
- Memory : 8GB
- mammoth-collector1 :
-
InfluxDB
- CPU : 8 cores
- Memory : 64GB
-
InfluxDB
- CPU : 4 cores
- Memory : 32GB