Skip to content

Commit

Permalink
Feature/Boost Up: Run multiple Python scripts/wrappers per multiple h…
Browse files Browse the repository at this point in the history 
…osts simultaneously (#91)

* Multiple python scripts per host

* Support different script definition cases

* Add different script definitions, update style

* Update documentation and examples

* Update template style

* Format with Black

* Add notes to documentation

* Add installation legend
  • Loading branch information
@manmolecular
manmolecular authored Apr 1, 2020
1 parent 13e1171 commit f431819
Show file tree
Hide file tree
Showing 3 changed files with 191 additions and 76 deletions.
131 changes: 86 additions & 45 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
- [Screenshots](#screenshots)
- [Description](#description)
1. [Requirements](#requirements)
- [Legend](#legend)
- :pushpin: [Basic](#basic)
- :pushpin: [Accounts](#accounts)
- [Additional scanning](#additional-scanning)
Expand Down Expand Up @@ -82,21 +83,24 @@ To visualize gained data, the Grinder Framework provides an interactive world ma
For example, the hosts will be automatically checked for availability with ping from back-end, also for every host many additional features are available: current host can be directly opened in Shodan, Censys, and ZoomEye web interfaces, the host can be shown on Google Maps with all available information about geolocation. Also, it is possible to make an IP lookup or open raw information in JSON directly in a browser or from your application with provided API methods.

## Requirements
### Legend
:heavy_exclamation_mark: required
:heavy_plus_sign: not required to run (or required only for additional modules)
### Basic
- [Python 3.6+](https://www.python.org/downloads/)
- [python3-tk](https://docs.python.org/3/library/tkinter.html) library
- :heavy_exclamation_mark: [Python 3.6+](https://www.python.org/downloads/)
- :heavy_exclamation_mark: [python3-tk](https://docs.python.org/3/library/tkinter.html) library
### Accounts
- [Shodan](https://account.shodan.io/register) and [Censys](https://censys.io/register) accounts
- :heavy_exclamation_mark: [Shodan](https://account.shodan.io/register) and [Censys](https://censys.io/register) accounts
Required to collect hosts, both free and full accounts are suitable. Also, it's possible to use only one account (Censys or Shodan, Shodan is preferable).
- [Vulners](https://vulners.com/) account
- :heavy_plus_sign: [Vulners](https://vulners.com/) account
Required to make additional reports on vulnerabilities and exploits. If this feature is not required for you, you can use Grinder without Vulners account.
### Additional scanning
- [Nmap Security Scanner 7.60+](https://nmap.org/download.html)
- :heavy_plus_sign: [Nmap Security Scanner 7.60+](https://nmap.org/download.html)
Version 7.60 and newer has been tested with currently used in Grinder scripts (ssl-cert.nse, vulners.nse, etc.).
### TLS configuration
- [TLS-Attacker 3.0](https://github.com/RUB-NDS/TLS-Attacker/tree/3.0)
- :heavy_plus_sign: [TLS-Attacker 3.0](https://github.com/RUB-NDS/TLS-Attacker/tree/3.0)
Required only for TLS scanning.
- [TLS-Scanner 2.9](https://github.com/RUB-NDS/TLS-Scanner/tree/2.9)
- :heavy_plus_sign: [TLS-Scanner 2.9](https://github.com/RUB-NDS/TLS-Scanner/tree/2.9)
Required only for TLS scanning.

## Current Features
Expand All @@ -115,6 +119,10 @@ Required only for TLS scanning.
- :bulb: Searching for documents, security bulletins, public exploits and many more things based on detected by Grinder vulnerabilities and software

### Additional Modules
:rocket: **Note #1:** You can run multiple Python scripts simultaneously per multiple hosts, so you can build your own chain of scripts and checks to get the most information from your hosts. Feel free to add your modules with PR or give us an idea with feature issue.

:construction: **Note #2:** Multiple NSE scripts running task is still in WIP status. So, NSE scripts will be ran consistently, one after one. New NSE script engine for Grinder is comming up, stay tuned.

#### DICOM Patient Info Getter
**Location:** `py_scripts/dicom_getter/dicom_getter.py`
**Description:** This module allows you to grab different patient information (including files) from medical servers
Expand Down Expand Up @@ -149,8 +157,11 @@ Required only for TLS scanning.
:construction: **Note:** The Grinder framework is still in progress and got features to improve, so all the tasks and other features will always be described in this project. If you got some awesome ideas or any other interesting things for Grinder, you can always open a pull request or some issues in this repository.

## Grinder Installing
### Setup and Configure Environment
_Note: If you are familiar with pipenv package manager, all steps related to virtualenv can be replaced by `pipenv sync` command._
:bulb: **Note #1:** If you are familiar with pipenv package manager, all steps related to virtualenv can be replaced with `pipenv sync` command.

:bulb: **Note #2:** If you are familiar with Docker and docker-compose, you can build framework with Docker, see [Building and Running in Docker](#building-and-running-in-docker)

### Setup and Configure Environment
1. Install [Nmap Security Scanner](https://nmap.org/download.html) if not installed.
2. Install [python3-tk](https://docs.python.org/3/library/tkinter.html) package if not installed (Linux only)
```bash
Expand Down Expand Up @@ -470,72 +481,102 @@ To add your own vendors and products with queries you can simply create a new JS
```json
[
{
"vendor": "YOUR OWN VENDOR HERE",
"product": "YOUR OWN PRODUCT HERE",
"shodan_queries": [
"vendor":"Your vendor name here ('Apache', for example; any string is allowed and required)",
"product":"Your product name here ('HTTP Server', for example; any string is allowed and required)",
"shodan_queries":[
{
"query": "YOUR SHODAN QUERY HERE",
"query_confidence": "QUERY CONFIDENCE LEVEL {tentative | firm | certain}"
"query":"Shodan query here ('Apache', 'Server: Apache', for example; any string is allowed and required)",
"query_confidence":"Query confidence level ('tentative', 'firm' or 'certain' - you can sort and modify your search with these keywords, use query confidence flag for this purpose)"
}
],
"censys_queries": [
"censys_queries":[
{
"query": "YOUR CENSYS QUERY HERE",
"query_confidence": "QUERY CONFIDENCE LEVEL {tentative | firm | certain}"
"query":"Censys query here ('Apache', 'Server: Apache', for example; any string is allowed and required)",
"query_confidence":"Query confidence level ('tentative', 'firm' or 'certain' - you can sort and modify your search with these keywords, use query confidence flag for this purpose)"
}
],
"scripts": {
"py_script": "NAME OF MODULE AND PYTHON SCRIPT FROM /custom_scripts/py_scripts",
"nse_script": "NAME OF MODULE AND NSE SCRIPT FROM /custom_scripts/nse_scripts"
"scripts":{
"py_script":"3 types of definitions are allowed: you can use a simple string here, for example, 'test/test.py'; you can use a list here, for example, ['test1/test1.py', 'test2/test2.py']; you can use a dictionary here, for example, {'test': 'test/test.py'}",
"nse_script":"Currently, you can run only 1 NSE script per time, this is WIP. Define the script with string, for example, 'test/test.nse'"
},
"vendor_confidence": "VENDOR CONFIDENCE LEVEL {tentative | firm | certain}"
"vendor_confidence":"Vendor confidence level ('tentative', 'firm' or 'certain' - you can sort and modify your search with these keywords, use vendor and product confidence flags for this purpose)"
}
]
```
### Queries Example
:construction: **Note:** Queries in the `queries/` directory may be different due to the different stages of development, but mostly all of them are still supported and tested. The most actual query template and an example provided below, so if you need or want to try your queries, you can use this example to get the freshest features.
```json
[
{
"vendor": "Apache Software Foundation",
"product": "Apache HTTP Server",
"shodan_queries": [
"vendor":"Apache Software Foundation",
"product":"Apache HTTP Server",
"shodan_queries":[
{
"query":"Apache",
"query_confidence":"certain"
}
],
"censys_queries":[
{
"query":"Apache",
"query_confidence":"certain"
}
],
"scripts":{
"py_script":"http_status/http_status.py",
"nse_script":""
},
"vendor_confidence":"certain"
},
{
"vendor":"Nginx",
"product":"Nginx",
"shodan_queries":[
{
"query": "Apache",
"query_confidence": "certain"
"query":"Nginx",
"query_confidence":"certain"
}
],
"censys_queries": [
"censys_queries":[
{
"query": "Apache",
"query_confidence": "certain"
"query":"Nginx",
"query_confidence":"certain"
}
],
"scripts": {
"py_script": "http_response_grabber/http_response_grabber.py",
"nse_script": "test/test.nse"
"scripts":{
"py_script":[
"http_response_grabber/http_response_grabber.py",
"http_status/http_status.py",
"test/test.py"
],
"nse_script":""
},
"vendor_confidence": "certain"
"vendor_confidence":"certain"
},
{
"vendor": "Nginx",
"product": "Nginx",
"shodan_queries": [
"vendor":"Flask",
"product":"Flask",
"shodan_queries":[
{
"query": "Nginx",
"query_confidence": "certain"
"query":"Flask",
"query_confidence":"certain"
}
],
"censys_queries": [
"censys_queries":[
{
"query": "Nginx",
"query_confidence": "certain"
"query":"Flask",
"query_confidence":"certain"
}
],
"scripts": {
"py_script": "http_response_grabber/http_response_grabber.py",
"nse_script": "test/test.nse"
"scripts":{
"py_script":{
"grabber":"http_response_grabber/http_response_grabber.py",
"status":"http_status/http_status.py",
"test":"test/test.py"
},
"nse_script":""
},
"vendor_confidence": "certain"
"vendor_confidence":"certain"
}
]
```
59 changes: 52 additions & 7 deletions grinder/pyscriptexecutor.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,15 @@ def _exec_script(self, host_info, py_script) -> any:
"""
if isinstance(py_script, str) and py_script.endswith(".py"):
full_path = self.base_path.joinpath(py_script)
py_script_wo_extension = py_script.replace(".py", "")
loader = SourceFileLoader("main", str(full_path))
module = ModuleType(loader.name)
loader.exec_module(module)
script_result = {py_script_wo_extension: module.main(host_info)}
if not self.mute:
return module.main(host_info)
return script_result
with redirect_stderr(None), redirect_stdout(None):
return module.main(host_info)
return script_result

@exception_handler(expected_exception=PyScriptExecutorRunProcessError)
def run(self) -> None:
Expand All @@ -100,13 +102,22 @@ def run(self) -> None:
# Wait while queue will get some tasks to do
sleep(PyProcessingValues.EMPTY_QUEUE_POLLING_RATE)
continue
log_progress, log_host, py_script = (
"Error",
"Unknown host",
"Unknown script",
)
try:
current_progress, host_info, py_script = self.queue.get()
ip = host_info.get("ip")
port = host_info.get("port")

# Poll with POLLING_RATE interval
sleep(PyProcessingValues.POLLING_RATE)

# Setup logging
log_progress = f"[{current_progress[0]}/{current_progress[1]}] ({current_progress[2]})"
log_host = f"{host_info.get('ip')}:{host_info.get('port')}"
log_host = f"{ip}:{port}"

try:
result = self._exec_script(host_info=host_info, py_script=py_script)
Expand All @@ -117,15 +128,22 @@ def run(self) -> None:
self.queue.task_done()
continue
try:
PyProcessingResults.RESULTS.update({host_info.get("ip"): result})
if ip not in PyProcessingResults.RESULTS.keys():
PyProcessingResults.RESULTS.update({ip: result})
else:
old_result = PyProcessingResults.RESULTS.get(ip)
old_result.update(result)
PyProcessingResults.RESULTS[ip] = old_result
except (AttributeError, ConnectionRefusedError):
print(
f"{log_progress} -> Caught manager error on host {log_host}: simultaneous shared-dict call"
)
self.queue.task_done()
continue
except:
print(f'{log_progress} -> script "{py_script}" crash for {log_host}')
except Exception as script_err:
print(
f'{log_progress} -> script "{py_script}" crash for {log_host}: {str(script_err)}'
)
self.queue.task_done()
else:
print(f'{log_progress} -> script "{py_script}" done for {log_host}')
Expand Down Expand Up @@ -175,7 +193,34 @@ def organize_processes(self) -> None:
if not py_script:
continue
percentage = round((index / hosts_length) * 100, 2)
queue.put(((index, hosts_length, f"{percentage}%"), host_info, py_script))
# In case of:
# "py_script": "package/script.py"
if isinstance(py_script, str):
queue.put(
((index, hosts_length, f"{percentage}%"), host_info, py_script)
)
# In case of:
# "py_script": ["package1/script1.py", "package2/script2.py", ...]
elif isinstance(py_script, list):
for script in py_script:
if not script:
continue
queue.put(
((index, hosts_length, f"{percentage}%"), host_info, script)
)
# In case of:
# "py_script": {"script1": "package1/script1.py", "script2": "package2/script2.py"}
elif isinstance(py_script, dict):
for script_name, script_file in py_script.items():
if not script_file:
continue
queue.put(
(
(index, hosts_length, f"{percentage}%"),
host_info,
script_file,
)
)
queue.join()

def start(self) -> None:
Expand Down
Loading

0 comments on commit f431819

Please sign in to comment.