Skip to content

Commit 7a1e8e3

Browse files
author
Matteo Lodi
committed
added docs
1 parent 81b3070 commit 7a1e8e3

File tree

4 files changed

+282
-0
lines changed

4 files changed

+282
-0
lines changed

‎docs/Contribute.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Contribute
2+
3+
Intel Owl was designed to ease the addition of new analyzers. With a simple python script you can integrate your own engine or integrate an external service in a short time.
4+
5+
6+
## How to add a new analyzer
7+
You may want to look at the following analyzer example to start to build a new one: [Example](https://github.com/certego/IntelOwl/blob/master/api_app/script_analyzers/analyzer_template.py)
8+
9+
After having written the new python module, you have to remember to:
10+
* Put the module in the `file_analyzers` or `observable_analyzers` directory based on what it can analyze
11+
* Add the new module as a celery task in [tasks.py](https://github.com/certego/IntelOwl/blob/master/intel_owl/tasks.py)
12+
* Add a new entry in the [analyzer configuration](https://github.com/certego/IntelOwl/blob/master/configuration/analyzer_config.json):
13+
14+
Example
15+
```
16+
"Analyzer_Name": {
17+
"type": "file",
18+
"external_service": true,
19+
"leaks_info": true,
20+
"run_hash": true,
21+
"supported_filetypes": ["application/javascript"],
22+
"python_module": "haget_run",
23+
"additional_config_params": {
24+
"custom_required_param": "ANALYZER_SPECIAL_KEY"
25+
}
26+
},
27+
```
28+
29+
Remember to set at least:
30+
* `type`: can be `file` or `observable`. It specifies what the analyzer should analyze
31+
* `python_module`: name of the task that the analyzer must launch
32+
33+
The `additional_config_params` can be used in case the new analyzer requires additional configuration.
34+
In that way you can create more than one analyzer for a specific python module, each one based on different configurations.
35+
MISP and Yara Analyzers are a good example of this use case: for instance, you can use different analyzers for different MISP instances.
36+
37+
* If possible, add a new basic unit test in [tests.py](https://github.com/certego/IntelOwl/blob/master/api_app/tests.py)
38+
39+
Then follow the [Test](https://github.com/certego/IntelOwl/blob/master/docs/Tests.md) guide to start testing.
40+
41+
* Finally, add the new analyzer/s in the list in the doc: [Usage](https://github.com/certego/IntelOwl/blob/master/docs/Usage.md)
42+
43+
If everything is working, you can submit your pull request!

‎docs/Installation.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Installation
2+
3+
## Deployment options:
4+
* Docker-compose for classic server deployment
5+
* (Future) Docker for serverless deployment (ex: AWS Fargate)
6+
7+
We suggest you to clone the project, configure the required environment variables and run `docker-compose up` using the docker-compose file that is embedded in the project.
8+
9+
That file leverages a public docker image that is available in Docker Hub: certego/intelowl
10+
11+
## Deployment Info
12+
Main components of the web application:
13+
* Django
14+
* Rabbit-MQ
15+
* Celery (for async calls and crons)
16+
* Nginx
17+
* Uwsgi
18+
* Flower (optional)
19+
20+
All these components are managed by docker-compose
21+
22+
Database: PostgreSQL
23+
24+
## Deployment preparation
25+
### Environment configuration
26+
Before running the project, you must populate some environment variables in a file to provide the required configuration.
27+
In the project you can find a template file named `env_file_app_template`.
28+
You have to create a new file named `env_file_app` from that template and modify it with your own configuration.
29+
30+
Required variables to run the image:
31+
* DJANGO_SECRET: random 50 chars key, must be unique, generate it randomly
32+
* DB_HOST, DB_PORT, DB_USER, DB_PASSWORD: PostgreSQL configuration
33+
34+
Optional variables needed to enable specific analyzers:
35+
* ABUSEIPDB_KEY: AbuseIPDB API key
36+
* GSF_KEY: Google Safe Browsing API key
37+
* OTX_KEY: Alienvault OTX API key
38+
* CIRCL_CREDENTIALS: CIRCL PDNS credentials in the format: `user|pass`
39+
* VT_KEY: VirusTotal API key
40+
* HA_KEY: HybridAnalysis API key
41+
* INTEZER_KEY: Intezer API key
42+
* FIRST_MISP_API: FIRST MISP API key
43+
* FIRST_MISP_URL: FIRST MISP URL
44+
* CUCKOO_URL: your cuckoo instance URL
45+
46+
### Database configuration
47+
Before running the project, you must populate the basic configuration for PostgreSQL.
48+
In the project you can find a template file named `env_file_postgres_template`.
49+
You have to create a new file named `env_file_postgres` from that template and modify it with your own configuration.
50+
51+
Required variables (we need to insert some of the values we have put in the previous configuration):
52+
* POSTGRES_PASSWORD (same as DB_PASSWORD)
53+
* POSTGRES_USER (same as DB_USER)
54+
* POSTGRES_DB -> default `intel_owl_db`
55+
56+
If you prefer to use an external PostgreSQL instance, you should just remove the relative image from the `docker-compose.yml` file and provide the configuration to connect to your controlled instance/s.
57+
58+
### Web server configuration
59+
By default Intel Owl provides basic configuration for:
60+
* Nginx (`intel_owl_nginx_http` or `intel_owl_nginx_https`)
61+
* Uwsgi (`intel_owl.ini`)
62+
63+
You can find them in the `configuration` directory.
64+
65+
By default, the project would use the default deployment configuration and HTTP only.
66+
67+
I suggest you to change these configuration files based on your needs and mount them as volumes by changing the `docker-compose.yml` file.
68+
69+
In case you enable HTTPS, remember to set the environment variable `HTTPS_ENABLED` as "enabled" to increment the security of the application.
70+
71+
### Analyzers configuration
72+
In the file `analyzers_config.json` there is the configuration for all the available analyzers you can run.
73+
For a complete list of all current available analyzer please look at: [Usage](https://github.com/certego/IntelOwl/blob/master/docs/Usage.md)
74+
75+
You may want to change this configuration to add new analyzers or to change the configuration of some of them.
76+
77+
The name of the analyzers can be changed at every moment based on your wishes.
78+
You just need to remember that it's important that you keep at least the following keys in the analyzers dictionaries to let them run correctly:
79+
* `type`: can be `file` or `observable`. It specifies what the analyzer should analyze
80+
* `python_module`: name of the task that the analyzer must launch
81+
82+
For a full description of the available keys, check the [Usage](https://github.com/certego/IntelOwl/blob/master/docs/Usage.md) page
83+
84+
### Rebuilding the project
85+
If you make some code changes and you like to rebuild the project, launch the following command from the project directory:
86+
87+
`docker build --tag=<your_tag> .`
88+
89+
Then, you should provide your own image in the `docker-compose.yml` file.
90+
91+
92+
## AWS support
93+
At the moment there's a basic support for some of the AWS services. More is coming in the future.
94+
95+
### Secrets
96+
If you would like to run this project on AWS, I'd suggest you to use the "Secrets Manager" to store your credentials. In this way your secrets would be better protected.
97+
98+
This project supports this kind of configuration. Instead of adding the variables to the environment file, you should just add them with the same name on the AWS Secrets Manager and Intel Owl will fetch them transparently.
99+
100+
Obviously, you should have created and managed the permissions in AWS in advance and accordingly to your infrastructure requirements.
101+
102+
Also, you need to set the environment variable `AWS_SECRETS` to `True` to enable this mode.
103+
104+
You can customize the AWS Region changing the environment variable `AWS_REGION`.
105+
106+
### SQS
107+
If you like, you could use AWS SQS instead of Rabbit-MQ to manage your queues.
108+
In that case, you should change the parameter `CELERY_BROKER_URL` to `sqs://` and give your instances on AWS the proper permissions to access it.
109+
110+
Also, you need to set the environment variable `AWS_SQS` to `True` to activate the additional required settings.
111+
112+
### ... More coming
113+
114+
115+
## Run
116+
After having properly configured the environment files as suggested previously, you can run the image.
117+
The project uses `docker-compose`. You have to move to the project main directory to properly run it.
118+
119+
`docker-compose up`
120+
121+
122+
## After deployment
123+
### Users creation
124+
You may want to run `docker exec -ti intel_owl_uwsgi python3 manage.py createsuperuser` after first run to create a superuser.
125+
Then you can add other users directly from the Django Admin Interface after having logged with the superuser account.

‎docs/Tests.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Tests
2+
3+
If you want to run tests, you should:
4+
5+
- Change the following environment variables in the environment file based on the data you would like to test
6+
* TEST_TOKEN -> API key for testing purposes
7+
* TEST_JOB_ID
8+
* TEST_MD5
9+
* TEST_URL
10+
* TEST_IP
11+
* TEST_DOMAIN
12+
13+
- I didn't add to the project my test files because they are malware samples and I do not want people to use them in a wrong way.
14+
So, if you want to test files, you should add the following files to the folder `test_files`:
15+
* documento.pdf (to test PDF engines)
16+
* documento.rtf (to test RTF engines)
17+
* documento.doc (to test DOC engines)
18+
* file.exe (to test PE engines)
19+
* file.dll (to test DLL engines)
20+
* non_valid_pe.exe (to test non-valid PE)
21+
22+
23+
Run:
24+
`docker exec -ti intel_owl_uwsgi python3 manage.py test`
25+

‎docs/Usage.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Usage
2+
3+
## Client
4+
Intel Owl main objective is to provide a single API interface to query in order to retrieve threat intelligence at scale.
5+
6+
So, in order to interact with the Intel Owl APIs, you may want to use the specific client that is available at: https://github.com/mlodic/pyintelowl
7+
8+
We suggest to download the client, read the instructions and use it as a library for your own python projects or directly from the command line.
9+
10+
### Tokens creation
11+
The server authentication is managed by API keys. So, if you want to interact with Intel Owl, you have to create one or more unprivileged users from the Django Admin Interface and then generate a token for those users.
12+
Afterwards you can leverage the created tokens with the Intel Owl Client.
13+
14+
15+
## Available Analyzers
16+
17+
The following is the list of the available analyzers you can run out-of-the-box:
18+
19+
### File analyzers:
20+
* File_Info: static generic File analysis
21+
* PDF_Info: static PDF analysis
22+
* Rtf_Info: static RTF analysis
23+
* Doc_Info: static generic document analysis
24+
* PE_Info: static PE analysis
25+
* Signature_Info: PE signature extractor
26+
* Strings_Info_Classic: strings extraction
27+
* Strings_Info_ML: strings extraction plus strings ranking based on Machine Learning
28+
* VirusTotal_v3_Get_File: check file hash on VirusTotal
29+
* VirusTotal_v3_Get_File_And_Scan: check file hash on VirusTotal. If not already available, perform a scan
30+
* VirusTotal_v3_Scan_File: scan a file on VirusTotal
31+
* VirusTotal_v2_Get_File: check file hash on VirusTotal using old API endpoints
32+
* VirusTotal_v2_Scan_File: scan a file on VirusTotal using old API endpoints
33+
* Intezer Scan: scan a file on Intezer
34+
* Cuckoo_Scan: scan a file on Cuckoo
35+
* HybridAnalysis_Get_File: check file hash on HybridAnalysis
36+
* OTX_Check_Hash: check file hash on OTX Alienvault
37+
* MISP_Check_Hash: check a file hash on a MISP instance
38+
* MISPFIRST_Check_Hash: check a file hash on the FIRST MISP instance
39+
* Yara_Scan_Community: scan a file with community yara rules
40+
* Yara_Scan_Florian: scan a file with Neo23x0 yara rules
41+
* Yara_Scan_Intezer: scan a file with Intezer yara rules
42+
* Yara_Scan_Custom_Signatures: scan a file with your own added signatures
43+
44+
45+
### Observable analyzers (ip, domain, url, hash)
46+
* VirusTotal_v3_Get_Observable: search an observable in the VirusTotal DB
47+
* VirusTotal_v2_Get_Observable: search an observable in the VirusTotal DB using the old API endpoints
48+
* HybridAnalysis_Get_Observable: search an observable in the HybridAnalysis DB
49+
* OTXQuery: scan an observable on Alienvault OTX
50+
* TalosReputation: check an IP reputation from Talos
51+
* Robtex_Forward_PDNS_Query: scan a domain against the Robtex Passive DNS DB
52+
* Robtex_Reverse_PDNS_Query: scan an IP against the Robtex Passive DNS DB
53+
* Robtex_IP_Query: get IP info from Robtex
54+
* GoogleSafebrowsing: scan an observable against GoogleSafeBrowsing DB
55+
* GreyNoiseAlpha: scan an IP against the Alpha Greynoise API
56+
* CIRCLPassiveDNS: scan an observable against the CIRCL Passive DNS DB
57+
* CIRCLPassiveSSL: scan an observable against the CIRCL Passive SSL DB
58+
* MaxMindGeoIP: extract GeoIP info for an observable
59+
* AbuseIPDB: scan an IP against the AbuseIPDB
60+
* Fortiguard: scan an observable with the Fortiguard URL Analyzer
61+
* TorProject: check if an IP is a Tor Exit Node
62+
* MISP: scan an observable on a MISP instance
63+
* MISPFIRST: scan an observable on the FIRST MISP instance
64+
* Shodan_Search: _not_already_available_
65+
66+
## Analyzers customization
67+
You can create new analyzers based on already existing modules by changing the configuration values (`analyzer_config.json`).
68+
69+
The following are all the keys that you can change without touching the source code:
70+
* `leaks_info`: if set, in the case you specify via the API that a resource is sensitive, the specific analyzer won't be executed
71+
* `external_service`: if set, in the case you specify via the API to exclude external services, the specific analyzer won't be executed
72+
* `supported_filetypes`: can be populated as a list. If set, if you ask to analyze a file with a different mimetype from the ones you specified, it won't be executed
73+
* `not_supported_filetypes`: can be populated as a list. If set, if you ask to analyze a file with a mimetype from the ones you specified, it won't be executed
74+
* `observable_supported`: can be populated as a list. If set, if you ask to analyze an observable that is not in this list, it won't be executed. Valid values are: `ip`, `domain`, `url`, `hash`
75+
76+
Also, you can change the name of every available analyzer based on your wishes.
77+
78+
Changing other keys will break the analyzer. In that case, you should think about create a new python module or to modify an existing one.
79+
80+
To contribute to the project, see [Contribute](https://github.com/certego/IntelOwl/blob/master/docs/Contribute.md)
81+
82+
83+
84+
85+
86+
87+
88+
89+

0 commit comments

Comments
 (0)