Merge pull request #190 from intelowlproject/develop

Unpacme + whoisxml API + checkdmarc analyzer + Fix VT2 + doc on xs:code

Author: eshaan7

.env

@@ -5,7 +5,7 @@
 ### the COMPOSE_FILE variable each separated with ':'. If you are on windows, replace all ':' with ';'.
 ### Reference to Docker's official Docs: https://docs.docker.com/compose/reference/envvars/#compose_file#compose_file
 
-INTELOWL_TAG_VERSION=v1.6.1
+INTELOWL_TAG_VERSION=v1.7.0
 
 ###### Default (Production) ######
 

.github/FUNDING.yml

@@ -1 +1,2 @@
 liberapay: intelowlproject
+custom: ["https://xscode.com/intelowlproject/IntelOwl"]

README.md

@@ -1,118 +1,53 @@
-![Intel Owl](static_intel/intel_owl.jpeg)
+<img src="static_intel/intel_owl.jpeg" width=500 height=200 alt="Intel Owl"/>
 
 [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/intelowlproject/IntelOwl.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/intelowlproject/IntelOwl/context:python)
 [![CodeFactor](https://www.codefactor.io/repository/github/intelowlproject/intelowl/badge)](https://www.codefactor.io/repository/github/intelowlproject/intelowl)
 [![Build Status](https://travis-ci.com/intelowlproject/IntelOwl.svg?branch=master)](https://travis-ci.org/intelowlproject/IntelOwl)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-# Intel Owl
+<img src="static_intel/xscode-banner.png" width=600 height=125 alt="Get Support"/><br/>
+_For urgent issues and priority support, visit [https://xscode.com/intelowlproject/IntelOwl](https://xscode.com/intelowlproject/IntelOwl)._
 
-Do you want to get **threat intelligence data** about a file, an IP or a domain?
+# Intel Owl
 
-Do you want to get this kind of data from multiple sources at the same time using **a single API request**?
+Do you want to get **threat intelligence data** about a malware, an IP or a domain? Do you want to get this kind of data from multiple sources at the same time using **a single API request**?
 
 You are in the right place!
 
-This application is built to **scale out** and to **speed up the retrieval of threat info**.
-
-It can be integrated easily in your stack of security tools to automate common jobs usually performed, for instance, by SOC analysts manually.
-
-Intel Owl is composed of **analyzers** that can be run to retrieve data from external sources (like VirusTotal or AbuseIPDB) or to generate intel from internal analyzers (like Yara or Oletools)
-
-This solution is for everyone who needs a single point to query for info about a specific file or observable (domain, IP, URL, hash).
+Intel Owl is an Open Source Intelligence, or OSINT solution to get threat intelligence data about a specific file, an IP or a domain from a single API at scale. It integrates a number of analyzers available online and is for everyone who needs a single point to query for info about a specific file or observable.
 
-Main features:
+### Features
 
-- full django-python application
-- easily and completely customizable, both the APIs and the analyzers
-- clone the project, set up the configuration and you are ready to run
-- Official frontend client: **[IntelOwl-ng](https://github.com/intelowlproject/IntelOwl-ng)** provides features such as dashboard, visualizations of analysis data, easy to use forms for requesting new analysis, etc.
+- Provides enrichment of threat intel for malware as well as observables (IP, Domain, URL and hash).
+- This application is built to **scale out** and to **speed up the retrieval of threat info**.
+- It can be integrated easily in your stack of security tools ([pyintelowl](https://github.com/intelowlproject/pyintelowl)) to automate common jobs usually performed, for instance, by SOC analysts manually.
+- Intel Owl is composed of **analyzers** that can be run to retrieve data from external sources (like VirusTotal or AbuseIPDB) or to generate intel from internal analyzers (like Yara or Oletools)
+- API written in Django and Python 3.7.
+- Inbuilt frontend client: **[IntelOwl-ng](https://github.com/intelowlproject/IntelOwl-ng)** provides features such as dashboard, visualizations of analysis data, easy to use forms for requesting new analysis, etc. [Live Demo](https://intelowlclient.firebaseapp.com/).
 
-### Documentation
+## Documentation
 
 [![Documentation Status](https://readthedocs.org/projects/intelowl/badge/?version=latest)](https://intelowl.readthedocs.io/en/latest/?badge=latest)
 
-Documentation about IntelOwl installation, usage, contribution can be found at https://intelowl.readthedocs.io/.
-
-### Blog posts
-
-[Daily Swig Article](https://portswigger.net/daily-swig/intel-owl-osint-tool-automates-the-intel-gathering-process-using-a-single-api)
-
-[Honeynet Blog: v1.0.0 Announcement](https://www.honeynet.org/?p=7558)
-
-[Certego Blog: First announcement](https://www.certego.net/en/news/new-year-new-tool-intel-owl/)
-
-### Free Internal Modules Available
+Documentation about IntelOwl installation, usage, configuration and contribution can be found at https://intelowl.readthedocs.io/.
 
-- Static Document Analysis
-- Static RTF Analysis
-- Static PDF Analysis
-- Static PE Analysis
-- Static Generic File Analysis
-- Strings analysis with ML
-- PE Emulation with Speakeasy
-- PE Signature verification
-- PE Capabilities Extraction
-- Emulated Javascript Analysis
-- Android Malware Analysis
+## Blog posts
 
-**Free modules that require additional configuration**:
+To know more about the project and it's growth over time, you may be interested in reading the following:
 
-- Cuckoo (requires at least one working Cuckoo instance)
-- MISP (requires at least one working MISP instance)
-- Yara (Community, Neo23x0, Intezer and McAfee rules are already available. There's the chance to add your own rules)
+- [Intel Owl on Daily Swig](https://portswigger.net/daily-swig/intel-owl-osint-tool-automates-the-intel-gathering-process-using-a-single-api)
+- [Honeynet: v1.0.0 Announcement](https://www.honeynet.org/?p=7558)
+- [Certego Blog: First announcement](https://www.certego.net/en/news/new-year-new-tool-intel-owl/)
 
-### External Services Available
+## Available services or analyzers
 
-##### required paid or trial API key
+You can see the full list of all available analyzers, [here](https://intelowl.readthedocs.io/en/latest/Usage.html#available-analyzers).
 
-- GreyNoise v2
+| Inbuilt modules 	| External Services 	| Free modules that require additional configuration 	|
+|-	|-	|-	|
+| - Static Document, RTF, PDF, PE, Generic File Analysis<br>- Strings analysis with ML<br>- PE Emulation with Speakeasy<br>- PE Signature verification<br>- PE Capabilities Extraction<br>- Emulated Javascript Analysis<br>- Android Malware Analysis<br>- SPF and DMARC Validator<br>- more... 	| - GreyNoise v2<br>- Intezer Scan<br>- VirusTotal v2+v3<br>- HybridAnalysis<br>- Censys.io<br>- Shodan<br>- AlienVault OTX<br>- Threatminer<br>- Abuse.ch<br>- many more.. 	| - Cuckoo (requires at least one working Cuckoo instance)<br>- MISP (requires at least one working MISP instance)<br>- Yara (Community, Neo23x0, Intezer and McAfee rules are already available. There's the chance to add your own rules) 	|
 
-##### required paid or free API key
-
-- VirusTotal v2 + v3
-- HybridAnalysis
-- Intezer
-- Farsight DNSDB
-- Hunter.io - Email Hunting
-- ONYPHE
-- Censys.io
-- SecurityTrails
-- Intelligence X
-- Pulsedive API (works w/o API key as well)
-
-##### required free API key
-
-- GoogleSafeBrowsing
-- AbuseIPDB
-- Shodan
-- HoneyDB
-- AlienVault OTX
-- MaxMind
-- Auth0
-
-##### needed access request
-
-- CIRCL PassiveDNS + PassiveSSL
-
-##### without api key
-
-- Fortiguard URL Analyzer
-- GreyNoise Alpha API v1
-- Talos Reputation
-- Tor Project
-- Robtex
-- Threatminer
-- Abuse.ch MalwareBazaar
-- Abuse.ch URLhaus
-- Team Cymru Malware Hash Registry
-- Tranco Rank
-- Google DoH
-- CloudFlare DoH Classic
-- CloudFlare DoH Malware
-- Classic DNS resolution
-
-### Legal notice
+## Legal notice
 
 You as a user of this project must review, accept and comply with the license
 terms of each downloaded/installed package listed below. By proceeding with the
@@ -142,21 +77,20 @@ license terms.
 [Quark-Engine](https://github.com/quark-engine/quark-engine)
 [IntelX](https://intelx.io/terms-of-service)
 
-### Acknowledgments
+## Acknowledgments
 
 This project was created and will be upgraded thanks to the following organizations:
 
 <img style="margin-right: 2px" src="static_intel/Certego.png" alt="Certego Logo"/>
 <img style="border: 0.2px solid black" src="static_intel/logo-thp-100.png" alt="Honeynet.org logo">
 
-
-### Google Summer Of Code
+#### Google Summer Of Code
 
 The project was accepted to the GSoC 2020 under the Honeynet Project!! A lot of [new features](https://www.honeynet.org/gsoc/gsoc-2020/google-summer-of-code-2020-project-ideas/#intel-owl-improvements) were developed by Eshaan Bansal ([Twitter](https://twitter.com/mask0fmydisguis)).
 
 Stay tuned for the upcoming GSoC 2021! Join the [Honeynet Slack chat](https://gsoc-slack.honeynet.org/) for more info.
 
-### About the author and maintainers
+## About the author and maintainers
 
 Feel free to contact the main developers at any time:
 - Matteo Lodi ([Twitter](https://twitter.com/matte_lodi)): Author and creator

api_app/script_analyzers/file_analyzers/unpac_me.py

@@ -0,0 +1,99 @@
+import requests
+import logging
+from api_app.script_analyzers.classes import FileAnalyzer
+from api_app.exceptions import AnalyzerRunException
+import time
+from intel_owl import secrets
+from typing import Dict
+
+logger = logging.getLogger(__name__)
+
+
+class UnpacMe(FileAnalyzer):
+    base_url: str = "https://api.unpac.me/api/v1/"
+
+    def set_config(self, additional_config_params):
+        self.api_key_name = additional_config_params.get(
+            "api_key_name", "UNPAC_ME_API_KEY"
+        )
+        private = additional_config_params.get("private", False)
+        self.private = "private" if private else "public"
+        self.__api_key = secrets.get_secret(self.api_key_name)
+        # max no. of tries when polling for result
+        self.max_tries = additional_config_params.get("max_tries", 30)
+        # interval b/w HTTP requests when polling
+        self.poll_distance = 5
+
+    def run(self):
+        if not self.__api_key:
+            raise AnalyzerRunException(
+                f"No API key retrieved with name: {self.api_key_name}"
+            )
+        self.headers = {"Authorization": "Key %s" % self.__api_key}
+        unpac_id = self._upload()
+        logger.info(f"md5 {self.md5} job {self.job_id} uploaded id {unpac_id}")
+        for chance in range(self.max_tries):
+            time.sleep(self.poll_distance)
+            logger.info(
+                f"unpacme polling, try n.{chance + 1}."
+                f" job_id {self.job_id}. starting the query"
+            )
+            status = self._get_status(unpac_id)
+            if status == "fail":
+                logger.error(f"md5 {self.md5} job {self.job_id} analysis has failed")
+                raise AnalyzerRunException("failed analysis")
+            if status != "complete":
+                logger.info(
+                    f"md5 {self.md5} job {self.job_id} id {unpac_id} status {status}"
+                )
+                continue
+            return self._get_report(unpac_id)
+
+    def _req_with_checks(self, url, files=None, post=False):
+        try:
+            if post:
+                r = requests.post(
+                    self.base_url + url, files=files, headers=self.headers
+                )
+            else:
+                headers = self.headers if self.private == "private" else {}
+                r = requests.get(self.base_url + url, files=files, headers=headers)
+            r.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            logger.error(
+                f"md5 {self.md5} job {self.job_id} url {url} has http error {str(e)}"
+            )
+            if post:
+                raise AnalyzerRunException("Monthly quota exceeded!")
+            raise AnalyzerRunException(e)
+        except requests.exceptions.Timeout as e:
+            logger.error(
+                f"md5 {self.md5} job {self.job_id} url {url} has timeout error {str(e)}"
+            )
+            raise AnalyzerRunException(e)
+        except requests.exceptions.RequestException as e:
+            logger.error(
+                f"md5 {self.md5} job {self.job_id} url {url} failed with error {str(e)}"
+            )
+            raise AnalyzerRunException(e)
+        return r
+
+    def _upload(self) -> str:
+        with open(self.filepath, "rb") as f:
+            file_data = f.read()
+        files = {"file": (self.filename, file_data)}
+        r = self._req_with_checks("private/upload", files=files, post=True)
+        response = r.json()
+        if "id" not in response:
+            raise AnalyzerRunException(
+                f"md5 {self.md5} job {self.job_id} function upload id not in response"
+            )
+        return response["id"]
+
+    def _get_status(self, unpac_me_id) -> str:
+        response = self._req_with_checks(f"{self.private}/status/{unpac_me_id}")
+        return response.json().get("status", False)
+
+    def _get_report(self, unpac_me_id) -> Dict:
+        response = self._req_with_checks(f"{self.private}/results/{unpac_me_id}")
+        return response.json()

api_app/script_analyzers/observable_analyzers/checkdmarc.py

@@ -0,0 +1,31 @@
+import subprocess
+import json
+from shutil import which
+
+from api_app.script_analyzers import classes
+from api_app.exceptions import AnalyzerRunException
+
+
+class CheckDMARC(classes.ObservableAnalyzer):
+    check_command: str = "checkdmarc"
+
+    def run(self):
+        if not which(self.check_command):
+            self.report["success"] = False
+            raise AnalyzerRunException("checkdmarc not installed!")
+
+        process = subprocess.Popen(
+            [self.check_command, self.observable_name],
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+        process.wait()
+        stdout, stderr = process.communicate()
+
+        dmarc_info = stdout.decode("utf-8"), stderr
+
+        dmarc_str = dmarc_info[0]
+
+        dmarc_json = json.loads(dmarc_str)
+
+        return dmarc_json

api_app/script_analyzers/observable_analyzers/cymru.py

@@ -20,15 +20,14 @@ def run(self):
         # reference: https://team-cymru.com/community-services/mhr/
         # if the resolution works, this means that the file is reported
         # as malware by Cymru
-        resolutions = []
+        domains = None
         try:
             query_to_perform = f"{self.observable_name}.malware.hash.cymru.com"
             domains = socket.gethostbyaddr(query_to_perform)
-            resolutions = domains[2]
         except (socket.gaierror, socket.herror):
             logger.info(f"observable {self.observable_name} not found in HMR DB")
-        if resolutions:
+        if domains:
             results["found"] = True
-        results["resolution_data"] = resolutions
+            results["resolution_data"] = domains[2]
 
         return results

api_app/script_analyzers/observable_analyzers/talos.py

@@ -32,7 +32,7 @@ def run(self):
     def updater():
         try:
             logger.info("starting download of db from talos")
-            url = "https://www.talosintelligence.com/documents/ip-blacklist"
+            url = "https://snort.org/downloads/ip-block-list"
             r = requests.get(url)
             r.raise_for_status()
 

api_app/script_analyzers/observable_analyzers/vt2_get.py

@@ -18,39 +18,43 @@ def run(self):
                 f"No API key retrieved with name: {self.api_key_name}"
             )
 
-        return vt_get_report(
+        resp = vt_get_report(
             self.__api_key, self.observable_name, self.observable_classification
         )
 
+        resp_code = resp.get("response_code", 1)
+        verbose_msg = resp.get("verbose_msg", "")
+        if resp_code == -1 or "Invalid resource" in verbose_msg:
+            self.report["errors"].append(verbose_msg)
+            raise AnalyzerRunException(f"response code {resp_code}. response: {resp}")
+        return resp
 
-def vt_get_report(api_key, observable_name, observable_classification):
+
+def vt_get_report(api_key, observable_name, obs_clsfn):
     params = {"apikey": api_key}
-    if observable_classification == "domain":
+    if obs_clsfn == "domain":
         params["domain"] = observable_name
         uri = "domain/report"
-    elif observable_classification == "ip":
+    elif obs_clsfn == "ip":
         params["ip"] = observable_name
         uri = "ip-address/report"
-    elif observable_classification == "url":
+    elif obs_clsfn == "url":
         params["resource"] = observable_name
         uri = "url/report"
-    elif observable_classification == "hash":
+    elif obs_clsfn == "hash":
         params["resource"] = observable_name
         params["allinfo"] = 1
         uri = "file/report"
     else:
         raise AnalyzerRunException(
-            "not supported observable type {}. Supported are: hash, ip, domain and url"
-            "".format(observable_classification)
+            f"not supported observable type {obs_clsfn}. "
+            "Supported are: hash, ip, domain and url."
         )
 
     try:
         response = requests.get(vt_base + uri, params=params)
         response.raise_for_status()
     except requests.RequestException as e:
         raise AnalyzerRunException(e)
-    result = response.json()
-    response_code = result.get("response_code", 1)
-    if response_code == -1:
-        raise AnalyzerRunException(f"response code -1. result:{result}")
-    return result
+
+    return response.json()