Merge pull request #240 from intelowlproject/develop

New analyzers, New auth, Many optimizations

Author: eshaan7

.dockerignore

@@ -16,4 +16,5 @@ env_file_app_travis
 docs/
 integrations/
 docker-compose*
-*.quark.log
+*.quark.log
+.pre-commit-config.yaml

.env

@@ -5,24 +5,28 @@
 ### the COMPOSE_FILE variable each separated with ':'. If you are on windows, replace all ':' with ';'.
 ### Reference to Docker's official Docs: https://docs.docker.com/compose/reference/envvars/#compose_file#compose_file
 
-INTELOWL_TAG_VERSION=v1.7.1
+### DO NOT CHANGE THIS VALUE !!
+### It should be updated only when you pull latest changes off from the 'master' branch of IntelOwl.
+INTELOWL_TAG_VERSION=v1.8.0
 
 ###### Default (Production) ######
-
 COMPOSE_FILE=docker-compose.yml
-
 # To run all additional integrations in production
-#COMPOSE_FILE=docker-compose.yml:./integrations/docker-compose.peframe.yml:./integrations/docker-compose.thug.yml:./integrations/docker-compose.capa.yml:./integrations/docker-compose.boxjs.yml:./integrations/docker-compose.apk.yml
+#COMPOSE_FILE=docker-compose.yml:./integrations/docker-compose.peframe.yml:./integrations/docker-compose.thug.yml:./integrations/docker-compose.capa.yml:./integrations/docker-compose.boxjs.yml
 
 ###### For Tests or local development ######
-
 #COMPOSE_FILE=docker-compose-for-tests.yml
-
 # To run all additional integrations in development
-#COMPOSE_FILE=docker-compose-for-tests.yml:./integrations/docker-compose-for-tests.peframe.yml:./integrations/docker-compose-for-tests.thug.yml:./integrations/docker-compose-for-tests.capa.yml:./integrations/docker-compose-for-tests.boxjs.yml:./integrations/docker-compose-for-tests.apk.yml
-
+#COMPOSE_FILE=docker-compose-for-tests.yml:./integrations/docker-compose-for-tests.peframe.yml:./integrations/docker-compose-for-tests.thug.yml:./integrations/docker-compose-for-tests.capa.yml:./integrations/docker-compose-for-tests.boxjs.yml
 
 ###### For travis ######
+#COMPOSE_FILE=docker-compose-for-tests.yml:./integrations/docker-compose-for-tests.peframe.yml:./integrations/docker-compose-for-tests.thug.yml:./integrations/docker-compose-for-tests.capa.yml:./integrations/docker-compose-for-tests.boxjs.yml
 
-#COMPOSE_FILE=docker-compose-for-tests.yml:./integrations/docker-compose-for-tests.peframe.yml:./integrations/docker-compose-for-tests.thug.yml:./integrations/docker-compose-for-tests.capa.yml:./integrations/docker-compose-for-tests.boxjs.yml:./integrations/docker-compose-for-tests.apk.yml
+##### Docker Override #####
+# this can be used to add additional custom configuration
+#:docker-compose-override.yml
 
+##### Nginx Dockerfile for Tests #####
+DOCKERFILE_NGINX=Dockerfile_nginx
+# To test IntelOwl without the interface (IntelOwl-ng Angular APP)
+#DOCKERFILE_NGINX=Dockerfile_nginx_no_angular

.flake8

@@ -8,6 +8,7 @@ exclude =
     Dockerfile,
     docker-compose*,
     venv,
+    docs,
     migrations,
     virtualenv,
     ldap_config.py

.github/CHANGELOG.md

@@ -0,0 +1,101 @@
+# Changelog
+
+[**Upgrade Guide**](https://intelowl.readthedocs.io/en/latest/Installation.html#update-to-the-most-recent-version)
+
+## [v1.8.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.8.0)
+
+**BREAKING CHANGE:**
+- New Token authentication method using the django-rest-durin package. When upgrading IntelOwl to `v1.8.0`, pyintelowl users must upgrade it too to `v2.0.0`. Also, pyintelowl users must create a new valid Token to interact with IntelOwl. More details, [here](https://github.com/intelowlproject/pyintelowl#generate-api-key).
+- Many analyzer variants for VirusTotal and Thug have been removed from `analyzer_config.json` file. 
+Explanation at [#224](https://github.com/intelowlproject/IntelOwl/issues/224). With added docs on how to use custom analyzer configuration at runtime.
+- Other analyzers were renamed due to better clarity and format:
+    * `ActiveDNS_Classic` -> `Classic_DNS`
+    * `ActiveDNS_CloudFlare` -> `CloudFlare_DNS`
+    * `ActiveDNS_CloudFlare_Malware` -> `CloudFlare_Malicious_Detector`
+    * `ActiveDNS_Google` -> `Google_DNS`
+
+
+**NEW INBUILT ANALYZERS:**
+- Added [URLScan](https://urlscan.io/about-api) analyzer.
+- Added [Quad9](https://www.quad9.net/) analyzers (DNS + Malicious_Detector).
+- Added [Phishtank](http://phishtank.org/) analyzer.
+- Added [Stratosphere YARA rules](https://github.com/stratosphereips/yara-rules) analyzer.
+- Upgraded Speakeasy to 1.4.7.
+- Added extra options to DNSDB analyzer + support for API v2.
+- Added [PDFid](https://github.com/mlodic/pdfid) analysis to `PDF_Info` analyzer.
+
+**FIXES/IMPROVEMENTS/Dependency upgrades:**
+
+- Changed Oletools pointer to main repository version (0.56).
+- Changed docs style to use the `Sphinx` theme.
+- Fix for issue [#138](https://github.com/intelowlproject/IntelOwl/issues/138).
+- Update Django and Django-Rest-Framework versions.
+- Updates to recent versions of postgres, nginx and rabbit-mq docker images.
+- Loads of internal changes and code optimizations.
+- Added more info in contributing section of docs.
+
+## [v1.7.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.7.1)
+
+Improvements to recent malicious document analysis:
+* Added [XLMMacroDeobfuscator](https://github.com/DissectMalware/XLMMacroDeobfuscator) analyzer, refer #196 thanks to @0ssigeno 
+* Updated oletools to last available changes
+
+Other:
+* updated black to 20.8b1 and little fix in the docs
+
+## [v1.7.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.7.0)
+
+- 3 new analyzers which can be used out of the box:
+   * `UnpacMe_EXE_Unpacker`: [UnpacMe](https://www.unpac.me/) is an automated malware unpacking service. (Thanks to @0ssigeno)
+   * `CheckDMARC`: [checdmarc](https://github.com/domainaware/checkdmarc) provides SPF and DMARC DNS records validator for domains. (Thanks to @goodlandsecurity)
+   * `Whoisxmlapi`: Fetch WHOIS record data, of a domain name, an IP address, or an email address. (Thanks to @tamthaitu) 
+- Some fixes to Cymru Malware and VT2 analyzers.
+- Now you or your organization can get paid support/extra features/custom integrations for IntelOwl via xscode platform. [Details](https://xscode.com/intelowlproject/IntelOwl).
+
+## [v1.6.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.6.1)
+
+This patch allows to download the most recent docker image of IntelOwl. Previous version was downloading the old (`v1.5.1`) docker image.
+
+Please see [v1.6.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.6.0) for release details.
+
+## [v1.6.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.6.0)
+
+* added new analyzer for [FireEye speakeasy](https://github.com/fireeye/speakeasy)
+* updated [FireEye Capa](https://github.com/fireeye/capa) to 1.1.0
+* updated docs, including instructions for [Remnux](https://docs.remnux.org) users and a new ["How to use pyintelowl" video](https://www.youtube.com/watch?v=fpd6Kt9EZdI).
+
+## [v1.5.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.5.1)
+
+Patch after **v1.5.0**.
+- Fixed `runtime_configuration` JSON serialization bug when requesting file scan.
+
+## [v1.5.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.5.0)
+
+> This release contains a bug that was fixed in v1.5.1. We recommend cloning the `master` branch.
+
+**Features:**
+- Ability to pass a JSON field `runtime_configuration` for dynamic configuration per scan request. [Demo GIF](https://imgur.com/5sxp9JP).
+- IntelligenceX's phonebook API for observables.
+- Increased JWT token lifetime for webapp. ([Ref.](https://github.com/intelowlproject/IntelOwl/issues/163#issuecomment-678223186)).
+
+**Breaking Changes:**
+- Moved `ldap_config.py` under `configuration/` directory. If you were using LDAP before this release, please refer the [updated docs](https://intelowl.readthedocs.io/en/develop/Advanced-Usage.html#ldap).
+
+**Fixes:**
+- Updates and fixes to: `Doc_info`, `PE_Info`, `VirusTotal` v3 and `Shodan_Honeyscore` analyzers.
+- Added migration files for DB.
+
+## [v1.4.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v1.4.0)
+
+- Inbuilt Integration for [Pulsedive](pulsedive.com/) analyzer for IP, URL, Domain and Hash observables. Works without API key with rate limit of 30 requests/minute.
+- Inbuilt integration for Integrated [Quark-engine](https://github.com/quark-engine/quark-engine) for APKs - *An Obfuscation-Neglect Android Malware Scoring System*.
+- Increase `max_length` for `file_mimetype` column. Thanks to @skygrip for the report.
+- Index the fields that are used in `ask_analysis_availability` for faster fetching.
+- Update LDAP documentation, add section about GKE deployments.
+- Fixed: `is_test` issue in `_docker_run`. Thanks to @colbyprior.
+- Fixed: `active_dns` now returns proper result.
+- The base docker image is now based on Python 3.7.
+- Refactor test cases/classes to reduce duplicate code.
+
+
+_For version prior to `v1.4.0`, you can directly refer to the releases tab._

.github/workflows/codeql-analysis.yml

@@ -0,0 +1,67 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+name: "CodeQL"
+
+on:
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [master, develop]
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        # Override automatic language detection by changing the below list
+        # Supported options are ['csharp', 'cpp', 'go', 'java', 'javascript', 'python']
+        language: ['python']
+        # Learn more...
+        # https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#overriding-automatic-language-detection
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+      with:
+        # We must fetch at least the immediate parents so that if this is
+        # a pull request then we can checkout the head.
+        fetch-depth: 2
+
+    # If this run was triggered by a pull request event, then checkout
+    # the head of the pull request instead of the merge commit.
+    - run: git checkout HEAD^2
+      if: ${{ github.event_name == 'pull_request' }}
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file. 
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    #- name: Autobuild
+    #  uses: github/codeql-action/autobuild@v1
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 https://git.io/JvXDl
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

.gitignore

@@ -6,4 +6,5 @@ env_file_app
 env_file_postgres
 env_file_integrations
 venv/
-docker-compose-override.yml
+docker-compose-override.yml
+compose-elk.yml

.pre-commit-config.yaml

@@ -0,0 +1,9 @@
+repos:
+-   repo: https://github.com/psf/black
+    rev: 20.8b1
+    hooks:
+    - id: black
+-   repo: https://gitlab.com/pycqa/flake8
+    rev: 3.8.4
+    hooks:
+    - id: flake8

.travis.yml

@@ -17,8 +17,6 @@ install:
 - sudo docker-compose -f docker-compose-for-travis.yml build
 - sudo docker-compose -f docker-compose-for-travis.yml up -d
 script:
-- sudo docker exec -ti intel_owl_uwsgi black . --check --exclude "migrations|venv"
+- sudo docker exec -ti intel_owl_uwsgi black . --check
 - sudo docker exec -ti intel_owl_uwsgi flake8 . --count
-- sudo docker exec -ti intel_owl_uwsgi python manage.py test tests
-after_success:
-- bash <(curl -s https://codecov.io/bash)
+- sudo docker exec -ti intel_owl_uwsgi python manage.py test tests

Dockerfile_nginx

@@ -1,8 +1,8 @@
 # Stage 1: Get build artifacts from intelowl-ng
-FROM intelowlproject/intelowl_ng:v1.5.1 AS angular-prod-build
+FROM intelowlproject/intelowl_ng:v1.6.3 AS angular-prod-build
 
 # Stage 2: Inject the build artifacts into nginx container
-FROM library/nginx:1.16.1-alpine
+FROM library/nginx:1.19-alpine
 
 COPY --from=angular-prod-build /usr/src/app/dist /var/www/angular_build
 VOLUME /var/log/nginx

Dockerfile_nginx_no_angular

@@ -1,2 +1,2 @@
-FROM library/nginx:1.16.1-alpine
+FROM library/nginx:1.19-alpine
 VOLUME /var/log/nginx