several fixes + 2 new analyzers

IMPORTANT FIX
We changed docker-compose file names for optional analyzers. In the v.2.0.0 this broke Docker Hub builds, causing them to fail. Please upgrade to this version to be able to use the optional analyzers again.

NEW INBUILT ANALYZERS:

• added CRXCavator analyzer for malicious Chrome extensions
• added CERT Polska MWDB analyzer for malicious files

FIXES/IMPROVEMENTS/Dependency upgrades:

• updated Quark_Engine to last version and fixed rules
• Maxmind analyzer now retrieves City data too
• fixes for Qiling analyzer
• re-enabled APKiD_Scan_APK_DEX_JAR analyzer for Android samples
• adjusts to auto-build, PR template and documentation

Happy First Birthday IntelOwl!

Note: There were some major bugs in this version so we request you to checkout the latest version here instead.

Happy 1st Birthday IntelOwl! The gift is a new major release 🚀

BREAKING CHANGES:

• moved docker and docker-compose files under docker/ folder.
• users upgrading from previous versions need to manually move env_file_app, env_file_postgres and env_file_integrations files under docker/.
• users are to use the new start.py method to build or start IntelOwl containers
• moved the following analyzers together in a specific optional docker container named static_analyzers.
  • Capa
  • PeFrame
  • Strings_Info_Classic (based on flarestrings)
  • Strings_Info_ML (based on stringsifter)

Please see docs to understand how to enable these optional analyzers

NEW INBUILT ANALYZERS:

• added Qiling file analyzer. This is an optional analyzer (see docs to understand how to activate it).
• added Stratosphere blacklists analyzer
• added FireEye Red Team Tool Countermeasures Yara rules analyzer
• added emailrep.io analyzer
• added Triage analyzer for observables (search API)
• added InQuest analyzer
• added WiGLE analyzer
• new analyzers were added to the static_analyzers optional docker container (see docs to understand how to activate it).
  • FireEye Floss strings analysis.
  • Manalyze file analyzer

FIXES/IMPROVEMENTS/Dependency upgrades:

• upgraded main Dockerfile to python 3.8
• added support for the generic observable type. In this way it is possible to build analyzers that can analyze everything and not only IPs, domains, URLs or hashes
• added Multi-queue option to optimize usage of Celery queues. This is intended for advanced users.
• updated GUI to new IntelOwl-ng version
• upgraded Speakeasy, Quark-Engine and Dnstwist analyzers to last versions
• moved from Travis CI to Github CI
• added CodeCov coverage support (so we will be improving the test coverage shortly)
• moved PEFile library pointer to a forked pip repo that contains some fixes.
• fix to log directiories that could result in some optional analyzers to break
• added milliseconds to logs

fix release with some improvements and new analyzers

This version was released earlier to fix installation problems triggered by the new version of pip (peepdfpackage was incompatible and had to be changed).

NEW INBUILT ANALYZERS:

• Added MalwareBazaar_Google_Observable analyzer: Check if a particular IP, domain or url is known to MalwareBazaar using google search
• Added InQuest YARA rules analyzer.
• Added StrangerealIntel Daily Ioc Yara rules analyzer.

FIXES/IMPROVEMENTS/Dependency upgrades:

• changed peepdf pip repo to peepdf-fork to fix broken installation
• adjustments to documentation
• upgraded quark-engine to v20.11
• fixes to UnpacMe_EXE_Unpacker and PE_Info analyzers
• managed RAM utilization by celery to avoid issues when using IntelOwl for a lot of analysis.
• added PR template
• removed nginx banner

new analyzers + some tweaks

NEW INBUILT ANALYZERS:

• Added Triage file analyzer.
• Added Zoomeye analyzer.
• Added Dnstwist analyzers.
• Added Ipinfo analyzer.
• Added ReversingLabs YARA rules analyzer.
• Added Samir YARA rules analyzer.

FIXES/IMPROVEMENTS/Dependency upgrades:

• several little fixes on analyzers (OTXQuery, DNSDB, Classic_DNS, Fortiguard, XMLDeobfuscator)
• increased filename max_length to 512
• added validation checks to avoid DB problems
• upgraded Yara to v4.0.2
• added Yara rule location to the analyzer output

Major Release: v1.8.0; Nov'20

Refer to CHANGELOG.md.

Improvements to recent malicious document analysis

Update Guide

Improvements to recent malicious document analysis:

• Added XLMMacroDeobfuscator analyzer, refer #196 thanks to @0ssigeno
• Updated oletools to last available changes

Other:

• updated black to 20.8b1 and little fix in the docs

Unpacme + whoisxml API + checkdmarc analyzer + Fix VT2

Update Guide

• 3 new analyzers which can be used out of the box:
  • UnpacMe_EXE_Unpacker: UnpacMe is an automated malware unpacking service. (Thanks to @0ssigeno)
  • CheckDMARC: checdmarc provides SPF and DMARC DNS records validator for domains. (Thanks to @goodlandsecurity)
  • Whoisxmlapi: Fetch WHOIS record data, of a domain name, an IP address, or an email address. (Thanks to @tamthaitu)
• Some fixes to Cymru Malware and VT2 analyzers.
• Now you or your organization can get paid support/extra features/custom integrations for IntelOwl via xscode platform. Details.

[Patch] fixed version number - Added SpeakEasy, upgraded Capa and updated docs

This patch allows to download the most recent docker image of IntelOwl. Previous version was downloading the old (v.1.5.1) docker image.

Please see v1.6.0 for release details.

Upgrade guide

Added SpeakEasy, upgraded Capa and updated docs

• added new analyzer for FireEye speakeasy
• updated FireEye Capa to 1.1.0
• updated docs, including instructions for Remnux users and a new "How to use pyintelowl" video

[Patched] IntelX phonebook API + Dynamic Analyzer's Conf.

Patch after v1.5.0.

• Fixed runtime_configuration JSON serialization bug when requesting file scan.