Fixes to dependencies, documentation build and Mordor

Hotfix release

Fixes

• 08b048d@If AzureCLI section is not in msticpconfig.yaml, no longer throws exception
• Updated AzureSentinel API notebook so that it has data (obfuscated).
• 3384c87@Updated .pre-commit-config.yaml to exclude tests from pylint and flake8 checks
  7b276a3@Adding azure-mgmt-core>=1.2.1 to requirements.txt
• Added exceptions to test_pkg_imports.py to skip vt and vt_graph_api (now extras)
• d3ad345@ Fixing to mordor_driver for Mitre web site change and making web scraping from Mitre more resilient
• Adding two scripts for testing pre-release msticpy.
• Some additions to VTLookupV3.ipynb text and trapping for import of nest_asyncio without installing
• ReadTheDocs conf.py now retrieves version number from main package build
• d8cc378@Updated ReadtheDocs requirements.txt
• db6e6f9@Changes to requirements,txt to remove unneeded dependencies (#114)*
• Added exclusions to conf.py and setup.py to skip install of some dependencies for ReadTheDocs (e.g. Kqlmagic).
  These packages will be mocked in their build - previously they caused dependency conflict errors in pip install
• Bug in azure_auth_core.py - raise from should have been an exception object, not a class
• Bug in secret_settings.py - potential uninitialized variable
• Updated test_mordor_driver.py to remove files on completion, Also marked file download tests as CI-only
• Added pylint and flake8 as pre-commit hooks - should stop fewer errors getting into build
• Some tidying of code (Sourcery) in azure_auth_core and secret_settings.
• Updated version to 0.8.7
• Fixing some typos and linting warnings in the documentation. Clarifying language.
• 3c87222@Updated vt-py version in requirements.txt (#113)

AzureML Hotfix

Hotfix release for compatibility problem with Azure ML notebooks.

Authentication, Mordor and VirusTotal v3 API

This release includes several new features:

• Support for VirusTotal v3 API
• Streamlined authentication for Azure using chained authentication
• Azure Sentinel API support
• Mordor data provider and data browser

New Features

• VirusTotal V3 API support (#97 and #106)
  The VirusTotal team contributed a new module that gives access to the latest version of their API. This
  includes building of relationships between malicious items and viewing the relationship graph.
  (thanks to Andres Ramirez/@aramirezmartin and Juan Infantes at VirusTotal!).
  See the VirusTotal V3 notebook
  To use this feature you will need to install msticpy with the "vt3" extra

   pip install msticpy[vt3]

• Azure Authentication (#109)
  We previously has several mechanism to authenticate to Azure components - mainly Key Vault and the Azure Data module.
  This caused unnecessary authentications. These now use a single mechanism and can also piggy-back on existing AzureCLI
  and Managed Service Identity (MSI) credentials. For example, if you have run az login from Azure CLI before launching
  your notebook the credentials from Azure CLI will be used to fetch the required authentication token for other
  Azure modules in MSTICPy. Note: this does not yet include Kqlmagic but we hope to have that in the next release.
  To see this working, check out the AzureSentinel API notebook link below.
• Azure Sentinel API Support (#109)
  This release includes initial support for Azure Sentinel APIs to retrieve Azure Sentinel Workspaces, Alert Rules,
  Hunting Queries and Hunting bookmarks (more to follow in a later release).
  See the Azure Sentinel API notebook
• Mordor data (#108)
  Mordor is an open source data repository of logs illustrating different kinds
  of adversary behavior and referenced to the Mitre ATT&CK framework. The MSTICPy
  Mordor package includes a notebook browser to search through and display the data sets and a dataprovider
  allowing you to download datasets into pandas DataFrames to use in your notebooks.
  See the Mordor Data notebook

Experimental Features

• Split queries by time period (#110)
  This allows you to split a very large or long-running query into subsets split along by time interval (e.g. 1day, 1hour, etc.)
  The DataProvider data queries now accept a split_query_by option to split queries by time period. The time period
  follows the pandas Timedelta syntax e.g. "1D", "6H". The query is divided, each piece run sequentially and the results
  re-assembled into a single DataFrame.

Fixes

1bce3f3@Bug fix and nasty workaround for old test setup removed in pkg_config.py
0a21b59@Updating pre-commit version for black
36d2539@Updated formatting for new black version
7897cb9@Moving VT3 dependencies to extras. (#111)

GetEnvironmentString-hotfix

Hot fix for - GetEnvironmentString widget throws exception when variable is already defined

Azure-mgmt-monitor fix

This is a maintenance release to temporarily fix an import conflict with the release of 1.0.0 of azure-mgmt-monitor.

It also includes a pre-release version of the VirusTotal V3 API library

New Widgets, Data Obfuscation, TimeSeries time range extraction

New Features

• #90 - new gettext and option button widgets.
• Registered widgets - some widgets (GetText and QueryTimes) now have a "memory" - so if used in Jupyter notebooks and you accidentally re-run the cell they will revert to their last settings rather than the defaults.
• Registered widgets can also accept notebook parameters from papermill
• Added query_browser module
• Added method to tilookup and data_providers modules to access respective browser widgets.
• #93 Added class diagrams for DataProviders, TIProviders and Secrets sub-packages.
• #94 - Added function to extract anomalous time ranges from time series anomaly results
• Added functions to obfuscate/hash data to protect PII
• #96 kql query failure and nbinit notebook initialization now have friendly errors
• Updated data queries list

Fixes

• #90 Splunk and data_providers fixes

• Tooltip formatting fix for datetimes in timeline and timeseries Bokeh tools

• #92 - Fixes for deprecated networkx API
• #93 Error in pkg_config validate when no config sections are populated.
• #95 - Removed redundant typing library from requirements
• #96 - Miscellaneous linter and spelling fixes
• Error in pkg_config validate when no config sections are populated.
• Updated Splunk queries to use datetime type for parameters.
• Adding some extra checks for null entries in msticpyconfig in pkg_config
• Making nbinit skip but report any exceptions while validating msticpyconfig
• Change dataproviders to use custom paths outside of the package
• Fix to entity entityschema
• Fix to kql_driver to handle running if not in IPython.
• Add version param to test-pypi-test-pkg.cmd help.
• Unit test for query_browser.
• Formatting using updated black v20
• Fixing conda-reqs-pip.txt requirements
• Updating msal req to ~=1.0.0
• Fixing mypy errors, incorrect annotation in query_source
• Fixing time format bug in timeline
• 3fc07e7@timeformat re-ordering
• 9d20842@updated requirements for dateutil bug-fixes

Dependency version fix

Maintenance release to fix import errors caused by incorrect package dependency versions.
Also includes minor documentation updates.

Splunk data connector and data uploaders

The main features of this release are a data provider for Splunk and data uploaders for Azure Sentinel and Splunk. The Splunk provider uses the Spunk SDK and makes it eas to use msticpy functionality with results from Splunk queries.

The data uploaders let you upload results or additional data to either Azure Sentinel or Splunk.

New Features

splunk connector (#81)

• Authentication and initialization unified with existing data providers
• Parameterized template queries runnable from data provider class
• Retrieve and display saved searches as part of the available query set
• Splunk provider documentation notebook

Data Uploaders (#87)

• uploaders from pandas DataFrame, CSV/delimited file and folder
• upload to Azure Sentinel or Splunk
• documentation notebook and read-the-docs page

Data providers (#81)

• fc046a5Added support for populating dynamic query set at connect time (after connect)
• Added populating queries with SavedSearches in splunk_driver
• Added provider specific formatting for parameters (e.g. how a given query language expects a list of items or a datetime to be formatted)
• Support for fully hierarchical query list (to help organize queries into categories, subcategories, etc.)

4bbf785 Blackhat Demo Notebook

Fixes

87dab39 Adding unit tests for kql and splunk drivers (latter has a failure)
7123511 Bug in data_providers and param_extractor
e2ea5c6 Fix for tooltip formatting for timeline charts
2a5a734 Error in pkg_config validate when no config sections are populated.
a141f3b Temporarily restricting pandas version to <=1.0.5
9901b72 TI Browser widget
d1e6430 Fixes to splunk driver test

TI Results Browser

New Features

• Threat Intelligence results browser widget (#84)

Fixes

805d71d@ Error in pkg_config validate when no config sections are populated. (#85)
860f7a4@Fixing URLs in exception help links
f92ef4d@update SelectAlert
1a72495@Fixing mypy error in wsconfig
Fixing odd test break in ti_providers (didn't have default msticpyconfig in current directory)
Removing redundant test in test_nb_tools
Fixing bug in unit_test_lib - setting ENV VAR to "" instead of removing it.
a031c2a@Incorrect capitalization in Readme
c6468aa@Left in an unwanted cell from testing
d8098ee@Test error in EventClustering notebook
1c14a12@Fixing timestamp Timezone issue in process_tree_utils and eventcluster
a08a348@Added User exception to BrowshotAdded exception handling to ip_utils.def_whois_info
Fixed Output widgets where action function returns single dataframe
Fixed syslog_utils.cluster_syslog_logons_df so that it uses only source data to generate datetime objects - this should avoid TypeErrors from pandas when comparing datetimes where one dt has TZ info and the other doesn't. As long as the source is consistent this should avoid this error.
4f73b44@Making pkginstall and notebook init errors more friendly
1dafc1f@Fixed problem with test cases not being able to do absolute imports of msticpy modules(deleting rogue init.py at root of package
Updating Readme to make images clearer.
Add check_version to root init.py to check version against latest on PyPI
Also added check version code to tools.
Minor change to nbinit._check_config() to make testing easier.
8bbf417@Prevent default output from queries (Kqlmagic post 0.1.111)
8eff722@Fix for the problem with pandas datetime representation - you cannot compare timezone-aware and timezone-naiive dates. Redoing logic so that the code only uses timezone awareness state from data values. This should work as long as all or none of the data sets
have timezone info.

Package Conflict Fix

Maintenance release to fix some package version conflicts.