Friendly Exceptions

Mostly housekeeping release.
We've introduced a set of friendly exceptions for common configuration-related problems that users might encounter while using msticpy in notebooks. In most cases, the cause is lack of a config item such as an API key or setting. Hitting one of these exceptions gives you a few lines of explanatory text plus links to documentation on readthedocs to help you fix it. In most cases the traceback is irrelevant so this is not shown for these exception types (although this can be re-enabled).

Adding some new capabilities like being able to use our time series analysis module on any time-stamped log data. We've also done a lot of documentation additions/improvements - for Anomalous Sequence, TimeSeries and msticpy configuration plus miscellaneous freshening and improving.

A lot of the other work is related to compatibility with nteract environment in Azure Machine Learning - this will soon be the default notebook environment for Azure Sentinel.

New Features

• Friendly exceptions and exception framework for notebooks #73
• Replaced the use of the ipywidgets Output widget with IPython updatable display() objects. nteract/AML compat. #71
• Added timeseries decomposition (using Statsmodel STL) so that you can do time series analysis on any data #69
  (previously we relied on Kql/Azure Sentinel to do the decomposition/analysis part). Documentation and notebook added
• Update to Anomalous Sequence modules
  • Added modellable_params argument in model classes - this can be used to override the default of using rough heuristics to
    decide which params have modellable values, and instead, manually specify the parameters. #65
  • Added sample notebook and RST/ReadtheDocs documentation for Anomalous Sequence.
• Added check_version() function that will check the current version against latest on PyPI

  import msticpy
  msticpy.check_version()
  

• Relaxing some version requirements in setup.py/requirements.txt to match AzureNotebooks resulting in quicker install #68
• Updated display of logon details in nbdisplay
• Updated README to add missing details, correct some things and add a few more images.
• Added documentation for Azure Sentinel configuration for notebooks - how to use the various config files.
• Updated a lot of the introductory sections to readthedocs to bring up-to-date and align with README
• Replace some badly-formatted tables in readthedocs docs
• Added analysis subpackage to docs so that Anomaly Sequence and TimeSeries module/API docs appear in the API doc tree
  (readthedocs)
• Adding new queries for Notebooklets project #67

Fixes

• Some fixes to comp_reqs.py test tool to show missing packages and handle version comparison operators.
• New release of pandas/numpy surfaced a bug where we were doing datetime comparisons between timezone
  naive and timezone-aware datetimes - fixed in timeline, processtree and eventcluster.
• New behavior in Kqlmagic (> 0.1.111) causes queries in the middle of a cell to output by default. Added workaround
  to suppress this.
• Rogue init.py in root of repo (had been there forever) was preventing test cases using absolute imports of tested
  modules. removed the offending file and updated all tests to use absolute imports
• Seems that we'd been inadvertently (at some point) including test files in our setuptools/PyPI package - these should
  now be gone.
• Fixed an issue with GeoIPLite (maxmind) database download and cleaned up logic.
• Added better exception handling in nbinit
• Making pkginstall and notebook init errors more friendly
• Removing some deprecated terms.
• Fixing timestamp Timezone issue in process_tree_utils, eventcluster, process_tree and syslog_utils.cluster_syslog_logons_df
• Fixing/constraining some azure dependencies for Sphinx #68
• Fix for schema property in kql_driver #70

LocalData Provider

New Features

• db86480:
  • LocalDataDriver for using CSV and pickled DF files as a QueryProvider (#64)
    This is primary for demonstration and test purposes where you do not have access to online data sources. It
    replicates the functionality of QueryProvider allowing drop-in replacement in existing notebooks.
  • Updated DataQueries.rst ReadtheDocs page with new queries
  • Add documentation for LocalDataDriver to DataProviders.rst and updated section on creating query files.
• 66a66d2:
  • Checked in notebook to create DataQueries.rst
  • Added "AzureSentinel" alias for LogAnalytics DataEnvironment

Breaking Changes

• db86480:
  • Removed deprecated kql.py, query_builtin_queries, query_mgr.py, query_schema.py
  • Changed location of query_defns.py and made pkg reference updates in several modules and notebooks.
  • Some fixes to support local_data_driver in query_store.py, driver_base.py and data_providers.py
  • Unit test - test_localdata_queries.yaml and supporting data and query files.
  • Fixed test in test_utils.py to work on Linux
  • Reduced warnings produced during pytest run to something more reasonable (mainly by removing
    deprecated code
• 8a32ad5:
  • Changed tilookup and kql_base/kql_driver so that handling failure to load is a bit friendlier. E.g. running
    TILookup in a non-IPython environment (with ASTI provider) will now just cause a warning, not an exception.
  • kql_driver.py also updated to check for get_ipython() returning None and output friendlier message.
  • Changed driver_base.py and derived class to take additional QuerySource parameter for query() method -
    not yet used but required so that we can implement driver-specific checks on query parameters.

Anomaly Sequence Detection and Morph Charts (re-release)

This release includes:

Anomaly sequence analysis and visualization using Markov chain @karishma-dixit
Morph Chart visualization of log events @petebryan

(originally released as v0.4.1 but updated to v0.5.0)

New Features

• Anomalous sequences (#60)
  Markov Chain anomaly analysis for sequences of commands/patterns in a session
• Morph Charts visualization - 3D visualization of event data using experimental (#58)
  Morph Charts exploration
• nbinit: a neater and more robust startup/setup function for Jupyter notebooks
  handling package installs, imports and option setting (#62)
• Azure Sentinel Queries
  • Added two Logon fail queries for Linux (#62)
  • Add Linux logons for host
• Added msticpy.common.pkg_config.validate_config() to validate current config
  or external config file (#62)

Fixes

• f78a29e:
  • Change return type on for bokeh graphs to return whole layout
  • Improved geoip error messages when Api key is missing
  • Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
• 31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
• 827477b: make titles consistent on the widgets page (#59)
• 7964b5f: Fix to utility.py - check_and_install_missing_packages to all package version to be specified.
• f793d55:
  • Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
  • Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
  • Fixed an error in test-pypi-test-pkg.cmd
• 3e42e42: Doc fix and OutOfBoundsDatetime catch
• efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting (#55)
• 0ad166a: fixing headings in rst docs for timeseries
• 606fc8f: Fixing broken Readthedocs link (#53)
• 4810e1f: Fixing some documentation omissions/errors (#52)
• 43bbd3c: Updating pylintrc to change limits for some checks.
• f50eec2: Notebooklet queries and timeline hide option
• 13c3f3f Flake8 error with unknown "QuerySource" (#63)
• 9921352 Adding pkgs to conda-reqs-pip.txt Removing Python 3.7 version setting from pre-commit
• 921370c (#63)
  • requirements.txt and setup.py changes to avoid version conflicts (causing sphinx to fail)updated version to 5.0
• c900386 Fixed issue causing test failure (#63)
• 5c9db2d Adding get_all_entities feature used in Alerts Notebook (#63)

Anomaly Sequence Detection and Morph Charts

This release includes:

Anomaly sequence analysis and visualization using Markov chain @karishma-dixit
Morph Chart visualization of log events @petebryan

New Features

• Anomalous sequences
  Markov Chain anomaly analysis for sequences of commands/patterns in a session
• Morph Charts visualization - 3D visualization of event data using experimental
  Morph Charts exploration
• nbinit: a neater and more robust startup/setup function for Jupyter notebooks
  handling package installs, imports and option setting
• Azure Sentinel Queries
  • Added two Logon fail queries for linux
  • Add Linux logons for host
• Added msticpy.common.pkg_config.validate_config() to validate current config
  or external config file

Fixes

• f78a29e:
  • Change return type on for bokeh graphs to return whole layout
  • Improved geoip error messages when Api key is missing
  • Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
• 31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
• 827477b: make titles consistent on the widgets page (#59)
• 7964b5f: Fix to utility.py - check_and_install_missing_packages to all package version to be specified.
• f793d55:
  • Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
  • Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
  • Fixed an error in test-pypi-test-pkg.cmd
• 3e42e42: Doc fix and OutOfBoundsDatetime catch
• efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting
• 0ad166a: fixing headings in rst docs for timeseries
• 606fc8f: Fixing broken Readthedocs link (#53)
• 4810e1f: Fixing some documentation omissions/errors (#52)
• 43bbd3c: Updating pylintrc to change limits for some checks.
• f50eec2: Notebooklet queries and timeline hide option

Azure Data, TimeSeries and Key Vault Secrets

This release includes:

• Expansion of Azure Data API for retrieving additional data about subscriptions and resources from Azure APIs.
• Time Series anomaly detection for arbitrary Kusto data sets together with visualization of time series charts in
  Jupyter Notebooks using Bokeh Charts.
• Using KeyVault and Python Keyring to store secrets used to authenticate to web data providers.
  Examples include API keys for Threat Intel and Geo IP Providers. Other provider types will be included in
  a future release.

New Features

• Azure data expansion and documentation
• Keyvault and keyring secrets management with support for multiple Azure clouds
  • config2kv.py KV secret update tool
• Timeseries - Bokeh with KQL and documentation
  • KQL generic time series decomposition queries
  • Bokeh time series visualization
• Added pandas version of get_whois_info and added as DataFrame accessor function.
• Added cmd script to test PyPi test deployment
• Added Conda package requirements files
• Updated TI providers to provide more consistent output and reduce false positives
  • Using text rather than number to express severity
• Made TISeverity class comparable and parsable from string or int
• Added mp_demo_data.py notebook helper to tools.
• SecurityAlert has more flexible recognition of entities
• Added additional dependencies for azure mgmt, keyvault and others.

Fixes

• Fixed get_ip_type ordering to return more accurate IP types
• Fix entity extraction in SecurityAlert to allow nested entities to work correctly
• Additional test cases

Pandas Extensions

This release includes early implementations of pandas extensions so that you can invoke msticpy functionality directly from a DataFrame:

  my_events_df.mp_timeline.plot()
  my_proc_events_df.mp_process_tree.plot()

So far, IoCExtract, Base64Unpack, Timeline and ProcessTree have these extensions but we will be adding them to more msticpy modules over time.

Also, IoCExtract and Base64 decode functions have IPython magics (%%ioc and %%b64) allowing you to paste a block of text into a notebook cell and run the function directly on that text.

Most other changes are primarily maintenance and house-keeping improvements such as increasing unit testing code coverage.

New Features

• process_tree - added pandas extension and changed main function so that it returns the plot figure and layout
• timeline - added pandas extension. added support for DateTime column in Tooltips (display as date time rather
  than number)
• base64unpack - added pandas extension, added IPython %%b64 magic
• iocextract - added pandas extension, added IPython %%ioc magic
• Added documentation and notebook examples for the pandas extensions and magics.
• wsconfig - added method to display available workspaces
• README.MD - added some graphics to brighten the page up a little
• Added unit test test_folium.py
• Adding FoliumMap.ipynb sample notebook
• Added additional geolocation centering functions for FoliumMap
• Updates to GeoIPLookups.ipynb
• Add parameter checks to timeline.py and process_tree.py so that invalid **kwargs produced a helpful error message.
• Added requirements-dev.txt

Fixes

• Typos in AzureData.rst
• Adding GeoIP tests.
• Removing deprecated lines from coverage reports.
• Cleaned up pytest coverage report.
• Adding suppression file for credscan false positives
• Removing SecurityAlertandEntities notebook with misleading content
• Removed failing cell from end of GeoIPLookups notebook
• Fixed a few errors in foliummap.py
• Fixed bug in GeoIP DB downloader
• Changed foliummap center functions to use median by default
• Removed largely redundant os_family param from iocextract.py functions
• Fixed sectools_magics iocextract class
• Update test_ioc_extractor for new parameters
• domain_tools - changed tld_index and ssl_bl attributes to properties that auto-load on first use (prevents remote
  http request if data on class instantiation)
• Added more tests for utility.py
• Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
• Tidied up/refactored some code in base64unpack.py

mstipy User guide

New Features

The documentation now includes a user guide covering many aspects of msticpy
It includes the following sections:

• Getting started section (Installation and configuration)
• Data Acquisition (querying and data)
• Data Enrichment (GeoIP, Threat Intel)
• Data Analysis (IoC extraction, decoding, clustering
• Visualization (Event timeline, Process tree, Mapping, widgets)

Documentation is on ReadTheDocs

Fixes

• Broken links and outdated docs updated
• Fixes to some unit tests

MDATP, Azure, ProcessTree

Overview

Note: This release is consolidation of v0.2.8 and some additional features and fixes. Adding everything to the release notes for this version since the previous one did not get published to PyPi.

This release contains three important features:

• Query support for Microsoft Defender ATP
• Interactive Process Tree browser.
• Support for querying Azure properties for subscriptions and resources

New Features

• Microsoft Defender Query Support. Added a query provider/driver to
  query Defender alerts, machines, processes and arbitrary KQL queries of
  the Hunting data
• Template queries for MDATP for hunting and standard entities
• Process Tree Viewer - Bokeh interactive graphical view for one or
  more Process Trees in a data set. Supports both Windows and Linux.
• Process tree utilities - data library to create and query process trees.
• Azure properties of subscriptions and resources such as VMs can be
  queried from Notebooks.
• Query providers now accept ISO-string format for datetime fields
  for queries (in addition to datetime and timedelta)
• Added Progress widget to nbwidgets.
• Added config support for GeoIP providers from msticpyconfig.yaml
  • GeoIP classes try to obtain API key from config if not supplied
• Refactored ti_provider_settings to generic provider_settings module

Fixes

• Miscellaneous linting/checker bugs
• Spelling and path errors in docs
• Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
• Updating dependency to Bokeh 1.40
• Fixed timeline legend bug
• Fix for Maxmind requiring authentication for GeoLite DB download (GeoIP classes
  are no longer loaded by default.
• Added missing pytz and pyyaml packages to requirements.txt and setup.py.
• GeoLite2 url and archive extraction changes

MDATP/AzureData Support and ProcessTree Viewer

Overview

This release contains three important features:

• Query support for Microsoft Defender ATP
• Interactive Process Tree browser.
• Support for querying Azure properties for subscriptions and resources

New Features

• Microsoft Defender Query Support. Added a query provider/driver to
  query Defender alerts, machines, processes and arbitrary KQL queries of
  the Hunting data
• Template queries for MDATP for hunting and standard entities
• Process Tree Viewer - Bokeh interactive graphical view for one or
  more Process Trees in a data set. Supports both Windows and Linux.
• Process tree utilities - data library to create and query process trees.
• Azure properties of subscriptions and resources such as VMs can be
  queried from Notebooks.
• Query providers now accept ISO-string format for datatime fields
  for queries (in addition to datetime and timedelta)
• Added Progress widget to nbwidgets.

Fixes

• Miscellaneous linting/checker bugs
• Spelling and path errors in docs
• Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
• Updating dependency to Bokeh 1.40
• Fixed timeline legend bug

November 2019

Overview

Most of the features and fixes this month were done to support the new Entity Explorer series of notebooks
published on Azure-Sentinel-Notebooks repo

New Features

• Added syslog_utils module for common syslog data manipulation

• Added cmd_line module for syslog CMD analysis

• Added ip_utils module for common IP Address operations

• Added domain_utils module for common domain/DNS/URL operations

• Added new TI providers - open_page_rank and tor_exit_nodes

• Added package auto-install function added to utility.py

• nbwidgets: added filtering to all select/list widgets

• In query templates you can now express date parameters (and query defaults) as
  a pos/neg number relative to current date or a KQL-like range (-30d, 15h, etc.)

• wsconfig.py - can create a Connection string directly from wsconfig.

• Added settings as a top-level msticpy attribute.

• TI dataframes derive their column names directly from LookupResult so don't need to keep two lists.

• A number of new queries were added in the following categories:

  • Syslog/linux
  • Office Activity
  • Azure/Azure Active Directory
  • Azure Network Analytics
  • KQL time series queries
  • Hunting bookmarks

• Documentation updates:

  • new documentation page on msticpy configuration - msticpyconfig.yaml
  • Updated Readme.md for new features.

Fixes

• Typo in wsconfig.py - wsconfig throws meaningful error if config values are not found
• fix yaml parsing error in timeseries kql
• Miscellaneous fixes from notebook testing for new Explorer notebooks
• Changed param_extractor to always prefer supplied params over defaults
• tilookup fix - exception thrown if an empty IoCs list sent to it
• geoip - fixed multiple problems with the DF lookup version of the API
• Updated Pandas requirement to 0.25
• Query fixes - miscellaneous
• Added dependencies on cryptography, tqdm, ipwhois