10 May 19:44

ianhelle

857835d

MSTICPy V2.0.0 Pre-Release 1 Pre-release

Pre-release

One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to
use and maintain. Several years of organic growth had seen modules created in places that
seemed like a good idea at the time but did not age well.

The discussion about the V2 structure can be found here #320.

This first pre-release of V2.0 has a small number of new features but is mostly
about testing compatibility. We'll be adding features to future pre-releases in
the next week or two.

**Due to the re-organization, many features are no longer in places where they used
to be imported from! **

We have tried to maintain compatibility with old locations by adding "glue" modules.
These allow import of many modules from their previous locations but will issue a Deprecation warning if loaded from here.
The warning will contain the new location of the module - so you should update your code to point to this new location.

This table gives a quick overview of the V2.0 structure

folder	description
analysis	data analysis functions - timeseries, anomalies, clustering
auth	authentication and secrets management
common	common used utilities and definitions (e.g. exceptions)
config	configuration and settings UI
context	enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup
data	data acquisition/queries/storage/uploaders
datamodel	entities, soc objects
init	package loading and initialization - nbinit, pivot modules
nbwidgets	nb widgets modules
transform	simple data processing - decoding, reformatting, schema change, process tree
vis	visualization modules including browsers

Notable things that have moved:

most things from the sectools folder have migrated to context, transform or analysis
most things from the nbtools folder have migrated to:
- init - (not to be confused with init) - package initialization
- vis - visualization modules
pivot functionality has moved to init

If you are having trouble finding a module, we have added a simple search function

    import msticpy
    msticpy.search("riskiq")

Any matches will be returned in a table with links to the module documentation

Modules matching 'riskiq'

Module	Help
msticpy.context.tiproviders.riskiq	msticpy.context.tiproviders.riskiq

The Help link is to the current release version of the documentation, so will not
map to the correct path on ReleaseTheDocs for the release documentation. You should be able to
map this to the pre-release doc version by adding v2.0.0.rc1 into the URL.

https://msticpy.readthedocs.io/en/v2.0.0rc1/api/msticpy.sectools.tiproviders.html

Other Features/Changes

Pandas accessors have been consolidated into two main ones:

df.mp.func - contains all of the transformation functions like base64 decoding, ioc searching, etc.
df.mp_plot.func - contains all of the visualization accessors (timeline, process tree, etc.)

We've also done a lot of work to clean up warnings, move some remaining Python unittest tests to pytest
format and random other things that you do when you're cleaning house. A fuller but
not especially readable list of changes follows.

Please let us know if you hit any issues with the re-jigged structure
or any other bugs. File them as issues or reach out to [email protected]

List of changes

Co-authored-by: Ian Hellen [email protected]
Co-authored-by: Pete Bryan [email protected]

857835d@Ianhelle/mpconfigedit fix from main 2022 05 22 (#396)* Fix for MpConfigEdit ValueError
Updating Dockerfile source to mcr anaconda
bd96d40@Ianhelle/v2 reorg directories 2 2022 04 12 (#377)* Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver
Some fixes to Kusto common_imports
- now works with Kusto config entry without instance suffix
- can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
- added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
  in the data_families key.
Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.
Reorganized logic for handling parameters and failing on invalid file path in config module.
Moved analysis.data to transform folder
Moved data.context to context
Moved auth and secrets modules to auth folder
Moved nbwidgets to new folder
Updated deprecation warning in glue modules to v2.2
Fix to URLs in README.md
Adding init folder - moved:
- nbinit.py, user_config.py, pivot.py, azure_ml_tools.py to here
- also moved vt_pivot.py and pivot_ti_provider.py to init/pivot_init
Renaming datamodel/pivots to datamodel/pivot
Moved azure_blob_storage.py to data/storage folder
Refactored query_container to data_types.py - to be separate types for queries (query_container.py) and pivots (pivot_container.py)
Moved browsers to vis folder
Updating API docs for moves
Moved all pivot functions to init folder.
Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD)
Added automatic acquisition of globals() in nbinit.py
Fixed a couple of bugs in pivot_pipeline.py
Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization.
Added trap to timeline when supplied with no data.
Fixed incorrect escaping in regex in kql_driver.py
Notebook updates for errors and invalid links.
Added script to run all notebooks for testing
Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x). Other components use the default.
Updated typing rigor for pkg_config::get_http_timeout
Fixed bugs in test test_pkg_config.py and test_code_view.py
Fixed re-auth on query issue in KQL driver
Fixed kql_driver tests
Kusto provider
- now works with Kusto config entry without instance suffix
- can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
- added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
  in the data_famiies key.
Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.
Reorganized logic for handling parameters and failing on invalid file path in config module.
e2e743a@Ianhelle/main mergeback 2022 04 05 (#355)* Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters.
Fixed query_time widget so that you can reset time range from parameter
Added additional unit test for QueryTime setter
Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line.
Changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName.
Updated TimeSpan class so it has more flexible constructor
Added account_id as a parameter for list_aad_signins_by_account query
Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts
to defaults. This can cause errors if the defaults are different to user-specified parameters.
There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings.
aa14784@moved list_sentinel_workspaces to AzureData
f3e82d9@New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352.
ba33c38@Ianhelle/pivot dataprov selfload 2022 03 15 (#343)* Query providers load pivots dynamically when created.
Renamed query paths and changed data_providers so that only env-specific queries are loaded.
Moving ensure_df_datetimes to common/data_utils.py to avoid circular imports
Consolidated data-related pandas accessors into single module.
Fixing circular dependency in iocextract
Fixing linting errors in data_providers.py, azure_resource.py, host.py, process.py, pivot_data_queries.py
Adding default timeout values to httpx calls.
Changing tor_exit_nodes.py Tor provider to defer download of tor list until first lookup
Fixing test for trig...

Contributors

petebryan and ianhelle

Assets 2

10 May 18:06

ianhelle

v1.8.1

b0ba1f4

Hotfix release for MpConfigEdit

What's Changed

MpConfigEdit throws exception when passed file name
MpConfigEdit throws error with invalid file path. by @ianhelle in #395

Full Changelog: v1.8.0...v1.8.1

Contributors

ianhelle

Assets 2

04 May 22:17

ianhelle

v1.8.0

5c4c7f6

Sentinel Search API, Azure Authentication, Settings management, HTTPX timeouts

Summary

There are some feature changes and fixes in this release:

MS Sentinel Search API support in the Sentinel package - allowing you to create, check status and delete automated search jobs.
Authentication updates to
- support wider range of Azure authentication types (incl VSCode and Powershell)
- specify tenantID at auth time (this was a specific issue for Azure Data explorer users)
- lets you use MSAL token caching
Fixes to httpx timeouts (we recently switch from using requests to httpx and have
changed the default timeout to be none (like requests). You can also set this in configuration
and specify while calling a function that makes a network request, e.g. mde_prov.my_query(...params, timeout=30)
MpConfigEdit now behaves better when no existing msticpyconfig.yaml exists - making it easier to create a
config file from scratch.

What's Changed

Ianhelle/mp config edit load fix 2022 03 28 by @ianhelle in #352
Also fixes to multiple widgets for papermill/automation. Can now set QueryTime timespan programmatically.
Fix to Sentinel data provider to prevent re-auth for every query.
Pebryan/2022 3 29 auth updates by @petebryan in #351
- Re-implemented using DefaultCredential
- Added support for MSAL token cache
Bump sphinx from 4.4.0 to 4.5.0 by @dependabot in #350
Fixes for GeoLiteLookup and MpConfigEdit by @ianhelle in #356
- GeoIPLite no longer tries to update DB during initialization - only on first query
- Fixes to MpConfigEdit and MpConfig file for msticpyconfig path handling.
Some fixes to Kusto common_imports by @ianhelle in #358
- Kusto queries now support "database" element in query file. This controls DB usage rather than having to
  encode inside the datafamily. Read more here https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProv-Kusto.html
Changing the pattern for httpx timeout to default to Timeout(None). by @ianhelle in #378
- config based setting for http timeouts
- can be overridden in calls in several providers - OData (MSGraph, MDE) and HTTP-based TI providers.
Add Workflow to Tweet by @petebryan in #369
- testing to automatically tweet status on PR completion
Fixed minor issues by @petebryan in #371
- some issues with incident exploration having incomplete information
Fixing bug in local_data_driver.py if CSV with no TimeGenerated field by @ianhelle in #374
- also added new Sentinel query list_logon_attempts_by_ip
Ianhelle/nb fixes 2022 04 20 by @ianhelle in #379
Added new Sentinel Search Features: by @petebryan in #376
- Sentinel Search API - can create, delete and check status of an automated search
- Also added feature to treat cases where you only have one Sentinel workspace configured - this will be treated as the default
  even if it is not marked as the default.

Full Changelog: v1.7.5...v1.8.0

Contributors

petebryan, ianhelle, and dependabot

Assets 2

23 Mar 03:13

ianhelle

v1.7.5

3587ed4

Splunk async queries, powershell deobfuscator and viewer.

Introduction

An intermediate release while we prepare the ground for v2.0.0 of MSTICPy.
The release includes some small features a several fixes

What's Changed

Added Splunk async provider and unit_tests by @d3vzer0 in #337
The default is now to execute splunk queries asynchronously - the previous default was to run all queries as
one-shot queries so limiting the size of results sets that could be returned. To use the previous behaviour
add the oneshot=True parameter to the query function.

Big thanks to @d3vzer0 (Joey Dreyer) for this (in addition to previous Splunk queries!)
Powershell simple de-obfuscator and code viewer. by @ianhelle in #335

obfus_str = """
IF($PSVerSIONTAble.PSVErSion.MaJoR -gE 3){$43de2=[REF].AsSEmbly.GetTypE(\'System.Management.Automation.Utils\')."GEtFIe`LD"(\'cachedGroupPolicySettings\',\'N\'+\'onPublic,Static\');IF($43DE2)
...
"""
from msticpy.analysis.code_cleanup import format_powershell
from msticpy.vis.code_view import display_html
display_html(
    format_powershell(obfus_str),
    language="powershell"
)

output

if($psversiontable.psversion.major -ge 3)
{
    $43de2=[ref].assembly.gettype('system.management.automation.utils')."getfield"('cachedgrouppolicysettings','nonpublic,static')
    if($43de2)
    {
        $712db=$43de2.getvalue($null)
        if($712db['scriptblocklogging'])

Fixed incorrect use of httpx.codes enum in odata_driver by @ianhelle in #345
Added refresh and delete functions for keyring cached secrets by @ianhelle in #336
Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 by @dependabot in #339

Other Fixes and minor changes

3af4379@
Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339)Bumps [readthedocs-sphinx-ext]

c0fc7fa@

Fixed incorrect use of httpx.codes enum in odata_driver
Added httpx timeouts (30sec connect, 10sec other) to cybereason_driver and http_base and other users of httpx
Splitting keyring into its own module so that we can load without Keyring as a dependency - avoiding
load failure on linux systems without keyring support
Prevent VTLookupV3 from trying to load automatically and throwing exception
Moved IPStack check for API key out of constructor so avoid error if loaded with no configuration.
Change Msticpy exceptions to output plaintext if not running in a notebook (e.g. IPython)
Added -utf16 argument to %b64 magic to allow correct decoding of Windows strings

Contributors

ianhelle, dependabot, and d3vzer0

Assets 2

17 Feb 18:17

ianhelle

v1.7.0

9e5371f

Hackmonth updates: CyberReason, Insights, Splunk, AWS, Kusto

Highlights

We have two new providers - a Threat intel provider for IntSights
and a data/query provider for Cybereason
contributed by Florian Bracq - AXA (@FlorianBracq). The Cybereason docs are here.

A range of new Splunk queries for our Splunk data provider for Alert and Authentication data
contributed by Joey Dreijer (@d3vzer0)

Two cool sample notebooks:

Azure data explorer/Kusto notebook from Liam Kirton @liamkirton
AWS S3 HoneyBucket Log analysis notebook by Ashwin Patil @ashwin-patil

Complete replacement of requests with the async-capable httpx library by GrantV @grantv9

New Sentinel Security alert entity that automatically extracts related entities from
a MS Sentinel alert.

Overhaul of some of our documentation - including automated build/collection of API docs and
data provider query list (so these should now be up-to-date) and re-organization/re-write of the
Data provider docs.

Reorganization of setup and tool settings using setup.cfg and pyproject.toml.

We also added a workaround (the IPython version dependency) to allow msticpy to be installed
on Python 3.6 (even though we don't officially support it).

What's Changed

Add IntSights support by @FlorianBracq in #276
Setup.cfg and Pyproject.toml by @ianhelle in #287
Ianhelle/nbwidgets refactor 2022 01 25 by @ianhelle in #299
Add tenant_id parameter back by @liamkirton in #295
(fix): typo in README by @zscholl in #294
Add alert-oriented queries for Splunk using datamodels by @d3vzer0 in #297
Add authentication-oriented queries for Splunk using datamodels by @d3vzer0 in #298
Fix QueryTime unit change by @kubajir in #303
Update regular expression to extract parameters from source by @FlorianBracq in #291
Ianhelle/data docs rewrite 2022 01 27 by @ianhelle in #304
SentinelAlert entity creation by @petebryan in #293
Replaced requests with httpx 0.18 by @grantv9 in #285
Updating httpx requirements to work with out requirements by @petebryan in #312
Change to Querylist doc and py script to avoid phantom changes by @ianhelle in #310
Cybereason by @FlorianBracq in #306
AWS S3 HoneyBucket Log analysis notebook by @ashwin-patil in #311
Kusto Driver Example Notebooks by @liamkirton in #308
Bump sphinx from 4.3.2 to 4.4.0 by @dependabot in #283
doc updates by @ashwin-patil in #316
Adding dev container files for VSCode by @ashwin-patil in #321
Bump respx from 0.17.1 to 0.19.2 by @dependabot in #314
Updated Cybereason docs to fit pattern by @petebryan in #324
Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 by @ianhelle in #317
Changing magics creation so that they don't get created if not in ipython by @ianhelle in #332

New Contributors

@FlorianBracq made their first contribution in #276
@zscholl made their first contribution in #294
@d3vzer0 made their first contribution in #297
@kubajir made their first contribution in #303
@grantv9 made their first contribution in #285

Full Changelog: v1.6.1...v1.7.0

Contributors

liamkirton, zscholl, and 8 other contributors

Assets 2

19 Jan 18:58

ianhelle

v1.6.1

f1212ec

v1.6.1 - Sentinel APIs, Folium, VirusTotal, ProcessTree, and more

In case you're wondering - we never released 1.6.0 to PyPI so this is the official 1.6.0 release.

Summary

Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class (#227)

The FoliumMap class now supports layering and subgrouping. This was contributed by @tj-senserva
(from Senserva)

Adding New Features to Sentinel APIs (#280)

Sentinel APIs has had signficant rework and expansion to cover new public APIs. These include Analytics and Watchlists as
well as Incidents and Bookmarks from earlier releases. Each API now includes modification
operations such as Create, Delete, Update (not all types support all types of modification).

Note: the module and class names drop the "Azure" prefix following the
renaming of Azure Sentinel to Microsoft Sentinel

VirusTotal FileBehavior, FileBrowser and Pivot functions (#260)

The VirusTotal V3 module has been expanded to add additional operations and data visualization:

VT FileBehavior class lets you download and view file detonation data.
VT ObjectBrowser lets you download and view basic attributes of a file object
Pivot functions - these expose the VT3 Relationship APIs allowing you to pivot between
related IPs, URLs, Files and Domains.

Added instance handling to Dataproviders (#218)

Several providers now support instances - e.g. Splunk, Sumologic can support multiple
configuration entries in msticpyconfig.yaml with the use of an instance suffix, e.g.
Spunk-Env1, Splunk-Evn2.
This change also makes it possible to combine these instances into a single aggregate
data provider and run the same query across the combined instances. This also
works for multiple Microsoft Sentinel workspaces.

Process tree visualization updates to make schema-agnostic (#256)

The process tree visualization code previous had several dependencies on Sentinel or MDE
schema specifics. The code should now accept process event data sets from many sources.

Adding Sysmon ProcSchema (#267) - A schema mapping for Sysmon data was contribute by @nbareil

Build pipeline fixes for external forks (#270)

Previously builds triggered from external forks would fail for a variety of reasons
(no access to pipeline secrets, MS-Internal-only build actions). This is now fixed so that
external contributors should expect a clean build.

Added default caching option for mordor driver (#257)

The OTRF Security Datasets driver and browser now cache security data set and MITRE ATT&CK
data for quicker startup. By default, the cache files are stored in $HOME/.msticpy/mordor
or the location specified in the Dataproviders/Mordor section of the msticpyconfig.

What's Changed

Ianhelle/proc tree fixes 2022 01 03 by @ianhelle in #256
Bump sphinx from 4.2.0 to 4.3.2 by @dependabot in #241
Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class by @tj-senserva in #227
20220105 hackathon blob storage conn string by @lucky-luk3 in #262
Added instance handling to Dataproviders by @ianhelle in #218
Added default caching option for mordor driver by @ianhelle in #257
Ianhelle/virustotalv3 additions 2021 10 22 by @ianhelle in #260
Initial attempt at masking failing tests for external forks. by @ianhelle in #270
Adding Sysmon Process Create's ProcSchema by @nbareil in #267
Removing validate configuration for external forks by @ianhelle in #273
Update AzureData to use correct Azure subscription management API by @liamkirton in #269
Added details to README on upgrading msticpy by @danielc-evans in #274
Updated IPython elements and requirements by @petebryan in #278
Fixes and improvements to KqlDriver and KustoDriver Azure auth. by @liamkirton in #277
Miscellaneous fixes for VT, Auth and other items by @ianhelle in #281
Adding New Features to Sentinel APIs by @petebryan in #280
Correcting error introduced in nbinit refactoring by @ianhelle in #286

New Contributors

@dependabot made their first contribution in #241
@tj-senserva made their first contribution in #227
@lucky-luk3 made their first contribution in #262
@nbareil made their first contribution in #267
@liamkirton made their first contribution in #269
@danielc-evans made their first contribution in #274

Additional feature details

7fa1598@

Add ability to filter queries by substring in list_queries in data_providers.py
Adding coordinates property to GeoLocation entity geo_location.py
Simplify verbose option and refactoring complex method in init_notebook
c0a32a0@Merge pull request #277 from liamkirton/liamkirton/kql-kusto-driver-auth
Fixes and improvements to KqlDriver and KustoDriver Azure auth.
ae6bcd7@Updated IPython elements and requirements (#278)* Updated IPython elements and requirements
79910e2@Merge pull request #274 from danielc-evans/updated-readmeAdded details to README on upgrading msticpy
7038e28@Changed type hints in enum_parse in utility.py
Added autoload of Pivot and vt_pivot (if VT config detected)
Moved some common methods to IPyDisplayMixin class in nbwidgets.py
Added extra parameter for VT Private API definition to mpconfig_defaults.yaml. Also changed defaults for some providers to Primary=False
Updates to test_mp_config_controls.py to deal with new VT parameter
Add fixed vtobject_browser.py + unit test in test_vtlookupv3.py
8308849@Added the ability to connect to a blob storage using the connection string. Container name parameter changed to optional.

Additional fix details

7fa1598@Miscellaneous fixes for VT, Auth and other items (#281)* Updates to Timeseries analysis and plotting modules

Avoiding exception when cmdline is NaN or non-string in process_tree.py
Removing "Authenticating to Azure" output since it fires on every query.
Removing credential caching in azure_auth_core.py

f730ba2@Moved geoip GeoLiteLookup to check for and update DB before first query (rather than in init)
Added full copy of GeoLite2-City.mmdb to test data
Linting fixes in mordor_driver.py and test-test_mordor_driver.py
Cherry picked commit for python-package and azure-pipelines yamls

569eb75@Forcing install of pandas>=1.30 in pytest block
857b08e@Saved mde_proc_pub.pkl as pickle protocol 4 to work in Python 3.6
70b5c55@Adding updated notice file.
4599d6e@Fixed use of hard-code global endpoint in azure_blob_storage.py

cad8256@Added create incident and create rule features
70b5c55@Adding upgrade to pandas to so that it uses latest version and avoids pickle compatability issue with test data pickled with pandas > 1.2.5Also added higher version constraint to requirements-dev.txt and conda-reqs-dev.txt
Adding updated notice file.

4599d6e@Added instance handling to Dataproviders (#218)

Fixed use of hard-code global endpoint in azure_blob_storage.py
Adding better handling of response data in kql_driver.py
Removing default start and end parameters from queries. We should rely on the builtin querytimes control
Adding issue tracker URL and additional classifiers to setup.py
Updating import_analyzer.py to include resolved and unresolved paths for determining whether a package belongs to the std library. (these paths may be different if some of them are links)
0b9c81a@Bump sphinx from 4.2.0 to 4.3.2 (#241)

Full Changelog: v.1.5.2...v1.6.1

Contributors

nbareil, liamkirton, and 6 other contributors

Assets 2

24 Dec 00:23

ianhelle

v.1.5.2

8c0c602

Config correction for Splunk driver

Hotfix release to correct incorrect capitalization reading values for Splunk settings

Assets 2

22 Dec 22:56

ianhelle

v1.5.1

8406e87

MDE ProcTree fixes, Azure ML notebooks SSO

Summary

This release includes two minor changes:

Fixed schema for Microsoft Defender Device Process events - this should allow it to work with the Process Tree visualization
Single-sign-on in Azure ML notebooks using MSI

Fixes and New Features

MDE Process Tree

The schema originally deployed for generating process tree data from Microsoft Defender for Endpoint data was incorrect - we were using an internal schema that did not match the data retrieved via the Defender APIs.
This has now been fixed along with some related items:

DateTime fields returned in the Defender data are automatically converted to pandas Timestamps (previously they were left as strings)
The process tree and other visualizations will automatically convert required timestamp columns to datetime format - this is useful where data is retrieved from an API or file where the required datetime columns are stored as date strings

Single Sign-On for notebooks in Azure ML (in Microsoft Sentinel)

When you sign on an Azure ML workspace, Managed Identity (MSI) credentials are automatically enabled on the Azure Computes used to run the notebooks. The MSI credentials use the user principal of the user that has logged on (not the machine identity). This allows processes on the Computes to use these credentials to authorize to other Azure resources.
The Log Analytics team made some changes to allow the Managed Identity credentials from Azure ML to authorize to Log Analytics/Microsoft Sentinel data API.

With this release, connecting to Microsoft Sentinel will automatically try to use the MSI credentials. Existing authentication flow options such as Azure CLI and Device Code authentication are still available.

To use MSI-based SSO:

qry_prov = QueryProvider("AzureSentinel")   # we will change this to MicrosoftSentinel soon!
qry_prov.connect(WorkspaceConfig())

To force specific authentication flavors, you can set your preferences explicitly in the msticpconfig.yaml. You can also do this at the command line.

To specify an explicit set of authentication types to use (they are tried in the order specified):

qry_prov.connect(WorkspaceConfig(), mp_az_auth=["cli", "interactive"])

To skip integrated Azure authentication altogether and use interactive Device Code authentication:

qry_prov.connect(WorkspaceConfig(), mp_az_auth=False)
# or
qry_prov.connect(WorkspaceConfig(), mp_az_auth=["interactive"])

Detailed changes

4fdf286@Ianhelle/mde proctree fixes 2021 12 16 (#239)* Changes:

Removed checking of DataFamily in data_query_reader.py
Changed default behavior of kql_driver.py to use Azure authentication
Change security_base.py to use entities.OSFamily rather than query_defns.DataFamily
Added ensure_df_datetimes function to auto-convert specified columns from string to datetime
Added unit test test_query_defns.py
Added ensure_df_datetime to timeline.py
Added ensure_df_datetime to timeline_duration.py
Added ensure_df_datetime to proc_tree_build_mde.py
Added ensure_df_datetime to proc_tree_build_winlx.py
Changed mpconfig_defaults.yaml to include "msi"
Switching "Microsoft Sentinel" to "Azure Sentinel" in UI/messages.
Updated black version in .pre-commit-config.yaml
Using returned schema in mdatp_driver.py to auto-format datetimes to pandas timestamp
Returning full response from query_with_results in odata_driver.py
Some fixes in convert_mde_schema_to_internal - field naming and using tz-aware Unknown time value
Moved the proc tree schema code from proc_tree_builder.py to proc_tree_schema.py
Add mocked az_connect to test_kql_driver.py tests - since the change to defaulting to AZ auth in the kql_driver means that tests fail.
Added test case for MDE public data - mde_proc_pub.pkl
Adding tooltips to settings editor buttons
also changing button text for simple setting editor to "Update" (from "Save")
changing order of checks in base64unpack to let you specify UTF-16 decoding
updating version to 1.5.1

Assets 2

29 Nov 21:16

ianhelle

v1.5.0

7e62378

RiskIQ, Sentinel Incident Explorer, Kusto and MS Defender

Release Summary

There are several cool new features in this release:

Microsoft Sentinel Incident exploration
RiskIQ Threat Intellligence provider and pivot functions
Microsoft Defender 365 API/Data provider support
Preview Azure Data Explorer/Kusto data provider support
Microsoft graph provider now supports sovereign clouds
ProcessTree visualization support for Microsoft Defender data in Microsoft Sentinel

Microsoft Sentinel Incident Exploration

0037dd6@Merge pull request #211 from microsoft/pebryan/2021-8-18_IncidentTriage

Provides new features to support Incident Triage scenarios (as included in the new Microsoft Sentinel Incident Triage notebook Azure/Azure-Sentinel-Notebooks#120).

Entity Graph Visualization

This new visualization allows for the graphing of an investigation including any alerts, incidents or entities associated with that investigation.
The graph can be created, updated and visualized either as a graph or with a graph and a timeline of the graph event times.

from msticpy.vis.entity_graph_tools import EntityGraph
graph = EntityGraph(incident)
graph.plot()

The Entity Graph can also be created from a dataframe of events, either by passing in the dataframe or as a Pandas accessor.

df.mp_plot.incident_graph(timeline=True)

Microsoft Sentinel Incident API support

RiskIQ PassiveTotal TI provider and Pivot provider

9353d9f@RiskIQ threat intel provider & pivots (#175)

Delivers a new "RiskIQ" threat intel provider, enabling access to RiskIQ Summary Card and Reputation
datasets for IPs and hostnames with the same syntax and response format used with other TI Providers.

ti_lookup = TILookup()
lookup_result = ti_lookup.lookup_ioc(observable="137.184.153.130", providers="RiskIQ")
ti_lookup.result_to_df(lookup_result).T

Introduces a new mechanism for registering MSTICPy Pivot Functions provided by threat intel providers,
then uses that code to register capabilities for nearly all RiskIQ API endpoints on hostnames, FQDN and
IPAddress entities. These pivot functions enable easy integration of RiskIQ-provided intelligence in
existing pivot pipelines, and makes it easy to build new pipelines leveraging unique RiskIQ datasets.

(
    Dns.RiskIQ.resolutions("jquery.su")
    .query("recordtype=='A'")
    .mp_pivot.run(Dns.RiskIQ.reputation, column='resolve', join='left')
    .mp_pivot.run(IpAddress.RiskIQ.resolutions, column='resolve', join='left')
)

Preview support for Kusto (Azure Data Explorer)

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225)

This release contains a preview (i.e. not fully baked) of support for Kusto cluster queryingand Microsoft Defender 365 data providers.
The Kusto/Azure Data Explorer support is based on the existing Microsoft Sentinel provider.

kusto_provider = QueryProvider("Kusto")
kusto_provider.connect(
    "azure_data-Explorer://code;cluster='my_cluster';database='my_db';alias='my_alias'"
)

Some more documentation to follow about configuration in msticpyconfig and
how to build template queries for Kusto/Azure Data explorer.

Support for Microsoft 365 Defender data providers

This is slight expansion of our current support for MS Defender for endpoint. Using the new "M365D" provider name
will cause the provider to use the new combined Microsoft Defender APIs, giving you consolidated access
to data from multiple Defender services.

To create a provider for Microsoft 365 Defender:

m365d_provider = QueryProvider("M365D")
m365d_provider.connect()

All template queries for that were available for MDE are also loaded for M365D.

Using Microsoft Defender for Endpoint is still supported. To use only
the Microsoft Defender for Endpoint APIs use "MDE".

You can specify your connection parameters in msticpconfig.yaml

DataProviders:
  # ...
  MicrosoftDefender:
    Args:
      ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
      ClientSecret:
        KeyVault:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53

This configuration expects to find the client secret in your configured Key Vault with the name
"DataProviders-MicrosoftDefender-Args-ClientSecret" - the msticpy config editor will create this for you.
You can store the secret in your config file but, obviously, we would not recommend this.

You can also specify configuration/secret values in the call to m365d_provider.connect().
For more information see the Defender provider MSTICPy documentation

Other improvements

The Graph provider now supports sovereign clouds as well as the Azure global cloud.
You can configure settings for MS Defender and MS Graph via the MSTICPy settings editor.
If you have configured settings for these you can now connect with a simplified
syntax, with no parameters required.

graph_prov = QueryProvider("SecurityGraph")
graph_prov .connect()

The schema for the configuration is the same format as the Defender format described above.

DataProviders:
  # ...
  MicrosoftGraph:
    Args:
      ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
      ClientSecret:
        KeyVault:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53

Process Tree visualization supports MS Sentinel schema for MDE process data

The schema of the DeviceProcessEvents table in Microsoft Sentinel is different
from the data queried directly from the MS Defender advance hunting API. This
update allows you to query MS Defender process data from Azure Sentinel and
display in the Process Tree visualization.

New CI Pipelines using GitHub Actions and Azure Pipelines

The old Azure DevOps pipeline has been decommissioned and replaced by two new ones - GitHub actions and an improved version of the Azure DevOps pipeline.

The main advantage of the GitHub actions pipeline is that the results
of tests and code checks are visible to external contributors.

However, there are some component governance and security checks that
we are required to run by Microsoft but which are not currently available
in GitHub actions. We've opted to run both in parallel.

The new Azure DevOps pipeline is more sophisticated and comprehensive than
the previous one. It allows us to run builds and tests on Windows, Linux and
Mac platforms and to cover Python versions 3.6 and 3.8.
(Note: we will likely switch to 3.8 and 3.10 in the near future)

Other changes and fixes

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225)

Fixing issue in odata_driver.py where api_root would keep getting suffix appended if you connected multiple times.
Adding mutli-cloud support for MS Graph API in security_graph_driver.py
Fixing syntax errors in graph_alerts.yaml
Updating tests for test_drivers and test_kql_driver
Fixed an issue in kql_driver with Kusto returning a different status code format
1c2c1be@Disabling IPstack tests because of restricted API allowance (#224)*
Disabling IPstack tests because of restricted API allowance
Fixing incorrect assignment in test_geoip.py
56fcb81@Updating readme badge to reflect new build pipeline
cdff1ff@Update sphinx requirements in azure-pipelines.yml for Azure Pipelines
ce8cbe1@Update README.md to include downloads
0cb20fd@Updated way of getting instance and domain name for azure_ml_tools.py (#217)
- Updated checking for version of msticpy using setuptools/pkg_resources in azure_ml_tools.py
- Added warning if running with stale import of newly updated msticpy.
8043207@Ianhelle/timeline args check 2020 09 21 (#219)* Adding some error checking of column names to timeline_values and timeline_duration
a7498a3@Ianhelle/azure pipelines yaml 2021 09 09 (#201)*
- Pylint and mypy fixes in:
  - anomalous_sequence/model.py
  - anomalous_sequence/utils/cmds_params_values.py
  - common/pkg_config.py
  - common/wsconfig.py
  - config/comp_edit.py
  - config/mp_config_file.py
  - data/data_providers.py
  - data/data_query_reader.py
  - data/query_store.py
  - data/sql_to_kql.py
  - datamodel/entities/host.py
  - datamodel/entities/ip_address.py
  - datamodel/pivot_register_reader.py
  - nbtools/morph_charts.py
  - nbtools/nbdisplay.py
  - nbtools/security_alert.py
  - nbtools/security_base.py
  - sectools/auditdextract.py
  - sectools/base64unpack.py
  - sectools/cmd_line.cmd_line.py
Adding pylint --disable=duplicate-code
Updating pylint requirements in requirements-dev.txt and .pre-commit-config.yaml
Add [with_pyroma] to prospector install
Duplicating @noqa MC0001 suppressions since prospector/McCabe now seem to ignore decorators an locate the offending line as the "def" line rather than the decorator line.
Fixing MyPy-discovered error in nbwidgets.AlertSelector
9f122ee@refactored code in test_geoip
907e486@Remove un-needed code in test_geoip
8ae01f7@Updated test_geoip to fix failure case
e5aadac@Added missing parameters to list_all_signins_geo

Contributors

noqa

Assets 2

17 Nov 23:19

ianhelle

v.1.5.0pre3

4c41364

Pre-Release: RiskIQ, Sentinel Incident Explorer, Kusto and MS Defender Pre-release

Pre-release

Release Summary

There are several cool new features in this release:

Microsoft Sentinel Incident exploration
RiskIQ Threat Intellligence provider and pivot functions
Microsoft Defender 365 API/Data provider support
Preview Azure Data Explorer/Kusto data provider support
Microsoft graph provider now supports sovereign clouds
ProcessTree visualization support for Microsoft Defender data in Microsoft Sentinel

Microsoft Sentinel Incident Exploration

0037dd6@Merge pull request #211 from microsoft/pebryan/2021-8-18_IncidentTriage

Provides new features to support Incident Triage scenarios (as included in the new Microsoft Sentinel Incident Triage notebook Azure/Azure-Sentinel-Notebooks#120).

Entity Graph Visualization

from msticpy.vis.entity_graph_tools import EntityGraph
graph = EntityGraph(incident)
graph.plot()

The Entity Graph can also be created from a dataframe of events, either by passing in the dataframe or as a Pandas accessor.

df.mp_plot.incident_graph(timeline=True)

Microsoft Sentinel Incident API support

RiskIQ PassiveTotal TI provider and Pivot provider

9353d9f@RiskIQ threat intel provider & pivots (#175)

ti_lookup = TILookup()
lookup_result = ti_lookup.lookup_ioc(observable="137.184.153.130", providers="RiskIQ")
ti_lookup.result_to_df(lookup_result).T

(
    Dns.RiskIQ.resolutions("jquery.su")
    .query("recordtype=='A'")
    .mp_pivot.run(Dns.RiskIQ.reputation, column='resolve', join='left')
    .mp_pivot.run(IpAddress.RiskIQ.resolutions, column='resolve', join='left')
)

Preview support for Kusto (Azure Data Explorer)

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225)

kusto_provider = QueryProvider("Kusto")
kusto_provider.connect(
    "azure_data-Explorer://code;cluster='my_cluster';database='my_db';alias='my_alias'"
)

Some more documentation to follow about configuration in msticpyconfig and
how to build template queries for Kusto/Azure Data explorer.

Support for Microsoft 365 Defender data providers

To create a provider for Microsoft 365 Defender:

m365d_provider = QueryProvider("M365D")
m365d_provider.connect()

All template queries for that were available for MDE are also loaded for M365D.

Using Microsoft Defender for Endpoint is still supported. To use only
the Microsoft Defender for Endpoint APIs use "MDE".

You can specify your connection parameters in msticpconfig.yaml

DataProviders:
  # ...
  MicrosoftDefender:
    Args:
      ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
      ClientSecret:
        KeyVault:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53

You can also specify configuration/secret values in the call to m365d_provider.connect().
For more information see the Defender provider MSTICPy documentation

Other improvements

graph_prov = QueryProvider("SecurityGraph")
graph_prov .connect()

The schema for the configuration is the same format as the Defender format described above.

DataProviders:
  # ...
  MicrosoftGraph:
    Args:
      ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
      ClientSecret:
        KeyVault:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53

Process Tree visualization supports MS Sentinel schema for MDE process data

New CI Pipelines using GitHub Actions and Azure Pipelines

The old Azure DevOps pipeline has been decommissioned and replaced by two new ones - GitHub actions and an improved version of the Azure DevOps pipeline.

The main advantage of the GitHub actions pipeline is that the results
of tests and code checks are visible to external contributors.

However, there are some component governance and security checks that
we are required to run by Microsoft but which are not currently available
in GitHub actions. We've opted to run both in parallel.

Other changes and fixes

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225)

Fixing issue in odata_driver.py where api_root would keep getting suffix appended if you connected multiple times.
Adding mutli-cloud support for MS Graph API in security_graph_driver.py
Fixing syntax errors in graph_alerts.yaml
Updating tests for test_drivers and test_kql_driver
Fixed an issue in kql_driver with Kusto returning a different status code format
1c2c1be@Disabling IPstack tests because of restricted API allowance (#224)*
Disabling IPstack tests because of restricted API allowance
Fixing incorrect assignment in test_geoip.py
56fcb81@Updating readme badge to reflect new build pipeline
cdff1ff@Update sphinx requirements in azure-pipelines.yml for Azure Pipelines
ce8cbe1@Update README.md to include downloads
0cb20fd@Updated way of getting instance and domain name for azure_ml_tools.py (#217)
- Updated checking for version of msticpy using setuptools/pkg_resources in azure_ml_tools.py
- Added warning if running with stale import of newly updated msticpy.
8043207@Ianhelle/timeline args check 2020 09 21 (#219)* Adding some error checking of column names to timeline_values and timeline_duration
a7498a3@Ianhelle/azure pipelines yaml 2021 09 09 (#201)*
- Pylint and mypy fixes in:
  - anomalous_sequence/model.py
  - anomalous_sequence/utils/cmds_params_values.py
  - common/pkg_config.py
  - common/wsconfig.py
  - config/comp_edit.py
  - config/mp_config_file.py
  - data/data_providers.py
  - data/data_query_reader.py
  - data/query_store.py
  - data/sql_to_kql.py
  - datamodel/entities/host.py
  - datamodel/entities/ip_address.py
  - datamodel/pivot_register_reader.py
  - nbtools/morph_charts.py
  - nbtools/nbdisplay.py
  - nbtools/security_alert.py
  - nbtools/security_base.py
  - sectools/auditdextract.py
  - sectools/base64unpack.py
  - sectools/cmd_line.cmd_line.py
Adding pylint --disable=duplicate-code
Updating pylint requirements in requirements-dev.txt and .pre-commit-config.yaml
Add [with_pyroma] to prospector install
Duplicating @noqa MC0001 suppressions since prospector/McCabe now seem to ignore decorators an locate the offending line as the "def" line rather than the decorator line.
Fixing MyPy-discovered error in nbwidgets.AlertSelector
9f122ee@refactored code in test_geoip
907e486@Remove un-needed code in test_geoip
8ae01f7@Updated test_geoip to fix failure case
e5aadac@Added missing parameters to list_all_signins_geo

Contributors

noqa

Assets 2

Releases: microsoft/msticpy

MSTICPy V2.0.0 Pre-Release 1

Modules matching 'riskiq'

Other Features/Changes

List of changes

Contributors

Hotfix release for MpConfigEdit

What's Changed

Contributors

Sentinel Search API, Azure Authentication, Settings management, HTTPX timeouts

Summary

What's Changed

Contributors

Splunk async queries, powershell deobfuscator and viewer.

Introduction

What's Changed

Other Fixes and minor changes

Contributors

Hackmonth updates: CyberReason, Insights, Splunk, AWS, Kusto

Highlights

What's Changed

New Contributors

Contributors

v1.6.1 - Sentinel APIs, Folium, VirusTotal, ProcessTree, and more

Summary

Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class (#227)

Adding New Features to Sentinel APIs (#280)

VirusTotal FileBehavior, FileBrowser and Pivot functions (#260)

Added instance handling to Dataproviders (#218)

Process tree visualization updates to make schema-agnostic (#256)

Build pipeline fixes for external forks (#270)

Added default caching option for mordor driver (#257)

What's Changed

New Contributors

Additional feature details

Additional fix details

Contributors

Config correction for Splunk driver

MDE ProcTree fixes, Azure ML notebooks SSO

Summary

Fixes and New Features

MDE Process Tree

Single Sign-On for notebooks in Azure ML (in Microsoft Sentinel)

Detailed changes

RiskIQ, Sentinel Incident Explorer, Kusto and MS Defender

Release Summary

Microsoft Sentinel Incident Exploration

Entity Graph Visualization

Microsoft Sentinel Incident API support

RiskIQ PassiveTotal TI provider and Pivot provider

Preview support for Kusto (Azure Data Explorer)

Support for Microsoft 365 Defender data providers

Other improvements

Process Tree visualization supports MS Sentinel schema for MDE process data

New CI Pipelines using GitHub Actions and Azure Pipelines

Other changes and fixes

Contributors

Pre-Release: RiskIQ, Sentinel Incident Explorer, Kusto and MS Defender

Release Summary

Microsoft Sentinel Incident Exploration

Entity Graph Visualization

Microsoft Sentinel Incident API support

RiskIQ PassiveTotal TI provider and Pivot provider

Preview support for Kusto (Azure Data Explorer)

Support for Microsoft 365 Defender data providers

Other improvements

Process Tree visualization supports MS Sentinel schema for MDE process data

New CI Pipelines using GitHub Actions and Azure Pipelines

Other changes and fixes

Contributors