Skip to content

Releases: microsoft/msticpy

DataProviders, QueryEditor, CrowdSec and AbuseIPDB TIProviders

08 Aug 18:50
@ianhelle ianhelle
v2.7.0.pre1
1c6a689
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
DataProviders, QueryEditor, CrowdSec and AbuseIPDB TIProviders Pre-release
Pre-release

Preview release of 2.7.0

More detailed release notes in the full release.

Main Changes

Two new TI Providers:

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

These were introduced in v2.5.0 but are now the default drivers for these providers.

Query Editor

ipywidgets based query template editor - this is somewhat provisional so please be sure to test and
report bugs.

Updates to Authentication - esp for the AzureData and MicrosoftSentinel API modules

  • You can now authenticate by supplying an AzureCredential as a credential parameter
  • The connect methods for these support cloud parameter to specify different sovreign clouds
  • The init and connect methods are instrumented with logging to help debug issues:
import msticpy as mp
from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.set_logging_level("INFO")
mssentinel = MicrosoftSentinel()
mssentinel.connect()

Other items

What's Changed

New Contributors

Full Changelog: v2.6.0...v2.7.0.pre1

Contributors

ianhelle, rrevuelta, and 4 other contributors
Assets 2
Loading

v2.6.0 Parallel Queries, Velociraptor data

25 Jul 01:04
@ianhelle ianhelle
v2.6.0
a8a2de4
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.6.0 Parallel Queries, Velociraptor data

The three big changes in this release are:

  • Executing MS Sentinel and Kusto queries in parallel across multiple instance
  • Threaded (parallel) execution of time-split queries
  • Addition of data provider to query local (exported) Velociraptor logs

Many thanks to @d3vzer0 for inspiration and early work on the threaded query feature.
Many thanks @juju4 for inspiration and work on the Velociraptor support.

Support for running a query across multiple connections (with optional threaded operation)

It is common for data services to be spread across multiple tenants or workloads. E.g., multiple Sentinel workspaces,
Microsoft Defender subscriptions or Splunk instances. You can use the MSTICPy QueryProvider to run a query across multiple connections and return the results in a single DataFrame.

To create a multi-instance provider:

  • Create an instance of a QueryProvider for your data source and execute the connect() method to connect to the first instance of your data service.
  • Then use the add_connection() method. This takes the same parameters as the connect() method (the parameters for this method vary by data provider) to add additional instance connections.

add_connection() also supports an alias parameter to allow you to refer to the connection by a friendly name.

    qry_prov = QueryProvider("MSSentinel")
    qry_prov.connect(workspace="Workspace1")
    qry_prov.add_connection(workspace="Workspace2, alias="Workspace2")
    qry_prov.list_connections()

When you now run a query for this provider, the query will be run on all of the connections and the results will be returned as a single dataframe.

    test_query = '''
        SecurityAlert
        | take 5
        '''

    query_test = qry_prov.exec_query(query=test_query)
    query_test.head()

Some of the MSTICPy drivers support asynchronous execution of queries against multiple instances, so that the time taken to run the query is much reduced compared to running the queries sequentially. Drivers that support asynchronous queries will use this automatically. The initial set of multi-threaded drivers are:

  • MSSentinel_New (the new version of the MSSentinel driver)
  • Kusto_New (the new version of the Kusto/Azure Data Explorer driver)

By default, the queries will use at most 4 concurrent threads. You can override this by initializing the QueryProvider with the
max_threads parameter to set it to the number of threads you want. Although you should be cautious
about using too many simultaneous connections due to the potential impact on the cluster performance.

    qry_prov = QueryProvider("MSSentinel", max_threads=10)

Multi-threaded support for split/shared queries

MSTICPy has supported splitting large queries by time-slice for a while. This is documented here Splitting a Query into time chunks. With this release, we've added asynchronous support for this (if the driver supports threaded/async operation) so that multiple chunks of the query will run in parallel.

    qry_prov.SecurityAlert.list_alerts(start=start, end=end, split_by="1d")

Use the parameter split_query_by or split_by to specify a time range (the time unit uses the same syntax as pandas time intervals - e.g. "1D", "4h", etc. - the the pandas documentation for more details on this).

In this release sharding is also supported for ad hoc queries as long as you add "start" and "end" parameters to the query (this is still experimental, so let us know if you have issues with this).

Velociraptor Local Data Provider

The Velociraptor data provider can read Velociraptor log files and provide convenient query functions for each data set in the output logs.

The provider can read files from one or more hosts, stored in in separate folders. The files are read, converted to pandas DataFrames and grouped by table/event. Multiple log files of the same type (when reading in data from multiple hosts) are concatenated into a single DataFrame.

To use the Velociraptor provider, you need to create an QueryProvider instance, passing the string "Velociraptor" (or "VelociraptorLogs") as the data_environment parameter. You also need to add the data_paths parameter to specify specific folders that you want to search for log file (although you can set these paths in msticpyconfig.yaml, if you do this frequently).

You can specify multiple folders to have the logs from different hosts.

    qry_prov = mp.QueryProvider("VelociraptorLogs", data_paths=["~/my_logs"])

Calling the connect method triggers the provider to read the locations of the
log files (although the contents are not read until a query function is run).

    qry_prov.connect()


## Listing Velociraptor tables

```python3
    qry_prov.list_queries()
    ['velociraptor.Custom_Windows_NetBIOS',
    'velociraptor.Custom_Windows_Patches',
    'velociraptor.Custom_Windows_Sysinternals_PSInfo',
    'velociraptor.Custom_Windows_Sysinternals_PSLoggedOn',
   ....

Each query returns the table of data types retrieved from the logs.

    qry_prov.vc_prov.velociraptor.Windows_Forensics_ProcessInfo()
Name PebBaseAddress Pid ImagePathName CommandLine CurrentDirectory Env
LogonUI.exe 0x95bd3d2000 804 C:\Windows\system32\LogonUI.exe "LogonUI.exe" /flags:0x2 /state0:0xa3b92855 /state1:0x41c64e6d C:\Windows\system32\ {'ALLUSERSP
dwm.exe 0x6cf4351000 848 C:\Windows\system32\dwm.exe "dwm.exe" C:\Windows\system32\ {'ALLUSERSP
svchost.exe 0x6cd64d000 872 C:\Windows\System32\svchost.exe C:\Windows\System32\svchost.exe -k termsvcs C:\Windows\system32\ {'ALLUSERSP
svchost.exe 0x7d18e99000 912 C:\Windows\System32\svchost.exe C:\Windows\System32\svchost.exe -k LocalServiceNetworkRestricted C:\Windows\system32\ {'ALLUSERSP
svchost.exe 0x5c762eb000 920 C:\Windows\system32\svchost.exe C:\Windows\system32\svchost.exe -k LocalService C:\Windows\system32\ {'ALLUSERSP

What's Changed

Full Changelog: v2.5.3...v2.6.0

Contributors

juju4, ianhelle, and 2 other contributors
Assets 2
Loading

v2.5.3: ipwidgets, Sentinel and Kusto driver fixes

21 Jun 23:17
@ianhelle ianhelle
v2.5.3
ab444ca
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.5.3: ipwidgets, Sentinel and Kusto driver fixes

Minor release addressing the following:

  • Azure-monitor-query release 1.2.0 changed the format of the endpoint URLs that it accepts. Fixed the azure_kusto driver (currently invoked with the "Kusto_New" data environment) so that it will provide the correct format for 1.2.0+ and pre 1.2.0 versions
  • Bug in the kql_driver (MS Sentinel) was causing the kusto_driver to fail when querying. The latter is a subclass of the former and was failing due to an attribute that was defined in the parent (kql_driver) but not in the child (kusto_driver). This affected the older (current) Kusto driver version and does not affect the new azure_kusto ("Kusto_New") driver.
  • Updated requirements to allow ipywidgets 8.x to install by default (this is now supported by vs code
  • Updated documentation for the new Sentinel and Kusto drivers to add instructions for manually installing the required SDK components (azure-monitor-query and azure-kusto-data)

What's Changed

  • Azure monitor endpoint URL has changed format in v1.2.0 by @ianhelle in #677

Full Changelog: v2.5.2...v2.5.3

Contributors

ianhelle
Assets 2
Loading

v2.5.2: Hotfix for Holoviz panel compatibility

13 Jun 23:55
@ianhelle ianhelle
v2.5.2
f4e2cb0
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.5.2: Hotfix for Holoviz panel compatibility
  • Release is mainly to align bokeh version requirements with the new release of Holoviz panel.
    • moved bokeh from <3.0.0 to < 4.0.0
  • Also fixes an issue with the MicrosoftSentinel attribute disappearing from msticpy

What's Changed

Full Changelog: v2.5.1...v2.5.2

Contributors

ianhelle
Assets 2
Loading

v2.5.1: Hotfix for import failure

02 Jun 22:50
@ianhelle ianhelle
v2.5.1
9466a77
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.5.1: Hotfix for import failure

Some minor fixes that address:

  • importing msticpy without some non-default azure packages installed failed
  • added more resiliency to query reader so that the whole thing does not fail if there is bad query file.
  • removed initialization dependency on azure-resourcegraph in MicrosoftSentinel class.

What's Changed

Full Changelog: v2.5.0...v2.5.1

Contributors

ianhelle
Assets 2
Loading

v2.5.0

30 May 16:52
@ianhelle ianhelle
v2.5.0
1f87529
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.5.0

Summary of main changes

  • New MS Sentinel and Azure Kusto drivers/data providers - these include support for multi-threaded parallel queries, proxies and user-defined query timeouts.
  • Extensibility model for MSTICPy - you can create private data providers, TI and Context providers and load them into MSTICPy alongside the built-in providers.
  • MS Sentinel repo query download - add current detection and hunting queries from the Sentinel repo as Sentinel queries runnable from MSTICPy/notebooks
  • OSQuery data provider - makes it easy to import OS Query logs to dataframes to do additional processing/analysis on them.
  • Panel tabulator now supported as default data viewer (a million times better than the one we built!)

More details on these changes below

Sentinel and Kusto provider new drivers

This change adds replacement drivers for the MSSentinel and Kusto data providers.
In place of Kqlmagic, these drivers use the azure-kusto-data and azure-monitor-query SDKs, respectively.

Currently these drivers are enabled alongside the existing versions - in a future version we will make these the defaults for Sentinel and Kusto.

Some of the main changes with these new versions:

  • They use the provider names MSSentinel_New and Kusto_New when creating a QueryProvider instance.
  • Both drivers support setting proxies for firewall-protected networks
  • Both drivers support custom configuration of the server timeout via a timeout parameter
  • Both drivers use integrated Azure authentication by default and support the auth_types and tenant_id parameters used elsewhere
    in MSTICPy
  • Both drivers support threaded execution for parallelizing queries (across multiple workspaces/clusters or split by time) - this functionality, however, will be exposed in v2.6.0 via a separate feature.
  • The MSSentinel_New driver allows you to execute the same query across multiple workspaces in parallel and returns the results as a combined dataframe.
  • Some of the previous parameters have been deprecated:
    • mp_az_auth is replaced by auth_types (the former still works but will be removed in a future release).
    • mp_az_auth_tenant_id is replaced by tenant_id (the former is not supported in the new providers).

Note: in order to use these new versions you must have the azure-kusto-data and/or azure-monitor-query Python packages
installed. You can install these using pip install msticpy[azure] or install them separately using pip.

For more details on how to use these providers, see:

Changes specific to the MS Sentinel provider

Connecting to multiple workspaces allows you to run queries across these workspaces and return the combined results as a single Pandas DataFrame. The workspaces must use common authentication credentials and should have the same data schema.

# use workspace names if these workspaces are configured in msticpyconfig.yaml
qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"])

# or use a list of workspace IDs
qry_prov.connect(workspaces=["e6b4bc15-119b-45a2-8f3d-c39ed384ed37", "b17e0e5a...."])

# run query against connected workspaces
qry_prov.SecurityAlert.list_alerts()

Changes specific to the Kusto provider

  • The settings format has changed (although the existing format is still supported albeit with some limited functionality).
    See the Kusto provider documentation for details.
  • In the earlier implementation of driver you can specify a new cluster to connect to in when executing a query. This is no longer supported.
    Once the provider is connected to a cluster it will only execute queries against that cluster. (You can however, call the connect() function to
    connect the provider to a new cluster before running the query.)
  • Filtering pre-defined queries by cluster. If you have MSTICPy query definitions for the Kusto provider, these will all be attached as methods
    of the QueryProvider, when it is created. However, as soon as you connect to a specific cluster, the queries will be filtered down to show
    only the queries that are intended to run on that cluster.
  • New APIs (exposed via the query_provider):
    • get_database_names() - return list of databases for the connected cluster
    • get_database_schema() - return table schema for a database in the cluster
    • configured_clusters() - return a list of clusters configured in msticpyconfig.yaml
    • set_cluster() - switch connected to cluster to a different one (you can use the connect method to do this, which also lets you specify
      additional connection parameters).

Extend MSTICPy with Data provider, TI provider and Context provider plugins

This adds the ability to "side-load" data providers, TI providers and context providers. If you have a data/TI/context source that you want to use in MSTICPy you can write a provider (deriving from one of the base provider classes) and tell MSTICPy where to load it from.

In a future release we'll build on this framework to let you install plugins from external packages and provide some cookie-cutter templates to generate skelton provider classes.

Writing a TI provider or Context provider (partial example)

    class TIProviderHttpTest(HttpTIProvider):
        """Custom IT provider TI HTTP."""

        PROVIDER_NAME = "MyTIProvider"
        _BASE_URL = "https://api.service.com"
        _QUERIES = _QUERIES = {
            "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"),
            "ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"),

Telling MSTICPy to load the plugins

Load on demand

    import msticpy as mp

    mp.load_plugins(plugin_paths="/my_modules")

    # or multiple paths
    mp.load_plugins(
        plugin_paths=["./my_modules", "./my_other_modules"]
    )

Or specify in msticpyconfig.yaml

        ...
        Custom:
            - "testdata"
    PluginFolders:
        - tests/testdata/plugins
    Azure:
        ...

See the new Extending Msticpy section in our docs.
If you want to contribute any of the drivers you write, also check out the new Development section in the MSTICPy docs.

OS Query Provider

Great contribution from @juju4 here (with a bit of collaboration with @ianhelle).
Create a MSTICPy QueryProvider with the data environment name "OSQueryLogs" and load forensic logs from OSQuery.

# specify one or more paths to folders where the dumped JSON OSQuery logs can be found
qry_prov = mp.QueryProvider("OSQueryLogs", data_paths=["~/logs1", "~/logs2"])
qry_prov.connect()
qry_prov.list_queries()
['osquery.acpi_tables',
'osquery.device_nodes',
'osquery.dns_resolvers',
'osquery.events',
'osquery.fim',
'osquery.last',
'osquery.listening_ports',
'osquery.logged_in_users',
'osquery.mounts',
'osquery.open_sockets',
...

Each event type is available as a separate function that returns a pandas DataFrame with the combined events from the logs for that type

qry_prov.osquery.processes()

Downloading Sentinel Detection and Hunting queries for the Sentinel Query Provider

We haven't finished documenting this or integrating it fully, so will leave the full announcement of this until the next release. If you want to play around with it look at the following module:

from msticpy.data.drivers.sentinel_query_reader import download_and_write_sentinel_queries

download_and_write_sentinel_queries(
    query_type="Hunting",    # or "Detections"
    yaml_output_folder="./sentinel_hunting",
)
qry_prov = mp.QueryProvider("Sentinel_New", query_paths=["./sentinel_hunting"])

Since there are lots of queries, the import might take a little while in its current form.

Panel Tabulator now available as a DataViewer control.

HoloViz Panel is a powerful Bokeh-based data exploration & web app framework for Python. It has an immense amount of functionality that you can read about at the Panel documentation site. You need to have panel installed for the Tabulator-based viewer to run (pip install panel).

Unfortunately, the documentation for our Tabulator view never made it into this release but most of the functionality should be obvious from the UI. There are some useful load-time parameters that you can use at startup for things like:

  • selecting an initial column set.
  • adding columns to a per-row expando pane - useful for viewing long column values such as command-line.

We also kept the column chooser widget from the previous data viewer so that you can interactively select which columns to display. The Tabulator MSTICPy initialization parameters are documented in the code.

Most of the Tabulator init parameters are also passed through to the underlying control - which give you an immense amount of control over the viewer. These are described in the Panel Tabulator documentation

Big thanks to our contributors in this release!

@juju4
@jannieli
@ianhelle
@Tatsuya-hasegawa
@FlorianBracq
@danielyates2
@petebryan
@ashwin-patil

What's Changed PR Reference

  • Up...
Read more

Contributors

juju4, petebryan, and 7 other contributors
Assets 2
Loading

v2.4.0

07 Apr 02:23
@ianhelle ianhelle
v2.4.0
b182439
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.4.0

Main changes for this release

There are no huge changes in this release but a good variety of important updates and fixes.
We're also delighted to welcome 3 new contributors to the MSTICPy family:

Thanks so much!

New Threat Intel provider for Pulsedive from @fr0gger #609

This includes a standard MSTICPy TI provider (so you can include it in you collection of providers used for
regular TI checks on IPs, URLs, etc. This provider also contain a few custom methods that let to query
some other facets of the Pulsedive data. For example, the explore function that allows you to use
the pulsedive query language

pddetail = pdlookup.explore(query="ioc=pulsedive.com or threat=AgentTesla")
pddetail

You can also request a can on a domain or URL

pdscan = pdlookup.scan(observable= "alvoportas.com.br")
pdscan

To use any of the Pulsedive features you'll need an account and API key from Pulsedive
See more details of the usage in the Pulsedive notebook

Process tree updates #637

  • @ZeArioch added Process Tree support for FireEye HX data so it should be automatically recognized and render correct
  • We also added the ability to export a process tree as a text object - which is useful if you want to copy and paste
    a tree or part of it into a non-HTML document. See the Process Tree docs for more details
+--  Process: C:Program FilesMicrosoft Monitoring AgentAgentMonitoringHost.exe
   PID: 0x888
   Time: 1970-01-01 00:00:00+00:00
   Cmdline: nan
   Account: nan  LoginID: 0x3e7
   +--  Process: C:WindowsSystem32cscript.exe  PID: 0x364
      Time: 2019-01-15 04:15:26+00:00
      Cmdline: "C:Windowssystem32cscript.exe" /nologo
         "MonitorKnowledgeDiscovery.vbs"
      Account: WORKGROUPMSTICAlertsWin1$  LoginID: 0x3e7
   +--  Process: C:Program FilesMicrosoft Monitoring AgentAgentHealth Service
      StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe  PID:
      0x1c4
      Time: 2019-01-15 04:16:24.007000+00:00
      Cmdline: "C:Program FilesMicrosoft Monitoring AgentAgentHealth
         Service
         StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe"
         GetInventory "C:Program FilesMicrosoft Monitoring
         AgentAgentHealth Service
         StateCT_602681692workServiceStateServiceState.mof" "C:Program
         FilesMicrosoft Monitoring AgentAgentHealth Service
         StateCT_602681692workServiceState"
      Account: WORKGROUPMSTICAlertsWin1$  LoginID: 0x3e7

Miscellaneous fixes #644

This sounds like a small item but contain several important fixes:

  • Azure authentication (az_connect) now avoids throwing exceptions if you ask it to use authentication types (e.g. clientsecret) where parameters are not passed (or available in environment variables). It will now just ignore those credential types and only throw an exception if no usable credential types remain.
  • Updates to API documentation
  • A new IPython magic "%save_to_cell" - this lets you save a Python object (e.g. a DataFrame to a base64-encoded blob in a new cell. The cell contains code to restore the original data. This is subject to the usual caveats about pickle - including the security ones. Do Not run a cell that unpickles some arbitrary data in notebooks that you do not trust.
  • A bunch of changes/fixes to the Sentinel APIs
    • Most of these are fixes related to the newly-supported Sentinel Dynamic Summaries feature
    • Some minor fixes also to Sentinel core

Python Logging support #640

We should have had this from the beginning but it's never too late to start correcting your mistakes.
We've implemented a central logging module and started to instrument some of the code that is especially complex
and where people often get stuck with cryptic errors. E.g. the init_notebook function.
We also enabled in in the authentication modules (az_connect) in #644
Most of the time, this will be invisible. However, if you need it you can just do the following:

# import msticpy as mp   # if not already imported
mp.set_logging_level("INFO")

Then re-run the function that you are having trouble with again.
You can also use the MSTICPYLOGLEVEL variable to control this. And, if you want to log to a file, set the env variable MSTICPYLOGFILE to the path of your log file. (You'll need to restart the kernel/python session and reload MSTICPy for this to take effect).

Support for Bokeh 3.0 #630 #642 and #650

@ctoma73 did some awesome work to track down problems with compatibility with Bokeh 3.0 and fix all of them (a lot were tedious mypy/linting fixes due to some of the more dynamic nature of the Bokeh 3.0 object model).
You'll notice in #650 that we still have Bokeh 2.4.3 in the MSTICPy requirements. We're not going to change that just yet since we want compatibility with PyViz/HoloViz panel - you will likely see some panel-related features in the next minor release.
Despite this (and assuming you can ignore some pip warning about MSTICPy not being compatible with Bokeh 3.x) you can install Bokeh 3.0 after MSTICPy and enjoy the delights of the new release. All of our code should be compatible (tested with 3.0.0 and 3.1.0).

That's all for this release.
We'll likely be doing a follow-on 2.5.0 release that will include several contributions from our 2023 Hackmonth (which turned into a HackNMonths event).

What's Changed

  • Add support for FireEye HX acquisition packages in process_tree by @ZeArioch in #616
  • Adding Pulsedive as Threat Intel provider by @fr0gger in #609
  • Fix error when latest version 3.0.3 of bokeh is installed by @ctoma73 in #630
  • Adding logging and updating settings access by @ianhelle in #640
  • ProcTree and init_notebook fixes by @ianhelle in #637
  • Adding data query paths test for DEX support by @ianhelle in #638
  • Fixing RangeTool with bokeh 3.1.0 not a GestureTool by @ctoma73 in #642
  • Modified the upload_df method to split the data into batches of 10,00… by @jllangley in #633
  • Misc updates for 2.3.2 release: by @ianhelle in #644
  • Reverting to bokeh version 2.4.3 for default install by @ianhelle in #650

New Contributors

Full Changelog: v2.3.1...v2.4.0

Contributors

jllangley, ctoma73, and 3 other contributors
Assets 2
Loading

MSTICPy Feb 2023 Fixes

24 Feb 02:05
@ianhelle ianhelle
v2.3.1
e5a91e5
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
MSTICPy Feb 2023 Fixes

This is minor release with mostly fixes.

Some higlights from the #631 PR

#629 - You can now suppress progress bar for Threat Intel lookups (useful to avoid screen mess
when running multiple lookups from other code)

  tilookup.lookup_iocs(data, progress=False)

#572 - We've had a long-running issue in Azure Machine Learning where the UI does not correctly
handle javascript written by the notebook. This results in JS code in the output cells. While we're waiting
for AML to re-adopt the latest Azure Notebooks package and get rid of this bug altogether we've
added a fix to suppress javascript text for out Kqlmagic data provider

  • Fix to Azure ML use - automatic creation of msticpyconfig.yaml was writing the file to
    the wrong place, so users always got the message that no config file was found.

  • We had a request (again for batch jobs) to remove automatic display of license information in the geoip module.

  • Using MSTICPy offline or in isolated environment - it has always been our goal to support this but
    we recently discovered that we were running a check_version call from init_notebook. This function
    did not handle network failure and crashed the entire init_notebook process. This has been fixed
    so should be runnable offline or in air-gapped networks.

  • Related to this we've also cleaned up remaining units tests that make outbound network requests.

Full Changelist

New Contributors

Full Changelog: v2.2.3...v2.3.1

Contributors

petebryan, ianhelle, and 2 other contributors
Assets 2
Loading

v2.3.0

25 Jan 21:54
@ianhelle ianhelle
v2.3.0
941145e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.3.0

Some new data-related features in this release.

  • Support for the new (still in preview at time of writing) Dynamic Summaries feature of MS Sentinel
  • Added ability to create and use "ad-hoc" parameterized queries for data providers
  • Simple search mechanism for finding queries
  • Support for JSON queries for CyberReason

Support for Microsoft Sentinel Dynamic Summaries

Dynamic Summaries are a Sentinel feature that allow you to persist results of
query jobs in a summarized/serialized form. This might be useful for keeping
results of daily watch jobs, for example. We will be using it in MSTICPy notebooks
to publish more complex result sets from automated notebook runs.

MSTICPy operations available include:

  • Retrieve list of current dynamic Summaries
  • Retrieve a full dynamic summary
  • Create a dynamic summary
  • Delete a dynamic summary
  • Update an existing dynamic summary

Examples:

# list dynamic summaries
sentinel.list_dynamic_summaries()

# create a dynamic summary in Sentinel
sentinel.connect()
sentinel.create_dynamic_summary(
    name="My_XYZ_Summary",
    description="Summarizing the running of the XYZ job.",
    data=summary_df,
    tactics=["discovery", "exploitation"],
    techniques=["T1064", "T1286"],
    search_key="host.domain.dom",
)

The MSTICPy support also includes a DynamicSummary class that lets you
manipulate dynamic summary objects more easily

  # can also import the class directly
  # from msticpy.context.azure.sentinel_dynamic import DynamicSummary
  # dyn_summary = DynamicSummary(....)
  # This example shows using the "factory" method - new_dynamic_summary
  dyn_summary = sentinel.new_dynamic_summary(
      summary_name="My new summary",
      summary_description="Description of summary",
      source_info={"TI Records": "misc"},
      summary_items=ti_summary_df,
  )
  # Add the local summary object to add to the Sentinel dynamic summaries.
  sentinel.create_dynamic_summary(dyn_summary)

# Retrieve a dynamic summary from Sentinel
dyn_summary = sentinel.get_dynamic_summary(
      summary_id="cea27320-829c-4654-bbf0-b14367483418"
)
# the return value is a DynamicSummary object
dyn_summary
  DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0)

By default get_dynamic_summary returns the header data for the summary.

The next example shows how you can also fetch full data for the dynamic
summary (by adding summary_items=True). From the returned object,
you can convert the summary items to a pandas DataFrame.

Note: fetching summary items is done via the Sentinel QueryProvider
since the APIs do not support retrieving these.

    dyn_summary = sentinel.get_dynamic_summary(
        summary_id="cea27320-829c-4654-bbf0-b14367483418",
        summary_items=True
    )

dyn_summary.to_df()
index Ioc IocType QuerySubtype Provider Result Severity Details TimeGenerated
OTX hXXp://38[.]75[.]37[.]1/static/encrypt.min.js url   OTX True 2 {‘pulse_count’: 3, ‘names’: [‘Underminer EK’ 2022-12-15 01:55:15.135136+00:00
VirusTotal hXXp://38[.]75[.]37[.]1/static/encrypt.min.js url   VirusTotal False 0 Request forbidden. Allowed query rate may ha 2022-12-15 01:55:15.135136+00:00
XForce hXXp://38[.]75[.]37[.]1/static/encrypt.min.js url   XForce

You can also create dynamic summaries from a DataFrame and append
DataFrame records to an existing dynamic summary.

Read the full documentation in MSTICPy Sentinel Dynamic Summaries doc

New QueryProvider API to dynamically add a parameterized query.

MSTICPy has always supported the ability to run ad hoc text queries for different providers
and return the results as a DataFrame. Using a static query string like this is quick and easy
if you only want to run a query once but what if you want to re-run with different time
range or host name? A lot of tedious editing or string search/replace!

Adding a full query template to MSTICPy, on the other hand, is overkill for this kind of thing.
Dynamic parameterized queries are especially suited for notebooks - you can create an
in-line parameterized query and have it update with the new parameters every time
you run the notebook.

To use dynamic queries - define the query with parameter placeholders (delimited
with curly braces "{" and "}"), then create parameter objects (these handle any special
formatting for datetimes, lists, etc.).
You add the list of parameter objects along with the replaceable parameter values
when you run the query, as shown below.

# intialize a query provider
qry_prov = mp.QueryProvider("MSSentinel")

# define a query
query = """
SecurityEvent
| where EventID == {event_id}
| where TimeGenerated between (datetime({start}) .. datetime({end}))
| where Computer has "{host_name}"
"""
# define the query parameters
qp_host = qry_prov.Param("host_name", "str", "Name of Host")
qp_start = qry_prov.Param("start", "datetime")
qp_end = qry_prov.Param("end", "datetime")
qp_evt = qry_prov.Param("event_id", "int", None, 4688)

# add the query
qry_prov.add_custom_query(
    name="get_host_events",
    query=query,
    family="Custom",
    parameters=[qp_host, qp_start, qp_end, qp_evt]
)

# query is now available as
qry_prov.Custom.get_host_events(host_name="MyPC"....)

See Dynamically Adding Queries in MSTICPy Docs

QueryProvider - Query Search

As the number of queries for some providers grows, it has become more difficult to quickly
find the right query. We've implemented a simple search capability that lets you search
over the names or properties of queries. It takes four parameters:

  • search - search terms to look for in the
    query name, description, parameter names, table and query text.
  • table - search terms to match on the target table of the query.
    (note: not all queries have the table parameter defined in their metadata)
  • param - search terms to match on a parameter name
  • case - boolean to force case-sensitive matching (default is case-sensitive).

The first three parameters can be a simple string or an iterable (e.g. list, tuple)
of search terms. The search terms are treated as regular expressions. This
means that a the search terms are treated as substrings (if no other
regular expression syntax is included).

Find all queries that have the term "syslog" in their properties

    qry_prov.search("syslog")
    # equivalent to qry_prov.search(search="syslog")
    ['LinuxSyslog.all_syslog',
    'LinuxSyslog.cron_activity',
    'LinuxSyslog.list_account_logon_failures',
    ...

See Search queries in MSTICPY Docs

Support for JSON queries in Data Providers

@FlorianBracq has updated the CyberReason data provider so that it supports JSON queries. The
mechanism that we used for KQL and SQL queries breaks JSON since it is a simple string substitution.
Other data sources that use JSON queries include Elastic - we are planning to leverage the same
mechanism to support parameterized Elastic queries in a future release.
Thanks @FlorianBracq!

What Else has Changed?

Full Changelog: v2.2.0...v2.3.0

Contributors

petebryan, ianhelle, and 3 other contributors
Assets 2
Loading

IoC Defanging, ServiceNow, GCC support for MDE, Python 3.11

29 Nov 04:03
@ianhelle ianhelle
v2.2.0
0fc1912
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
IoC Defanging, ServiceNow, GCC support for MDE, Python 3.11

Highlights

Re-architected context and TI providers

The biggest feature of this release is not directly visible but has involved a huge amount of work by @FlorianBracq.
Florian spotted that our HTTP TI provider (used for several TI services such as VirusTotal, OTX, XForce) could be used more generically, specifically for non-TI sources that provided valuable context, such as ServiceNow. So, he re-worked the TI providers sub-package to pull out generic context provider capabilities used by both TI and non-TI sources.
The immediate benefit of this is the next highlight

ServiceNow context provider

This is yet to be full documented but if you have a ServiceNow instance and want to hook up MSTICPy to query it try the following.

  1. Add your ServiceNow configuration to msticpyconfig.yaml
ContextProviders:
  ServiceNow:
    Primary: True
    Args:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53
      AuthKey: "authkey"
      AuthId: "authid"
    Provider: "ServiceNow"

Note: you can store the secrets in KeyVault in the same way as TI and other Providers - see the Key Vault Secrets section of MSTICPy Settings Editor

Import and instantiate a ContextProvider and look things up

from msticpy.context.contextlookup import ContextLookup

context_lookup = ContextLookup()
result = context_lookup.lookup_observable("10.0.0.1", providers=["ServiceNow"])
result2 = context_lookup.lookup_observable("[email protected]", providers=["ServiceNow"])

Defanging support for IoCExtract and TI Providers

In threat reports, IoCs are often de-fanged to make IP addresses, URLs, etc, not clickable. An example
de-fanged IP address would look something like this 17[.]34[.]21[.]195

Previously these would not be matched by the IoCExtract patterns due to the "escaped" dots.
IoCExtract now supports common de-fanged markup such as

  • "[.]" to escape dots in IP addresses and domains,
  • "@" replaced by "AT"
  • "http(s)" and "(s)ftp(s)" replaced by "hXXp(s)" and "(s)fXp(s)" respectively.

We have also added support for email address patterns to IoCExtract.

TI providers will also accept de-fanged IoCs, removing the de-fanging before submitting them to the provider for lookup.

We've also supplied a couple of utility functions defang_ioc and refang_ioc in msticpy.common.utility. These are not yet added as Pivot functions to IpAddress, Url, Dns, Account but will be added in a future release.

Added GCC support to MDE/M365 data providers

This allows customers working with government clouds to query the correct Defender endpoints.

Python 3.11 officially supported

Although there wasn't anything in our code that was a Py 3.11 blocker, some of our dependencies took
a little while to publish 3.11-compatible wheels. That was all done with SciPy, Statsmodels and ScikitLearn
and our build pipeline now in includes a full test pass on Python 3.11. Many thanks to @tonybaloney for
pushing us through this.

What's Changed

  • Add base for Context Providers by @FlorianBracq in #511
  • Adding skip and warning to test_vt_pivot.py by @ianhelle in #560
  • Improved bug template getting rid of irrelevant sections by @ianhelle in #559
  • Intsights endpoint update. by @FlorianBracq in #526
  • Added support for GCC and Regional Clouds to MDE driver by @petebryan in #525
  • Resourcegraph - Incomplete list returned by @pensivepaddle in #496
  • Bump sphinx-rtd-theme from 1.0.0 to 1.1.0 by @dependabot in #553
  • Sumologic driver: custom dtypes options+fix, add paging, remove days duration int casting by @juju4 in #481
  • New mypy failures in kql_base, elastic_driver, splunk_driver, sumolog… by @ianhelle in #564
  • Bump sphinx-rtd-theme from 1.1.0 to 1.1.1 by @dependabot in #563
  • Add 3.11 to test matrix by @tonybaloney in #546
  • Update dnspython requirement from <=2.0.0 to <3.0.0 by @dependabot in #289
  • Inability to fetch "all" incidents, only 50 by @pensivepaddle in #565
  • Add de-fanging support for iocextract and TI providers by @ianhelle in #536
  • Implementing isort for context classes, adding missing docs by @ianhelle in #567
  • Add support for context provider Service Now by @FlorianBracq in #556
  • Added Sentinel TI integration features. by @petebryan in #532
  • Ianhelle/pygeohash and exceptions 2022 11 11 by @ianhelle in #566
  • Removing debug prints and duplicate code. by @petebryan in #570
  • Moving ASN http lookup to execute at runtime, when whois lookup happens. by @ianhelle in #568
  • Added a new set of Sentinel queries related to network activity using the CommonSecurityLog data source. by @petebryan in #524
  • Fixed issues with dataprovider instances by @ianhelle in #549
  • Adding AzureAuthentication.rst by @ianhelle in #578

Full Changelog: v2.1.5...v2.2.0

Contributors

juju4, tonybaloney, and 5 other contributors
Assets 2
Loading