Skip to content

Releases: microsoft/msticpy

User Session Management, MaxMind Geolit fix, Extract nested dicts from Pandas

21 Oct 19:03
@ianhelle ianhelle
v2.14.0
deab7a5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
User Session Management, MaxMind Geolit fix, Extract nested dicts from Pandas Latest
Latest

User Session Configuration

Do you always have one or more data providers or other components that you need to load for every notebook you create?
I do, and got a bit fed up with typing the same lines of code over and over again.

User session configuration lets you specify which providers are loaded, whether or not to connect and which parameters
to supply at load and connect time. You put all of this into a straightforward YAML file and load it using the following:

import msticpy as mp   # you likely will already be doing this
mp.init_notebook()     # and this

mp.load_user_session("my_config.yaml")   # if you have a "mp_user_session.yaml" in the current directory
                                         # you can skip the parameter

This example shows the structure of the YAML:

QueryProviders:
  qry_prov_sent:
    DataEnvironment: MSSentinel
    InitArgs:
      debug: True
    Connect: True
    ConnectArgs:
      workspace: MySoc
      auth_methods: ['cli', 'device_code']
  qry_prov_md:
    DataEnvironment: M365D
Components:
   mssentinel:
      Module: msticpy.context.azure
      Class: MicrosoftSentinel
      InitArgs:
      Connect: True
      ConnectArgs:
          workspace: MySoc
          auth_methods: ['cli', 'device_code']

The providers/components created (e.g. qry_prov_sent in this example)
are published back to your notebook Python namespace, so you'll see
these available as variables ready to use.

This configuration file is equivalent to the following code:

qry_prov_sent = mp.QueryProvider("MSSentinel")
qry_prov_sent.connect(workspace="MySoc", auth_methods=['cli', 'device_code'])
qry_prov_md = mp.QueryProvider("M365D")

from msticpy.context.azure import MicrosoftSentinel
mssentinel = MicrosoftSentinel()
mssentinel.connect(workspace="MySoc", auth_methods=['cli', 'device_code'])

Not a huge saving, on the face of it, but if you create a lot of notebooks or want to use
msticpy in an automation scenario, it can be very helpful.
Include a verbose=True parameter to load_user_session to see more detailed logging of what is going on.
See the full documentation here

Maxmind GeoIPLite fix

Sometime recently (not too sure when) Maxmind changed their download procedure to use
a different URL and authentication mechanism. This was causing auto-update to fail. To use
the new mechanism you need to get your Maxmind User Account ID (login and look at your
account properties) and add that to your msticpyconfig.yaml as shown below.

OtherProviders:
  GeoIPLite:
    Args:
      AccountID: "1234567"
      AuthKey:
        EnvironmentVar: "MAXMIND_AUTH"
      DBFolder: "~/.msticpy"
    Provider: "GeoLiteLookup"

Extract nested dictionaries from pandas column to multiple rows/columns

@pioneerHitesh has added this as a new method in the mp_pivot pandas extension:

data_df.mp_pivot.dict_to_dataframe(col="my_nested_column")

It returns a dataframe with the column recursively expanded:

  • lists become new rows
  • dictionaries become new columns

So a column with the following structure:

NCol
0 {'A': ['A1', 'A2', 'A3'], 'B': {'B1': 'B1-1', 'B2': 'B2-1'}}
1 {'A': ['A3', 'A4', 'A5'], 'B': {'B3': 'B3-1', 'B4': 'B4-1'}} 
my_df = src_df.mp_pivot.dict_to_dataframe(col="NCol")
my_df

Would be unpacked to:

A.0 A.1 A.2 B.B1 B.B2 B.B3 B.B4
0 A1 A2 A3 B1-1 B2-1 nan nan
1 A3 A4 A5 nan nan B3-1 B4-1

What's Changed

  • Authentication module unit test by @ianhelle in #800
  • Use sessions config and GeoIP download failure by @ianhelle in #801
  • Added Inbuilt function to extract nested JSON by @pioneerHitesh in #798
  • Add max retry parameter to the execution prevent HTTP 429 by @vx3r in #802

New Contributors

Full Changelog: v2.13.1...v2.14.0

Contributors

vx3r, ianhelle, and pioneerHitesh
Assets 2
Loading

Hotfix for authentication error

19 Sep 18:26
@ianhelle ianhelle
v2.13.1
c01b629
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Hotfix for authentication error

We introduced a bug in azure_auth_core that caused Azure authentication to fail.

What's Changed

Full Changelog: v2.13.0...v2.13.1

Contributors

ianhelle and FlorianBracq
Assets 2
Loading

AI documentation assistant, BinaryEdge TI provider and other misc fixes

13 Sep 00:31
@ianhelle ianhelle
v2.13.0
b5e052a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
AI documentation assistant, BinaryEdge TI provider and other misc fixes

We've been quietly doing some work to introduce LLM/GPT/AI capabilities into msticpy.
@EileenG02 has helped us in that direction by building a document Q&A agent using Autogen.

You can try it out in a notebook using the following:

Load the magic extension

%load_ext msticpy.aiagents.mp_docs_rag_magic

Ask a question in a separate cell using the %%ask cell magic

%%ask
What are the three things that I need to connect to Azure Query Provider?

Awesome work @EileenG02!

There's also a new TI provider for BinaryEdge courtesy of @petebryan.

Alongside this there have been quite a few contributions to fix and improve things like:

  • Splunk improvements (thanks @Tatsuya-hasegawa)
  • Fixes for Sentinel provider get_alert_rules to use updated API (thanks @BWC-TomW)
  • A massive amount of type annotation work and fixes to context/TI providers by @FlorianBracq
  • Miscellaneous fixes to things like Sentinel TI provider, MSSentinel tidy-up to more consistently handle parameters,
    correct use of the term CountryOrRegionName from CountryName in geolocation contexts.

The gory details of the PRs follow:

What's Changed

New Contributors

Full Changelog: v2.12.0...v2.13.0

Contributors

petebryan, ianhelle, and 5 other contributors
Assets 2
Loading

Splunk and Sentinel Updates

10 May 23:25
@ianhelle ianhelle
v2.12.0
07a2f0d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Splunk and Sentinel Updates

Sentinel updates

WorkspaceConfig and Sentinel QueryProvider (azure_monito_driver) have had a few updates:

  • handle both old (Kqlmagic) and standard connection string formats in WorkspaceConfig
  • removing a lot of legacy code from WorkspaceConfig
  • Allow additional connection parameters to be used with MSSentinel QueryProvider for
    authentication parameters (e.g. you can now supply authentication parameters like "client_id", "client_secret" to query_provider.connect())
  • msticpyconfig.yaml now supports using an "MSSentinel" key in place of "AzureSentinel"
  • Workspace entries in msticpyconfig.yaml support an Args subkey, where you can add authentication parameters - these will be supplied to the connect() method if not overridden on the command line. Like Args sections for other providers, the values here can be text or references to environment variables or Azure Key Vault secrets.
  • Fix to MSSentinel API update_incident to add full properties

Splunk Updates

  • Added jwt authentication token expiry check.

Other fixes

Fix for vtlookup3.py

  • Fixed problematic way of using nestasyncio - this was causing failures when run from a langchain agent.
    Fix for lookup/tilookup
  • If the progress parameter was not passed it would still try to cancel a non-existent progress task and cause an exception.
    QueryProviders
  • Fix split query time-ranges calculation - thanks to @pjain90 for spotting this.

What's Changed

  • Set up CI with 1ES Azure Pipelines by @ianhelle in #763
  • Update ws_config to handle kqlmagic connection strings by @ianhelle in #767
  • Fix split query time-ranges calculation by @ianhelle in #762
  • Add support for ruff and u/p devcontainer by @ianhelle in #765
  • Add jwt auth token expire check and modify some messages when connecting Splunk by @Tatsuya-hasegawa in #770
  • WSConfig updates by @ianhelle in #771
  • Pass true for props into _build_sent_data when calling update_incident by @kylelol in #774
  • Changing cert thumbprint from Sha1 to Sha256 in Az Kusto driver by @ianhelle in #775

New Contributors

Full Changelog: v2.11.0...v2.12.0

Contributors

kylelol, ianhelle, and 2 other contributors
Assets 2
Loading

Sentinel Split Query fix

25 Mar 23:58
@ianhelle ianhelle
v2.11.0
b3daf13
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Sentinel Split Query fix

This is a minor release mainly to add a warning for Kusto/Sentinel queries that return partial results.
A close friend of MSTICPy (thx @Cyb3r-Monk) had spotted that MSTICPy does not report partial results when doing split queries so it's possible to lose data from the query range silently.

Due to an unfortunate admin error, the fix for this was committed direct to main, so no PR for this is available. :-(

If you want the query to fail (throw an exception) rather than just warn you can supply a new parameter fail_if_partial.
This only affects the Sentinel query provider and works for standard as well as split queries.

NOTE: the documentation has a typo and calls this fail_on_commit - we'll fix that in the next release to support both fail_if_partial and fail_on_partial

Example

qry_prov.exec_query(query_string, fail_if_partial=True)

What's Changed

New Contributors

Full Changelog: v2.10.0...v2.11.0

Contributors

ianhelle, Cyb3r-Monk, and 2 other contributors
Assets 2
Loading

v2.10.0

12 Feb 19:00
@ianhelle ianhelle
v2.10.0
d1c0912
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.10.0

What's Changed

New Contributors

Full Changelog: v2.9.0...v2.10.0

Contributors

aka0, ianhelle, and 7 other contributors
Assets 2
Loading

Defender Advanced hunting, IPQualityScore TI provider

15 Nov 22:08
@ianhelle ianhelle
v2.9.0
74eeb2e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Defender Advanced hunting, IPQualityScore TI provider

Some of the highlights of this release:

IPQualityScore

New TI provider submitted by @petebryan - provides a lot of interesting stats on IPs.

Defender Advanced Hunting API

Thanks to @d3vzer0 our MS Defender client is now able to use the support Graph-based API rather than the legacy
APIs. To use this, for the moment use the DataEnvironment name M365DGraph when you create
query provider. In the next 0.x release we will switch the other aliases for M365D, MDE, MDATP to use this
new interface and deprecate the existing ones.

Startup errors when running in unexpected environments.

init_notebook made some (incorrect) assumptions about when it would be running in a Synapse environment.
Azure Machine Learning have recently changed their default compute to be a Synapse environment.
Fixes here will correct failures due to faulty detection of environment type.

Startup fixes and perf improvements

We've optimized some of the imports done within the package at startup so msticpy should be quicker to
load.

Azure env credentials fix

Although we previously supported the Azure EnvironmentCredential credential type, our implementation allowed
you to use only with ClientID + ClientSecret. The changes allow it to be used with other supported
credential formats - notably username + password and certificate authentication using a certificate file.

Improvements to Entities

Although these are not visible to most people, we try to keep our Entity definitions in sync with the official
Microsoft "V3" entity definitions. We've added a few entity types and updated some of the attributes
to bring this in line, while still allowing backwards compatible attributes to be used.

What's Changed

New Contributors

  • @2xyo made their first contribution in #723

Full Changelog: v2.8.0...v2.9.0

Contributors

2xyo, petebryan, and 5 other contributors
Assets 2
Loading

Stability release

04 Oct 21:05
@ianhelle ianhelle
v2.8.0
8314145
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Stability release

A few bugs had crept in over the last couple of releases: some due to buggy coding, some due the world moving forward. So, many items in this release are to address these.

Among the feature improvements are the following:

  • Documentation and scripts from @ccianelli22 for creating a MSTICPy install for use in isolated (no Internet) environments. This is super useful for customers operating in sovereign clouds or other air-gapped high-security environments.
  • Added Splunk authentication method using security token rather than username/password - thanks @Tatsuya-hasegawa
  • Query yaml file validation by @FlorianBracq
  • Paging for large CyberReason queries by @FlorianBracq
  • Modern method to obtain cloud-specific URL endpoints for Azure services. Previously, we were relying on msrestazure, which is now deprecated for this purpose. Many thanks to @ccianelli22 for the work to do this.
  • Fix (by me) for a bug I'd introduced with the switch to using Azure-monitor-query library for MS Sentinel. When using a connection string with this new driver, the logic failed to parse and extract details from this correctly. Many thanks to @cindraw for reporting this bug.

What's Changed

New Contributors

Full Changelog: v2.7.0...v2.8.0

Contributors

ianhelle, praveenjutur, and 4 other contributors
Assets 2
Loading

2.8.0 pre-release

27 Sep 01:04
@ianhelle ianhelle
v2.8.0.pre1
0d1b18e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.8.0 pre-release Pre-release
Pre-release

Updated method to dynamically fetch Azure endpoints (rather than relying on deprecated msrestazure).
Updated version of Insight data provider

Assets 2
Loading

TI Providers, Sentinel/Kusto Drivers, Query Editor

15 Aug 16:24
@ianhelle ianhelle
v2.7.0
9136798
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
TI Providers, Sentinel/Kusto Drivers, Query Editor

Main Changes in this release

Two new TI Providers

Two cool new providers to add to the growing family in MSTICPy:

  • CrowdSec is a commercial Malicious IP threat service
    with a free tier for limited threat lookups. (big thanks to @sbs2001 for submitting this)
  • AbuseIPDB - is an open/free provider of threat intel
    on malicious IP addresses, providing a central abuse list to lookup IP addresses that have
    been associated with malicious activity. (big thanks to @rrevuelta for submitting this.)

As with other providers, these are automatically enabled for use if you include settings
for the API keys in your msticpyconfig.yaml

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

In v2.5.0 we introduced replacement drivers for the MS Sentinel/LogAnalytics/Azure Monitor
and Kusto/Azure Data Explorer providers.

The new drivers are based on the Azure SDKs for each data service. You can read the release notes
for them here.

The new drivers give several advantages, like being able to run queries across multiple workspaces
or Kusto clusters in parallel. Splitting large queries by time chunks (split_query_by parameter)
will also run multiple segments in parallel, dramatically speeding up the query. The default
parallelism is 4 simultaneous threads but you can change this (although be wary of the impact
on the data service for highly parallel queries - this may affect other users and services accessing
the data).

The new drivers are now the default drivers for these providers. They are used by default for
the "MSSentinel" and "Kusto" data environment identifiers. For backward compatibility, they will
also continue to support the "MSSentinel_New" and "Kusto_New" identifiers.

To invoke the previous Kqlmagic-based drivers use "MSSentinel_Legacy" or "Kusto_Legacy".

This change also brings a dependency change for MSTICPy. The following packages are now
part of the core installed dependencies:

  • azure-kusto-data
  • azure-monitor-query

Kqlmagic and its dependencies are no longer installed by default but can be installed with the "kql" extra:

python -m pip install msticpy[kql]

See these links to read more about the MSSentinel provider and Kusto providers.

Query Editor

We've added an ipywidgets based query template editor .

note: this is somewhat provisional so please be sure to test and report bugs.

The query editor allows you to edit existing query files or create new ones and helps manage
the various query properties (like parameter definitions) and query metadata.

Check out the documentation on how to use this in the Extending section of the MSTICPy documentation.

Updates to Authentication.

The improvements here mainly affect the AzureData and MicrosoftSentinel classes but'
also bring some improvements to the core authentication - such as being able to specify
the Azure cloud from the az_connect function and authenticate by providing an
AzureCredential.

  • You can now authenticate by supplying an AzureCredential as a credential parameter
    for AzureData and MicrosoftSentinel connect methods.
  • The connect methods for both these classes also support cloud parameter to specify different sovereign clouds
  • The __init__ and connect methods are instrumented with logging to help debug issues:
import msticpy as mp
from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.set_logging_level("INFO")
mssentinel = MicrosoftSentinel()
mssentinel.connect()

Other major items

Thanks

Our thanks to the following folks who contributed to this release.
@FlorianBracq
@sbs2001
@rrevuelta
@mbabinski
@Tatsuya-hasegawa

What's Changed

New Contributors

Full Changelog: v2.6.0...v2.7.0

Contributors

ianhelle, rrevuelta, and 5 other contributors
Assets 2
Loading