IoC Defanging, ServiceNow, GCC support for MDE, Python 3.11

Highlights

Re-architected context and TI providers

The biggest feature of this release is not directly visible but has involved a huge amount of work by @FlorianBracq.
Florian spotted that our HTTP TI provider (used for several TI services such as VirusTotal, OTX, XForce) could be used more generically, specifically for non-TI sources that provided valuable context, such as ServiceNow. So, he re-worked the TI providers sub-package to pull out generic context provider capabilities used by both TI and non-TI sources.
The immediate benefit of this is the next highlight

ServiceNow context provider

This is yet to be full documented but if you have a ServiceNow instance and want to hook up MSTICPy to query it try the following.

1. Add your ServiceNow configuration to msticpyconfig.yaml

ContextProviders:
  ServiceNow:
    Primary: True
    Args:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53
      AuthKey: "authkey"
      AuthId: "authid"
    Provider: "ServiceNow"

Note: you can store the secrets in KeyVault in the same way as TI and other Providers - see the Key Vault Secrets section of MSTICPy Settings Editor

Import and instantiate a ContextProvider and look things up

from msticpy.context.contextlookup import ContextLookup

context_lookup = ContextLookup()
result = context_lookup.lookup_observable("10.0.0.1", providers=["ServiceNow"])
result2 = context_lookup.lookup_observable("[email protected]", providers=["ServiceNow"])

Defanging support for IoCExtract and TI Providers

In threat reports, IoCs are often de-fanged to make IP addresses, URLs, etc, not clickable. An example
de-fanged IP address would look something like this 17[.]34[.]21[.]195

Previously these would not be matched by the IoCExtract patterns due to the "escaped" dots.
IoCExtract now supports common de-fanged markup such as

• "[.]" to escape dots in IP addresses and domains,
• "@" replaced by "AT"
• "http(s)" and "(s)ftp(s)" replaced by "hXXp(s)" and "(s)fXp(s)" respectively.

We have also added support for email address patterns to IoCExtract.

TI providers will also accept de-fanged IoCs, removing the de-fanging before submitting them to the provider for lookup.

We've also supplied a couple of utility functions defang_ioc and refang_ioc in msticpy.common.utility. These are not yet added as Pivot functions to IpAddress, Url, Dns, Account but will be added in a future release.

Added GCC support to MDE/M365 data providers

This allows customers working with government clouds to query the correct Defender endpoints.

Python 3.11 officially supported

Although there wasn't anything in our code that was a Py 3.11 blocker, some of our dependencies took
a little while to publish 3.11-compatible wheels. That was all done with SciPy, Statsmodels and ScikitLearn
and our build pipeline now in includes a full test pass on Python 3.11. Many thanks to @tonybaloney for
pushing us through this.

What's Changed

• Add base for Context Providers by @FlorianBracq in #511
• Adding skip and warning to test_vt_pivot.py by @ianhelle in #560
• Improved bug template getting rid of irrelevant sections by @ianhelle in #559
• Intsights endpoint update. by @FlorianBracq in #526
• Added support for GCC and Regional Clouds to MDE driver by @petebryan in #525
• Resourcegraph - Incomplete list returned by @pensivepaddle in #496
• Bump sphinx-rtd-theme from 1.0.0 to 1.1.0 by @dependabot in #553
• Sumologic driver: custom dtypes options+fix, add paging, remove days duration int casting by @juju4 in #481
• New mypy failures in kql_base, elastic_driver, splunk_driver, sumolog… by @ianhelle in #564
• Bump sphinx-rtd-theme from 1.1.0 to 1.1.1 by @dependabot in #563
• Add 3.11 to test matrix by @tonybaloney in #546
• Update dnspython requirement from <=2.0.0 to <3.0.0 by @dependabot in #289
• Inability to fetch "all" incidents, only 50 by @pensivepaddle in #565
• Add de-fanging support for iocextract and TI providers by @ianhelle in #536
• Implementing isort for context classes, adding missing docs by @ianhelle in #567
• Add support for context provider Service Now by @FlorianBracq in #556
• Added Sentinel TI integration features. by @petebryan in #532
• Ianhelle/pygeohash and exceptions 2022 11 11 by @ianhelle in #566
• Removing debug prints and duplicate code. by @petebryan in #570
• Moving ASN http lookup to execute at runtime, when whois lookup happens. by @ianhelle in #568
• Added a new set of Sentinel queries related to network activity using the CommonSecurityLog data source. by @petebryan in #524
• Fixed issues with dataprovider instances by @ianhelle in #549
• Adding AzureAuthentication.rst by @ianhelle in #578

Full Changelog: v2.1.5...v2.2.0