MSTICPy 2.0

[petebryan]

MSTICPy will shortly be releasing version 2.0. This version change will include a number of breaking changes and presents an opportunity to make some key changes to MSTICPy in order to improve usability and maintainability.

We have a Project to track and manage the work for 2.0 that you can find here: https://github.com/orgs/microsoft/projects/239
You can also find a number of Issues with the MSTICPy2.0 label: https://github.com/microsoft/msticpy/issues?q=is%3Aissue+is%3Aopen+label%3AMSTICPy2.0

We are looking for the community's input on our plans for 2.0. What do you think of the current suggestions? What should we use this opportunity to ensure we include in MSTICPy?

[2xyo]

It would be nice to be able to easily into account the actual period of malicious activity of each observable in lookup_iocs.

Use case: Design of an infrastructure tracking heuristic

Microsoft publishes a report where it announces the seizure and sinkhole of several malicious domains a month ago. If I enrich these domain names (WHOIS, open ports, banners, passive DNS, TLS certs, etc) with msticpy, the result will be the current status of these domains and not the status of these domains when the were malicious before the seizure.

Workaround : specify specific periods to search for : but no, as the time range is the same for all observables and observables can have a different temporality:

• domainA.com : ????-??-?? - 2020-01-31
• domainB.com : 2020-02-01 - 2020-02-30
• domainC.com : 2020-03-01 - ????-??-??
• ...

So, it would be awesome to allow ti_lookup.lookup_iocs to accept this kind of request in order to specify for each observable the period of activity:

observables = {
    'domainA.com': {'end':'2020-01-31' },
    'domainB.com': {'start':'2020-02-01', 'end':'2020-02-30' },
    'domainC.com': {'start':'2020-03-01'},
}

results = ti_lookup.lookup_iocs(data=observables, last_know=True) #last_know : keep only the last result for each observable 

[ianhelle]

This is an interesting idea - one we've thought about before but haven't come up with a good solution for.

I'm assuming that the domains in the example above are meant to be domainA, domainB and domainC (I guess it would be fine if they were different time ranges for the same observable as well).

Some things we'd need to think about:

1. Many provider services don't let you control the date range within the request (although may return a date validity range in the results) - we'd have to maybe do the date filtering in the TI provider class after the results are returned.
2. The TI provider code is pretty generic. I guess some queries (e.g. filehashes) won't have a validity range attached since the file is always bad. Others like IP and domain may have different results for different time ranges. So we'd need to differentiate between these per-provider.

Some options

• Have each TI provider implement its own filtering logic (since the data structure of the results is going to be unique to the service). So, if the lookup was called with a date range (either globally or per-observable), we'd do the query for the observables, then invoke the date filtering method (the default method wouldn't do anything, just return the results.).
• Have the provider parse and return the date ranges as additional columns for the TI results. Then we could do the filtering as a post-query step (or in some other wrapper function that applied the date filtering)

[FlorianBracq]

Hello,

A few ideas on what would be nice-to-have IMO:

• For queries on long time-period (for instance 1w or more): allow the user to save results on disk (and read from there when it needs to be accessed at a later date) rather than having to run the query again
• Improve Sentinel Native API integration to the existing provider to prevent having to handle 2 "Sentinel objects" (query provider + Sentinel API connector) depending on the needs
  • Rely on msticpyconfig.yaml to connect to Sentinel API rather than specifying the parameters when using the non-default instance
• Being able to detect transient query errors (such as a timeout being hit when querying ADX) and improve resilience by allowing to run the query again with either the same parameters or by using/decreasing "split_query_by"
• Support for more "Pivot entities" (username, email addresses, hashes, process id, file name, etc.)

[ianhelle]

I like the caching idea. At least one of the libraries we use (Kqlmagic) does caching like this but we don't expose it.
I imagine you could control by adding a parameter to the query (cache='create'|'read') which would cache results or request to read from a cached copy if it exists. We could prob use something like a hash of the query as a key for the cache.
I think we could use some of the same mechanism to handle split queries - caching each chunk and skipping on re-query (e.g. if it failed on chunk 10 out of 15).

@FlorianBracq - do you think (for transient failures/timeouts) we should automatically re-run the query or prompt? (I know that I've sometimes sent a query then realized that it was badly written and would time out or return mountains of data.

I like the idea of re-submitting with split_query_by. Although, automatically calculating an auto-split freq might have challenges since we wouldn't necessarily know why the query timed out. Maybe failing but suggesting use of split_query_by (on timeout) would be simpler.

@petebryan and I discussed the "all-in-one" Sentinel class. Seems like a good idea. In any case I think we should be able to pass just a workspace="ws_name" (msticpyconfig entry) to either class would be a good idea.
We could create a wrapper class that contained both data query and API, created both providers under the covers and then exposed the methods/queries for both.

On entities
Some of the examples are already there (File path, hash).
It is possible to register different pivot functions for specific attributes of an entity. E.g. Process could have pivot1 use Process.FilePath and pivot2 use Process.Pid (although I'm dubious about the actual example of a PID).
You can also register the same function multiple times for an entity (giving the pivot a different name) where each reference different attributes.
Usually, the only places that the attribute matters is when you use an entity instance as a parameter to the pivot function - then the pivot mechanism will try to extract the attribute value registered in the pivot definition. You can use some pivot functions from an entity instance if the pivot function has a "shortcut" to appear as a top-level method.
E.g.

my_ip = IpAddress(Address="123.4.5.6")
my_ip.geoip()    # will work using 'Address' attribute
my_ip.Sentinel.get_aad_logons_by_ip()   # won't work 

The second one doesn't work since I haven't found a way of getting the Python my_ip object to treat functions in a container (the Sentinel item in this example) as an instance method. These containers are attached to the IpAddress class.
One possibility is to attach (setattr) all pivot methods directly in the entity class (using some kind of munged private name) and then create the methods in the container as references back to the class method. It would look a bit messy if you did a dir() on the entity but I think it should work.

We can certainly add pivots for Mailbox (or account UP) - if we have queries that use these (I think we have a couple for UPN).
One constraint is that we are trying to stay in step with our V3 entity schema - this has some entities that we haven't implemented. We don't really want to create arbitrary entities if we don't have to (e.g. I'd argue that process id is not an entity, just an attrib of a Process entity).

Happy to talk more about this.

[FlorianBracq]

Hello @ianhelle, thanks for taking the time to answer!

I like the idea of the cache parameter, but using the query "hash" as a key might need some rework on the split_query_by (unless I missed something) as if we want to run a query for the past 7 days and split by "1d", I'm not sure that the split is made from midnight to midnight to ensure consistent times for future searches + the last chunk would have to be redone in all cases? Also, if we have 10 successful chunks out of 15 and the 11th fails, should we run the queries for chunks 12 to 15? I would say yes, but happy to hear your view!

For transient failures, I would assume that any query that is either:

• executed from a YAML file as been tested and its syntax is correct
• being executed with split_query_by and some chunks are already available
  are safe to re-run. A mechanism similar to HTTP requests with a retry parameters could be relevant: if this parameter is specified: re-run automatically (first time divinding the "split_query_by" by 2, and for the last retry use the lowest possible split_query_by parameter?

For PID my main example of pivots is to run queries against different tools based on previous results: start a search in ADX for a malicious process and then get more details on said process from EDR (the keys for the search would be PID, host and a time window).

Do you have a reference document for the "V3 entity schema"? I'm familiar with the Sentinel entities, but I'm not sure those are the ones you are aligned with? I agree that pid should be an attribute of process rather than an entity by itself + I probably need to spend more time understanding how pivot functions are defined to provide more relevant feedback...

[ianhelle]

Proposals for re-working of folder layout.
We've ended up with a couple of unanticipated problems with the initial re-work of the folder layout:

1. There are a bunch of things that are nested really deeply - making importing them a pain
2. Because we import things into the init.py files in intermediate folders, it causes a lot of circular import problems.

Scroll down to see proposals.

The problem with circular imports is something like the following:

1. module X imports data.context.tiproviders.foo.
   This means anything imported in the init.py of all the intermediate folders gets imported
2. Let's say context.init.py imports domain_tools
3. So domain_tools is imported and domain_tools module tries to import analysis.data.iocextract.
4. iocextract tries to import data.context.ip_tools
   This breaks because iocextract will be trying to import data and data_context, which are partway through being loaded from step 1.

It's all very nasty.

My initial proposal is for a flattened dir structure like this

/analysis
/api
/auth
/common
/config
/context
/data
/datamodel
/nbtools
/sectools (depr)
/transform
/vis
/widget

I based this on a bit of analysis of what is dependent on what.
It would (should) result in a layered model like this where we only have downward dependencies (although across is prob OK)

See attachments for layering diagram and module analysis.

The spreadsheet contains a few sheets:

• module_list is the list of all modules with internal imports along with a notes column and proposed new location
• module_imports is a reference - there is a separate row for each module/import combo so modules are repeated.
• new_location - shows what the locations would be if proposal in "module_list" is implemented - this is in the final "resultant_path" column
• common imports is another reference sheet. Lists modules that are imported by other modules in order of most to least common. This only lists modules imported by at least 2 other modules.

Module analysis module_list.xlsx

Some other thoughts:

• put widgets in their own sub-package - these are used all over the place but don't have any internal dependencies so should be a foundational sub-package

• I want to break pivot stuff up. There are bits that belong to datamodel - the bits to do with registering functions and attaching them to entities but there are some bits that belong outside - e.g. a bit going to data so that data depends on datamodel but there are no datamodel -> data dependencies. My thinking is that (mostly) things that want to register pivots functions should have their own bit of logic in their folder and call the more generic pivot functions.

• We can also have helper modules in the root (like TI that just imports stuff like TILookup, VT from context so you can do from msticpy.ti import TILookup. Since these are not in an init.py they shouldn't interfere and create circular imports.

[ianhelle]

@petebryan and I met yesterday and came up with a revised flatter structure

folder	description
analysis	Data analysis functions - timeseries, anomalies, clustering
auth	authentication and secrets management
common	common used utilities and definitions (e.g. exceptions)
config	configuration and settings
context	enrichment modules some modules may need subfolders - e.g. tiproviders, vtlookup
data	data acquisition/queries
datamodel	entities, soc, pivot core functions
init	package loading and initialization - nbinit, pivot creation modules
transform	simple data processing - decoding, reformatting, schema change, process tree
vis	visualization modules including browsers
widgets	nb widgets modules

Supplementary notes:

Import look and feel:

We're going to adopt a pandas-like approach to importing msticpy
so that the user doesn't have to import to much to get going.

import msticpy as mp
qp = mp.QueryProvider(...)
mp.az_auth()

Entities

Since entities are a critical access point for the pivot functionality we will:
import these into the top level __init__.py. init_notebook will import them
into the notebook global namespace.

mp.init_notebook()
IpAddress.ti_lookup("1.2.3.45")

or

mp.IpAddress.ti_lookup("1.2.3.45")

Pivots and Pandas accessors == 1st class citizens

These should be the primary way MP functionality is used.
Starting with the Sentinel notebook updates we should use these
rather than lower-level imported functions and classes to access functionality.
If the functionality isn't surfaced in one of these, we should add it.

Module search

Expand the current msticpy.search to be use an index that includes
doc strings - so you can find modules, functions etc. with a keyword search.

Misc housekeeping

• pivots - split between core data model stuff and the functions that initialize the pivots
  The latter will go into the init folder
• common.utils - rationalize this into sub-modules so it isn't just a random zoo of functions
• nbinit - re-factor to remove and re-org the functionality
• root-level helper modules - we might (for backward compat or convenience reasons)
  keep some modules in the root that import functionality to a higher level - e.g.
  ti, nbwidgets. These will just import functions, modules from other places and have no
  other code in them
• multi-class modules - may refactor some modules that have several class definitions in
  them (e.g. ti_provider_base)
• global classes that need instantiating. There are a few things written as classes that require
  instantiating before you use them (e.g. IoCExtract, TILookup) - however they have no
  real state - or only global state. These should be singletons. Pivots and pandas accessors
  typically instantiate these so the user doesn't have to - maybe this is enough.