Skip to content

MSTICPy V2.0.0 Pre-Release 1

Pre-release
Pre-release
Compare
Choose a tag to compare
@ianhelle ianhelle released this 10 May 19:44
· 624 commits to main since this release
v2.0.0.rc1
857835d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to
use and maintain. Several years of organic growth had seen modules created in places that
seemed like a good idea at the time but did not age well.

The discussion about the V2 structure can be found here #320.

This first pre-release of V2.0 has a small number of new features but is mostly
about testing compatibility. We'll be adding features to future pre-releases in
the next week or two.

**Due to the re-organization, many features are no longer in places where they used
to be imported from! **

We have tried to maintain compatibility with old locations by adding "glue" modules.
These allow import of many modules from their previous locations but will issue a Deprecation warning if loaded from here.
The warning will contain the new location of the module - so you should update your code to point to this new location.

This table gives a quick overview of the V2.0 structure

folder description
analysis data analysis functions - timeseries, anomalies, clustering
auth authentication and secrets management
common common used utilities and definitions (e.g. exceptions)
config configuration and settings UI
context enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup
data data acquisition/queries/storage/uploaders
datamodel entities, soc objects
init package loading and initialization - nbinit, pivot modules
nbwidgets nb widgets modules
transform simple data processing - decoding, reformatting, schema change, process tree
vis visualization modules including browsers

Notable things that have moved:

  • most things from the sectools folder have migrated to context, transform or analysis
  • most things from the nbtools folder have migrated to:
    • init - (not to be confused with init) - package initialization
    • vis - visualization modules
  • pivot functionality has moved to init

If you are having trouble finding a module, we have added a simple search function

    import msticpy
    msticpy.search("riskiq")

Any matches will be returned in a table with links to the module documentation

Modules matching 'riskiq'

ModuleHelp
msticpy.context.tiproviders.riskiq msticpy.context.tiproviders.riskiq

The Help link is to the current release version of the documentation, so will not
map to the correct path on ReleaseTheDocs for the release documentation. You should be able to
map this to the pre-release doc version by adding v2.0.0.rc1 into the URL.

https://msticpy.readthedocs.io/en/v2.0.0rc1/api/msticpy.sectools.tiproviders.html

Other Features/Changes

Pandas accessors have been consolidated into two main ones:

  • df.mp.func - contains all of the transformation functions like base64 decoding, ioc searching, etc.
  • df.mp_plot.func - contains all of the visualization accessors (timeline, process tree, etc.)

We've also done a lot of work to clean up warnings, move some remaining Python unittest tests to pytest
format and random other things that you do when you're cleaning house. A fuller but
not especially readable list of changes follows.

Please let us know if you hit any issues with the re-jigged structure
or any other bugs. File them as issues or reach out to [email protected]

List of changes

Co-authored-by: Ian Hellen [email protected]
Co-authored-by: Pete Bryan [email protected]

  • 857835d@Ianhelle/mpconfigedit fix from main 2022 05 22 (#396)* Fix for MpConfigEdit ValueError

  • Updating Dockerfile source to mcr anaconda

  • bd96d40@Ianhelle/v2 reorg directories 2 2022 04 12 (#377)* Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver

  • Some fixes to Kusto common_imports

    • now works with Kusto config entry without instance suffix
    • can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
    • added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
      in the data_families key.

  • Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration

  • Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.

  • Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.

  • Reorganized logic for handling parameters and failing on invalid file path in config module.

  • Moved analysis.data to transform folder

  • Moved data.context to context

  • Moved auth and secrets modules to auth folder

  • Moved nbwidgets to new folder

  • Updated deprecation warning in glue modules to v2.2

  • Fix to URLs in README.md

  • Adding init folder - moved:

    • nbinit.py, user_config.py, pivot.py, azure_ml_tools.py to here
    • also moved vt_pivot.py and pivot_ti_provider.py to init/pivot_init

  • Renaming datamodel/pivots to datamodel/pivot

  • Moved azure_blob_storage.py to data/storage folder

  • Refactored query_container to data_types.py - to be separate types for queries (query_container.py) and pivots (pivot_container.py)

  • Moved browsers to vis folder

  • Updating API docs for moves

  • Moved all pivot functions to init folder.

  • Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD)

  • Added automatic acquisition of globals() in nbinit.py

  • Fixed a couple of bugs in pivot_pipeline.py

  • Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization.

  • Added trap to timeline when supplied with no data.

  • Fixed incorrect escaping in regex in kql_driver.py

  • Notebook updates for errors and invalid links.

  • Added script to run all notebooks for testing

  • Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x). Other components use the default.

  • Updated typing rigor for pkg_config::get_http_timeout

  • Fixed bugs in test test_pkg_config.py and test_code_view.py

  • Fixed re-auth on query issue in KQL driver

  • Fixed kql_driver tests

  • Kusto provider

    • now works with Kusto config entry without instance suffix
    • can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
    • added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
      in the data_famiies key.

  • Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration

  • Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.

  • Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.

  • Reorganized logic for handling parameters and failing on invalid file path in config module.

  • e2e743a@Ianhelle/main mergeback 2022 04 05 (#355)* Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters.

  • Fixed query_time widget so that you can reset time range from parameter

  • Added additional unit test for QueryTime setter

  • Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line.

  • Changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName.

  • Updated TimeSpan class so it has more flexible constructor

  • Added account_id as a parameter for list_aad_signins_by_account query

  • Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts
    to defaults. This can cause errors if the defaults are different to user-specified parameters.
    There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings.

  • aa14784@moved list_sentinel_workspaces to AzureData

  • f3e82d9@New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352.

  • ba33c38@Ianhelle/pivot dataprov selfload 2022 03 15 (#343)* Query providers load pivots dynamically when created.

  • Renamed query paths and changed data_providers so that only env-specific queries are loaded.

  • Moving ensure_df_datetimes to common/data_utils.py to avoid circular imports

  • Consolidated data-related pandas accessors into single module.

  • Fixing circular dependency in iocextract

  • Fixing linting errors in data_providers.py, azure_resource.py, host.py, process.py, pivot_data_queries.py

  • Adding default timeout values to httpx calls.

  • Changing tor_exit_nodes.py Tor provider to defer download of tor list until first lookup

  • Fixing test for trigger Tor node download before running test.

  • Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. (from main branch)

    • Now loads with empty settings rather than throw exception.
    • Added unit test case

  • Aligning splunk_uploader params with base class

  • 243f196@Ianhelle/implement isort branch post-fixes 2022 03 21 (#346)* move query files

  • Stub files and restructure fixes

  • Merging in updates to Azure auth

  • Making tests more resilient for multiple environments

  • restructure cyberreason and splunk queries

  • Fixed incorrect vtlookup

  • Updating missed Conda version for respx

  • Updated cybereason folder names

  • Fixed Sentinel APIs

  • Updated test mocked data to match new Sentinel APIs

  • Renamed data.context_providers to context

  • Renamed data.common to data.core

  • Added docstrings to redirection files so that they link to right location in read-the-docs

  • Updated docs with new paths

  • Updated notebooks with new module paths

  • Changed RTD to generate an API page for each module.

  • Add text to deprecation warning that we'll remove in v2.0.0

  • Adding isort to pre-commit.yaml

  • Isorting all of the imports in msticpy, tests and tools

  • Renaming query folders

  • Removing some automatic imports from msticpy.init.py

  • Adding search function to find modules to utility.py

  • Fixing old paths in test_timeline.py

  • Adding triggers for release branches

  • Adding isort to requirements-dev and conda-reqs-dev.txt

  • Bandit FPs in anomaly sequence modules

  • Fixed failing clustering notebook

  • Errors in notebook and keyvault tests

  • Fixing test issues in MicrosoftDefender.ipynb and EventClustering.ipynb

  • Bug using wrong httpx code syntax in odata_driver.py

  • Removing auto-load of VTLookup in nbinit

  • Forcing notebook tests to use test msticpyconfig-test.yaml

  • Added missing init.py to tests/data/browsers

  • Removing vtlookup import from sectools init because of circular import error

  • Moving IPStack check for API key to first call (rather than init) to avoid error on load.

  • prospector config changed produces deprecation warning and non-zero exit code.

  • Fixing misconfigured prospector.yaml

  • Getting rid of warning from test_nbinit

  • Adding McCabe suppression to ip_utils.py

  • removing version restriction for prospector in Github actions python-package.yaml

  • Powershell viewer PR

  • Elastic driver skeleton and changes to allow driver-specific param substitution

  • Cleaning up some import redirections to point to new locations

  • Fixing circular import in vtfile_behavior

  • Adding placeholder class to allow imports to sort of work even if sub-modules fail to import

  • Refactored _value_or_default in query_source to reduce complexity

  • Fixed import errors in elastic_driver.py and splunk_driver.py

  • Fixed import from old location in nbinit

  • Fixing warning in code_view.py

  • Fixed test failure in test_code_view

  • Changing test_timeline.py to use new mp_plot accessor in place of deprecated one.

  • Fixing test failure in test_timeline.py

  • 3f3140a@Sync changes to main into v2 branch (#330)* adding devcontainer files (#321)

  • Syncing recent IPython-related changes (to skip magic creation if not in IPython)

  • f7923e5@Minor quality of life improvements to AzureData & MicrosoftSentinel (#331)* Updated names in AzureData to match MicrosoftSentinel

  • Added connection checks to Sentinel features

PRs

Full Changelog: v.1.5.2...v2.0.0.rc1

Contributors

petebryan and ianhelle
Assets 2
Loading