MSTICPy Hackathon 2023 Ideas

[ianhelle]

Idea 1 - expanding Dataproviders -

• OSQuery
• Velociraptor
• Velocity3
• Elastic
• ?

[dan-stats-1]

Hello
Can I suggest adding more models to the analysis module?
I'd love to get my hands on a sample of data and see if I can contribute something
Cheers

[Cyb3r-Monk]

@danielyates2 That would be great!

[ianhelle]

Daniel has the PR for his stuff. I'm also happy to help integrate your code @Cyb3r-Monk

[dan-stats-1]

@ianhelle I've got a branch on my fork for @Cyb3r-Monk's beaconing code. I've nearly finished the code but I only have unit tests for about half the functions so need to get that sorted.
Hopefully I'll have another PR for this soon!

Cheers
Dan

[dan-stats-1]

Hey @ianhelle, @Cyb3r-Monk

I got sidetracked with some other things so didn't get to work on this for quite a while. I've finally finished the code however I need to finish the unit testing.

I've been thinking about another model to add to the polling detection module after I've finished the above.

There's a paper by Francesco Sanna Passino and Nick Heard[1] where they develop a method of estimating which data on an edge is automatically generated and which data is human generated.

This builds on the current method we have in MSTICPY (https://github.com/microsoft/msticpy/blob/main/msticpy/analysis/polling_detection.py) and also @Cyb3r-Monk's beaconing code that I'm finishing off.

However Francesco has published the code (https://github.com/fraspass/human_activity/tree/master) and running this on the example datasets he provides takes the following amount of time to complete on my laptop (i7-1165G7 @ 2.80GHz, 16GB RAM).

• Outlook.txt (7,583 rows) -> 5 minutes and 22 seconds
• Dropbox_candy_mix (37,644 rows) -> 31 minutes and 32 seconds

I think there are 3 options:

1. Implement this model in python and try and make some optimisations where possible.
2. Implement the model as a 3rd party package which I would write in Rust with a python interface, hopefully the heavy lifting in rust would be a significant enough speed up. This would then be included as a msticpy dependency
3. Don't implement this model

Let me know what you think about the model in general and the options

Cheers
Dan

References

[1] Sanna Passino, F., Heard, N.A. Classification of periodic arrivals in event time data for filtering computer network traffic. Stat Comput 30, 1241–1254 (2020). https://doi.org/10.1007/s11222-020-09943-9

[Cyb3r-Monk]

I think I would choose the option 3. I'm also trying to develop a new beaconing detection module which is somewhat diffierent than RITA implementation.

[ianhelle]

More on DataProviders

Was thinking for OSQuery, Velociraptor, Velocity3:

• run the tool and gather data to source folder
• have simple local data provider that took source folder as parameter
• the provider has queries that know what data files are expected (e.g. processlist, handles, connections, etc.) and provides an easy way of reading these into dataframes

[ianhelle]

Other issues that might be good candidates for the hackmonth
hackmonth2023 Candidates for hack month 2023

[titanicshark]

Maybe adding details on using msticpy in Google Colaboratory to docs?

[ianhelle]

Do you have those details? :-)
I've done a bit of testing but nothing very definitive.

[petebryan]

Hi all, due to unforeseen circumstances the MSTICPy Hack Month has been moved to February 2023. We apologize for any inconvenience this causes but we look forward to working with you on your contributions next month.

[ianhelle]

The February Hack Month is back on - albeit no longer in January.

Ideas so far

These do not limit the scope - just throwing out some ideas.

1. Data providers for OS Query, Velociraptor, Velocity3 @juju4 Velociraptor & OS Query Data providers #611 Add support for OSQuery #235 Velociraptor Data providers #518
   • data providers may be straightforward
2. Analysis functions in msticpy? or in notebooklets? location TBD - see Analysis functions for MSTICPy #612
3. Data provider for Elastic (there is a partial implementation already checked in) https://github.com/microsoft/msticpy/blob/main/msticpy/data/drivers/elastic_driver.py
4. MSTICPy in Google Colab
5. Polling/beaconing detection @danielyates2 @Cyb3r-Monk Added documentation for the polling detection module #601
6. Adding extensibility framework for DataProviders and TI/Context providers @ianhelle
7. Interactive Timeline control (incrementally add data sets and annotations) @ianhelle
8. Simplified creation of TI/Context providers - yaml or simple code @ianhelle Plugins for TI Providers #508 https://github.com/microsoft/msticpy/tree/ianhelle/dynamic-ti-provider-2022-09-13
9. Integrate panel/Holoviz into msticpy - panel data browser is esp interesting @Cyb3r-Monk
10. Inbuilt function to extract nested JSON in AuditData field in M365 UAL logs @blueteam0ps Inbuilt function to extract nested JSON in AuditData field in M365 UAL logs #589
11. Make it easier to access "difficult" features Easier access to difficult features #558
12. Documentation/Videos - intro to data manipulation and/or visualization in MSTICPy see Documentation Priorities #516
13. OSSEM integration OSSEM Data model integration with MSTICPy #357
14. 2. MSTICPy as conda package

[4R9UN]

Hello @ianhelle , @petebryan,

Would like to see a ChatGPT function in the MSTICPY module that will assist to summarise all of the existing MSTICPY features and will advise the design of the notebook depending on the user's requirements (the main objective is to create dynamic notebooks and function with the help of ChatGPT and MSTICPY for sentinel), what you think about this idea, do we have any workaround?

[fr0gger]

Hi @4R9UN, I have been experimenting on adding a GPT module for MSTICpy, for the moment my experimentation is more focus on analysing some piece of logs and get more details from GPT. But the possibilities are endless. I like your idea as well; a cool idea would be to add a GPT plugin which is able to understand your current investigation and provides some tips (like Clippy but for MSTICpy and powered by GPT) :)

[ianhelle]

That would be more than awesome. I love the idea.
Both from the mechanics of how someone uses msticpy (e.g. the visualizations) and suggestions on the current investigation.
I think that ChatGPTs training set is all 2021 and prior so it has some pretty outdated advice on how to use msticpy.

[4R9UN]

@fr0gger Actually, my initial concept was to streamline the notebooks and ensure that the AI model receives thorough training on our MSTICPY functions. The goal is to have the AI model effectively execute MSTICPY functions and provide results with clear descriptions. The user interface (UI) should remain straightforward and uncluttered, similar to a co-pilot. This means we can simply make requests like, "Analyze Incident number xxxx and retrieve associated User and host information." In response, the system should provide an overview of the incident and potentially offer suggestions for next steps, such as "There are two hosts (Host 1 and Host 2) associated with this incident; let's analyze them," or "Two users are linked to this incident; let's analyze their behavior over the last five days." similar for IOC and other Entities ... , and in last, we will make a button which will generate entire report and export in PDF or HTML format.

This functionality will increase the efficacy of investigation and people can use all the functionality as best suggested by the AI model.

[ianhelle]

Top ideas for hack month so far:

1. Data providers for OS Query, Velociraptor, Velocity3 @juju4 Velociraptor & OS Query Data providers #611 Add support for OSQuery #235 Velociraptor Data providers #518
2. Analysis functions in msticpy? or in notebooklets? location TBD - see Analysis functions for MSTICPy #612
   • Framework for contributing functions @ianhelle @FlorianBracq
   • Analytic functions themselves - Windows, Linux, Cloud apps - Feel free to contribute ideas and implementations
3. MSTICPy notebooks in Google Colab @ashwin-patil
4. Polling/beaconing detection @danielyates2 @Cyb3r-Monk Added documentation for the polling detection module #601
5. Adding extensibility framework for DataProviders and TI/Context providers @ianhelle
6. Integrate panel/Holoviz into msticpy - panel data browser is esp interesting @Cyb3r-Monk
7. Function to extract nested JSON in AuditData field in M365 UAL logs @blueteam0ps Inbuilt function to extract nested JSON in AuditData field in M365 UAL logs #589
   • could be something we add to mp_pivot accessor

plus a perennial favorite:
Documentation/Videos/Notebooks - intro to data manipulation and/or visualization in MSTICPy see #516