Skip to content

Commit

Permalink
Doc notebooks (#4)
Browse files Browse the repository at this point in the history 
* implmenting pylint recommendation

* Implementing pylint recommendations

* Implementing more pylint recommendations

* First set of doc notebooks.

Fixing a few bugs found during the creation of doc notebooks.
Updated several modules with numpy-style docstrings - to allow for auto-documentation

* Fixing failing test by specifying include_paths

* Updates to README.md and updates for unit tests to add additional entities and fix for changes to iocextract defaults. Also doc fix for iocextract.py

* Adding unit test for auditdextract.py.

Use pandas df search in base64unpack.py instead of iterrows then search.
Fixing auditdextract.py to only create process events for EXECVE syscalls. Added code to rename fields if name clash from different sub-records.
  • Loading branch information
@ianhelle
ianhelle authored Mar 27, 2019
1 parent 900c3cb commit 02e9e94
Show file tree
Hide file tree
Showing 27 changed files with 7,037 additions and 494 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,4 @@ venv.bak/

# mypy
.mypy_cache/
/msticpy.code-workspace
187 changes: 153 additions & 34 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,18 +1,63 @@
# MSTIC Jupyter and Python Security Tools

Microsoft Threat Intelligence Python Security Package:
Microsoft Threat Intelligence Python Security Tools.

The **msticpy** package was initially developed to supported Jupyter Notebook
authoring for [Azure Sentinel](https://azure.microsoft.com/en-us/services/azure-sentinel/).
However, many of the components can be used in other security scenarios for threat hunting
and threat investigation. There are three main sub-packages:

- sectools - python security tools to help with data analysis or investigation
- nbtools - Jupyter-specific UI tools such as widgets and data display
- data - data interfaces specific to Sentinel/Log Analytics

The package is in an early preview mode so there are likely to be bugs and there are several
that are not yet optimized for performance. We welcome feedback, bug reports and suggestions
for new features.

## Installing

`pip install msticpy`

or for the latest dev build

`pip install git+https://github.com/microsoft/msticpy`

## Documentation

The public functions, classes and public class methods have docstrings that describe the
parameters and, for more complex functions give a more detailed description of functionality
and outputs. We are in the process of producing more formal documentation on read-the-docs.

Until then, the functionality is described in the following sections and accompanying notebooks.
You can also browse through the sample notebooks (especially the *Windows Alert Investigation* notebook)
to see some of the functionality used in context.

---

## Security Tools Sub-package - `sectools`

## sectools
This subpackage contains several modules helpful for working on security
investigations and hunting:

### base64unpack
Base64 and archive (gz, zip, tar) extractor. Input can either be a single string or a specified column of a pandas dataframe. It will try to identify any base64 encoded strings and decode them. If the result looks like one of the supported archive types it will unpack the contents. The results of each decode/unpack are rechecked for further base64 content and will recurse down up to 20 levels (default can be overridden).

Base64 and archive (gz, zip, tar) extractor. Input can either be a single string
or a specified column of a pandas dataframe. It will try to identify any base64 encoded
strings and decode them. If the result looks like one of the supported archive types it
will unpack the contents. The results of each decode/unpack are rechecked for further
base64 content and will recurse down up to 20 levels (default can be overridden).
Output is to a decoded string (for single string input) or a DataFrame (for dataframe input).

[Base64Unpack Notebook](./doc/Base64Unpack.ipynb)

### iocextract
Uses a set of builtin regular expressions to look for Indicator of Compromise (IoC) patterns. Input can be a single string or a pandas dataframe with one or more columns specified as input.

Uses a set of builtin regular expressions to look for Indicator of Compromise (IoC) patterns.
Input can be a single string or a pandas dataframe with one or more columns specified as input.

The following types are built-in:

- IPv4 and IPv6
- URL
- DNS domain
Expand All @@ -23,68 +68,142 @@ You can modify or add to the regular expressions used at runtime.

Output is a dictionary of matches (for single string input) or a DataFrame (for dataframe input).

[Base64Unpack Notebook](./doc/IoCExtract.ipynb)

### vtlookup
Wrapper class around Virus Total API (https://www.virustotal.com/en/documentation/public-api/).

Wrapper class around [Virus Total API](https://www.virustotal.com/en/documentation/public-api/).
Input can be a single IoC observable or a pandas DataFrame containing multiple observables.
Processing requires a Virus Total account and API key and processing performance is limited to
Processing requires a Virus Total account and API key and processing performance is limited to
the number of requests per minute for the account type that you have.
Support IoC Types:

- Filehash
- URL
- DNS Domain
- IPv4 Address

[VTLookup Notebook](./doc/VTLookup.ipynb)
[VTLookup Notebook](./doc/VirusTotalLookup.ipynb)

### geoip

Geographic location lookup for IP addresses.
This module has two classes for different services:
- GeoLiteLookup - Maxmind Geolite (see https://www.maxmind.com)
- IPStackLookup - IPStack (see https://ipstack.com)
Both services offer a free tier for non-commercial use. However,
a paid tier will normally get you more accuracy, more detail and
a higher throughput rate. Maxmind geolite uses a downloadable database,

- GeoLiteLookup - Maxmind Geolite (see <https://www.maxmind.com>)
- IPStackLookup - IPStack (see <https://ipstack.com>)
Both services offer a free tier for non-commercial use. However,
a paid tier will normally get you more accuracy, more detail and
a higher throughput rate. Maxmind geolite uses a downloadable database,
while IPStack is an online lookup (API key required).

[GeoIP Lookup Notebook](./doc/GeoIPLookups.ipynb)

### eventcluster
This module is intended to be used to summarize large numbers of
events into clusters of different patterns. High volume repeating

This module is intended to be used to summarize large numbers of
events into clusters of different patterns. High volume repeating
events can often make it difficult to see unique and interesting
items. The module uses a pattern-based approach rather than
matching on exact strings - so an admin command that
does some maintenance on thousands of servers with a commandline such as:
```install-update -hostname {host.fqdn} -tmp:/tmp/{GUID}/rollback```
Will be collapsed using the pattern of the command and ignoring
individal host names and guids.
items.

The module contains functions to generate clusterable features from
string data. For example, an administration command that
does some maintenance on thousands of servers with a commandline such as:<br>
```install-update -hostname {host.fqdn} -tmp:/tmp/{GUID}/rollback```<br>
can be collapsed into a single cluster pattern by ignoring the character values
in the string and using delimiters or tokens to group the values.

This is an unsupervised learning module implemented using SciKit Learn DBScan.
[EventClustering Notebook](./doc/EventClustering.ipynb)

## nbtools
This is a collection of data access, display and utility modules
designed to make working with Log Analytics data in Jupyter notebooks
### outliers

Similar to the eventcluster module but a little bit more experimental (read 'less tested').
It uses SkLearn Isolation Forest to identify outlier events in a single data set or using
one data set as training data and another on which to predict outliers.

### auditdextract

Module to load and decode Linux audit logs. It collapses messages sharing the same
message ID into single events, decodes hex-encoded data fields and performs some
event-specific formatting and normalization (e.g. for process start events it will
re-assemble the process command line arguments into a single string). This is still
a work-in-progress.

## Notebook tools sub-package - `nbtools`

This is a collection of display and utility modules
designed to make working with security data in Jupyter notebooks
quicker and easier.
- nbwidgets - groups common functionality such as list pickers,

- nbwidgets - groups common functionality such as list pickers,
time boundary settings, saving and retrieving
environment variables into a single line callable command.
- nbdisplay - functions that implement common display of things like
- nbdisplay - functions that implement common display of things like
alerts, events in a slightly more consumable way than print()
- entityschema - implements entity classes (e.g. Host, Account, IPAddress)
used in Log Analytics alerts and in many of these modules.
- entityschema - implements entity classes (e.g. Host, Account, IPAddress)
used in Log Analytics alerts and in many of these modules.
Each entity encaspulates one or more properties related to the entity.
- query manager - collection of modules that implement common

[Notebooks Tools](./doc/NotebookWidgets.ipynb)

## Data sub-package - `data`

These components are currently still part of the nbtools sub-package but will be
refactored to separate them into their own package.

- query manager - collection of modules that implement common
kql/Log Analytics queries using KqlMagic
- security_alert and security_event - encapsulation classes for alerts
and events. Each has a standard 'entities' property reflecting the
entities found in the alert or event. These can also be used as
- security_alert and security_event - encapsulation classes for alerts
and events. Each has a standard 'entities' property reflecting the
entities found in the alert or event. These can also be used as
meta-parameters for many of the queries. For example the query:
```qry.list_host_logons(provs==[query_times, alert])``` will extract the
value for the ```hostname``` query parameter from the alert.

# Contributing
---

## Clone the notebooks in this repo to Azure Notebooks

Requires sign-in to Azure Notebooks
<a href="https://notebooks.azure.com/import/gh/Microsoft/msticpy"><img src="https://notebooks.azure.com/launch.png" /></a>

## More Notebook Examples

See the following notebooks for more examples of the use of this package in practice:

- Windows Alert Investigation in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb)
or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb)
- Windows Host Explorer in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb)
or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb)
- Office 365 Exploration in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb)
or [NbViewer](https://nbviewer.jupyter.org/github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb)
- Cross-Network Hunting in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb)
or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb)

## To-Do Items

- Refactor data modules into separate package.
- Replace custom data schema with [Intake](https://intake.readthedocs.io/en/latest/).
- Add additional notebooks to document use of the tools.

## Supported Platforms and Packages

- msticpy is OS-independent
- Requires Python 3.6 or later
- Requires the following python packages: pandas, bokeh, matplotlib, seaborn, setuptools, urllib3,
ipywidgets, numpy, attrs, requests, networkx, ipython, scikit_learn, typing
- The following packages are recommended and needed for some specific functionality: Kqlmagic, maxminddb_geolite2,
folium, dnspython, ipwhois

See [requirements.txt](requirements.txt) for more details and version requirements.

---

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.microsoft.com.
the rights to use your contribution. For details, visit <https://cla.microsoft.com>.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
Expand Down
Loading

0 comments on commit 02e9e94

Please sign in to comment.