Multi timeline plot (#26)

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Updated version numbers

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Added Linux Queries and made minor updates

* Addded demo notebook for Data

* Import YAML Files with package

* testing if init required

* update file path for queries

* manifest update

* update manifest

* update manifest again

* Including an exmple query file to manually import

* query update

* query updates

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Added defauly location in case geoip unsucessful

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Added defauly location in case geoip unsucessful

* added cluster_syslog_logons

* added cluster_syslog_logons

* fixed merge issue

* Update variable names

* updated cluster_syslog_logons

* improved user_logon query

* Fixed merge issue

* Added doc_strings

* query updates

* replaced ' with " in query

* Added additional queries

* Black formatted

* formatting test files with black

* updated logon clusters

* reverting black test formatting

* updated queries

* update queries

* running python black formatting against python36

* running python black formatting against python36

* Adding python black line length of 90 characters to match flake8

* update get host data with data lib

* update imports

* updated with new data packages

* added comments

* added host record function and unit test

* Added sudo evet detections

* aded comments to track work

* Include detections doc

* fixed typo

* typo fixes

* query updates

* typo fix

* typo_fix

* added new test

* Added clustering of sudo sessions

* Added application detection in host record

* parse datetime for test data correctly

* removed un-needed functions

* added risk session detection

* renamed linuxsyslog

* update tests with new packge name

* Updated risky cmd detection to include syslog

* query_update

* query update

* query update

* query_udpate

* query update

* added network queries

* query updates

* Query Updates

* syslog_utils

* merges

* merge

* fixes

* fixes

* custom overlay color

* fixes

* format changes

* fixed test failure

* Fixes and feature adds

* minor fixes

* folium fixes

* fixes

* More broken doclinks. Updated README to include TIProviders summary.

* Added notebook tests for nbdisplay and nbwidgets.

Updated Base64Unpack, EventClustering, NotebookWidgets and TIProviders notebooks.
Minor change to base64unpack.py to prevent pandas warning
Updates to documentation/README.md

* Initial update of timeline

* fixes

* Removed Linux elements

* linux events add

* formatting

* Adding checks for no providers or missing keys.

Updating TIProvider docs for this.
Adding missing data query yaml - kql_sent_winevent.yaml

* Initial merge of timeline display.

Adding WinSecurityEvent.json events file

* Adding pre-commit hooks including download_tlds.py

* Moved timeline to timeline.py

Added support for legacy usage, dict usage, and grouped DF usage through single API
Fixed some things with range control:
- time formatted axis labels
- increasing min/max range by 10% (so first/last events are not on the edge of the graph)
Implemented positioning of legend - inline, left, right
Tooltip columns (for dict usage) now taken from all data sources
Correcting some mypy and pylint errors.

* Fixes from testing notebook development:

-entityschema:  fixing __repr__ in
- nbwidgets: added filtering to all select widgets
- security_base: removing broken and deprecated properties adding __repr__
- timeline: setting bigger default range, bug fix not setting tooltip columns
- ti_lookup - remove unneeded import
- ti_provider_base: added severity to LookupResult

* nbwidgets - bug in restoring current index in selected items list

* Fixed error with legend parameter

* Fixing foliummap error to display in notebook.

* More bugs in timeline and eventcluster.

* Missing some changed files

* Black formatting for utility.py

* Fixing some linting warnings.

* Documentation for Event Timeline

* Fixing bandit issue with urlopen

Fixing doc warnings with TIProviders.rst

* 2nd attempt at suppressing Bandit warning. This is using a a fixed http url

* Changing RangeTool title and adding small font help string

* Refactored and consolidated code. Implemented review comments from Pete.

* Bug in getting ref_time value

.pre-commit-config.yaml

@@ -1,6 +1,23 @@
 repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v2.3.0
+    hooks:
+    -   id: check-yaml
+    -   id: check-json
+    -   id: trailing-whitespace
+        args: [--markdown-linebreak-ext=md]
 -   repo: https://github.com/ambv/black
     rev: stable
     hooks:
     - id: black
-      language_version: python3.6
+      language: python
+      language_version: python3.7
+-   repo: local
+    hooks:
+    - id: download_tlds
+      name: download_tlds
+      entry: python -m pre-commit.download_tlds
+      pass_filenames: False
+      language: python
+      types: [python]
+      args: [--tld-seed=./msticpy/sectools/tld_seed.txt]

MANIFEST.in

@@ -1,3 +1,4 @@
 include msticpy/data/queries/*.yaml
 include msticpy/msticpyconfig.yaml
 include msticpy/sectools/tld_seed.txt
+include msticpy/resources/*

README.md

@@ -12,7 +12,8 @@ and threat investigation. There are three main sub-packages:
 - data     - data interfaces specific to Sentinel/Log Analytics
 
 The package is in an early preview mode so there are likely to be bugs and there are several
-areas that are not yet optimized for performance. 
+areas that are not yet optimized for performance.
+
 We welcome feedback, bug reports, suggestions for new features and contributions.
 
 ## Installing
@@ -131,6 +132,7 @@ values of the host and guids in the string and using delimiters or tokens to
 group the values. This allows you to more easily see distinct patterns of 
 activity.
 
+[Event Clustering Notebook](./docs/notebooks/EventClustering.ipynb)
 
 
 ### outliers