Updated user documentation with initial content. (#40)

* Updated user documentation with initial content.

* Fixing some formatting and links

* Redirects needed for readthedocs

* Documenation and minor changes

* typos

* Updated documentaion

* Additional tidy of docs - added FoliumMap.rst

Fixed a number of links
Updated GeoIPLookups.rst
Updated PackageStructure.rst with more complete list and added doc and module links
Added test for test_auditdextract (read_file and clustering)
Change pylint to alert on dup code > 8 lines
Some changes to geoip.py to read dbfolder from config and prevent test errors
Changes to test_linuxsyslog and test_pkg_config to prevent errors due to missing config
Added notebook test to test_process_tree_utils
Warning suppression for test_tiproviders to prevent expected test warnings

* Updating the version number

Co-authored-by: Pete Bryan <[email protected]>

.pylintrc

@@ -361,7 +361,7 @@ ignore-docstrings=yes
 ignore-imports=yes
 
 # Minimum lines number of a similarity.
-min-similarity-lines=6
+min-similarity-lines=8
 
 
 [SPELLING]

README.md

@@ -7,12 +7,12 @@ authoring for [Azure Sentinel](https://azure.microsoft.com/en-us/services/azure-
 Many of the included tools can be used in other security scenarios for threat hunting
 and threat investigation. There are three main sub-packages:
 
-- sectools - python security tools to help with data analysis or investigation
-- nbtools  - Jupyter-specific UI tools such as widgets and data display
-- data     - data interfaces specific to Sentinel/Log Analytics
-
-The package is in an early preview mode so there are likely to be bugs and there are several
-areas that are not yet optimized for performance.
+- **sectools** - Python security tools to help with data enrichment,
+  analysis or investigation.
+- **nbtools** - Jupyter-specific UI tools such as widgets, plotting and
+  other data display.
+- **data** - data layer and pre-defined queries for Azure Sentinel, MDATP and
+   other data sources.
 
 We welcome feedback, bug reports, suggestions for new features and contributions.
 

docs/notebooks/Base64Unpack.ipynb

@@ -23,7 +23,7 @@
    },
    "source": [
     "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
-    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Title:-msticpy---Base64-Decoder\" data-toc-modified-id=\"Title:-msticpy---Base64-Decoder-1\">Title: msticpy - Base64 Decoder</a></span><ul class=\"toc-item\"><li><span><a href=\"#Description:\" data-toc-modified-id=\"Description:-1.1\">Description:</a></span></li><li><span><a href=\"#Table-of-Contents\" data-toc-modified-id=\"Table-of-Contents-1.2\">Table of Contents</a></span></li><li><span><a href=\"#Decoding-Base64-String\" data-toc-modified-id=\"Decoding-Base64-String-1.3\">Decoding Base64 String</a></span></li><li><span><a href=\"#Using-a-DataFrame-as-Input\" data-toc-modified-id=\"Using-a-DataFrame-as-Input-1.4\">Using a DataFrame as Input</a></span><ul class=\"toc-item\"><li><span><a href=\"#Notes\" data-toc-modified-id=\"Notes-1.4.1\">Notes</a></span></li></ul></li><li><span><a href=\"#Interpreting-the-DataFrame-output.\" data-toc-modified-id=\"Interpreting-the-DataFrame-output.-1.5\">Interpreting the DataFrame output.</a></span><ul class=\"toc-item\"><li><span><a href=\"#SourceIndex-column-allows-you-to-merge-the-results-with-the-input-DataFrame\" data-toc-modified-id=\"SourceIndex-column-allows-you-to-merge-the-results-with-the-input-DataFrame-1.5.1\">SourceIndex column allows you to merge the results with the input DataFrame</a></span></li></ul></li><li><span><a href=\"#Decoding-Nested-Base64/Archives\" data-toc-modified-id=\"Decoding-Nested-Base64/Archives-1.6\">Decoding Nested Base64/Archives</a></span></li><li><span><a href=\"#To-Do-Items\" data-toc-modified-id=\"To-Do-Items-1.7\">To-Do Items</a></span></li></ul></li></ul></div>"
+    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Title:-msticpy---Base64-Decoder\" data-toc-modified-id=\"Title:-msticpy---Base64-Decoder-1\">Title: msticpy - Base64 Decoder</a></span><ul class=\"toc-item\"><li><span><a href=\"#Description:\" data-toc-modified-id=\"Description:-1.1\">Description:</a></span></li><li><span><a href=\"#Decoding-Base64-String\" data-toc-modified-id=\"Decoding-Base64-String-1.2\">Decoding Base64 String</a></span></li><li><span><a href=\"#Using-a-DataFrame-as-Input\" data-toc-modified-id=\"Using-a-DataFrame-as-Input-1.3\">Using a DataFrame as Input</a></span><ul class=\"toc-item\"><li><span><a href=\"#Notes\" data-toc-modified-id=\"Notes-1.3.1\">Notes</a></span></li></ul></li><li><span><a href=\"#Interpreting-the-DataFrame-output.\" data-toc-modified-id=\"Interpreting-the-DataFrame-output.-1.4\">Interpreting the DataFrame output.</a></span><ul class=\"toc-item\"><li><span><a href=\"#SourceIndex-column-allows-you-to-merge-the-results-with-the-input-DataFrame\" data-toc-modified-id=\"SourceIndex-column-allows-you-to-merge-the-results-with-the-input-DataFrame-1.4.1\">SourceIndex column allows you to merge the results with the input DataFrame</a></span></li></ul></li><li><span><a href=\"#Decoding-Nested-Base64/Archives\" data-toc-modified-id=\"Decoding-Nested-Base64/Archives-1.5\">Decoding Nested Base64/Archives</a></span></li><li><span><a href=\"#To-Do-Items\" data-toc-modified-id=\"To-Do-Items-1.6\">To-Do Items</a></span></li></ul></li></ul></div>"
    ]
   },
   {
@@ -529,9 +529,9 @@
    "source": [
     "<a id='mergeresults'></a>[Contents](#contents)\n",
     "### SourceIndex column allows you to merge the results with the input DataFrame\n",
-    "Where an input row has multiple IoC matches the output of this merge will result in duplicate rows from the input (one per IoC match). The previous index is preserved in the second column (and in the SourceIndex column).\n",
+    "Where an input row has multiple decoded elements (e.g. a nested encoding or a zip or other archive file), the output of this merge will result in duplicate rows from the input (one per element match). The DataFrame index from the source is preserved in the `src_index` column.\n",
     "\n",
-    "Note: you will need to set the type of the SourceIndex column. In the example below case we are matching with the default numeric index so we force the type to be numeric. In cases where you are using an index of a different dtype you will need to convert the SourceIndex (dtype=object) to match the type of your index column."
+    "Note: you may need to force the type of the `src_index` column to be the same type as the original DataFrame in order to merge. In the example below case we are matching with the default numeric index so we force the type to be numeric. In cases where you are using an index of a different dtype you will need to convert the `src_index` (dtype=object) to match the type of your index column."
    ]
   },
   {
@@ -918,6 +918,24 @@
    "pygments_lexer": "ipython3",
    "version": "3.7.3"
   },
+  "latex_envs": {
+   "LaTeX_envs_menu_present": true,
+   "autoclose": false,
+   "autocomplete": true,
+   "bibliofile": "biblio.bib",
+   "cite_by": "apalike",
+   "current_citInitial": 1,
+   "eqLabelWithNumbers": true,
+   "eqNumInitial": 1,
+   "hotkeys": {
+    "equation": "Ctrl-E",
+    "itemize": "Ctrl-I"
+   },
+   "labels_anchors": false,
+   "latex_user_defs": false,
+   "report_style_numbering": false,
+   "user_envs_cfg": false
+  },
   "toc": {
    "base_numbering": 1,
    "nav_menu": {