Documentation update

Author: giltirn

sphinx/source/introduction/ad.rst

@@ -41,10 +41,10 @@ Advanced anomaly analysis
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 1. Histogram Based Outlier Score (HBOS) is a deterministic and non-parametric statistical anomaly detection algorithm. It is implemented as part of Chimbuko's anomaly analysis module. HBOS is an unsupervised anomaly detection algorithm which scores data in linear time. It supports dynamic bin widths which ensures long-tail distributions of function executions are captured and global anomalies are detected better. HBOS normalizes the histogram and calculates the anomaly scores by taking inverse of estimated densities of function executions. The score is a multiplication of the inverse of the estimated densities given by the following Equation
 
-.. math::
-    HBOS_{i} = \log_{2} (1 / density_{i})
+   .. math::
+      HBOS_{i} = \log_{2} (1 / density_{i} + \alpha)
 
-   where :math:`i` is a function execution and :math:`density_{i}` is function execution probability. HBOS works in :math:`O(nlogn)` using dynamic bin-width or in linear time :math:`O(n)` using fixed bin width. After scoring, the top 1% of scores are filtered as anomalous function executions. This filter value can be set at runtime to adjust the density of detected anomalies.
+   where :math:`i` is the index of a particular a function execution and :math:`density_{i}` is function execution probability. The offset :math:`\alpha` is chosen such that the scores lie in the range :math:`0\to 100`. HBOS works in :math:`O(nlogn)` using dynamic bin-width or in linear time :math:`O(n)` using fixed bin width. After scoring, the top 1% of scores are filtered as anomalous function executions. This filter value can be set at runtime to adjust the density of detected anomalies.
 
 2. Another algorithm is added into Chimbuko's advanced anomaly analysis called the COPula based Outlier Detection (COPOD), which is a deterministic, parameter-free anomaly detection algorithm. It computes empirical copulas for each sample in the dataset. A copula defines the dependence structure between random variables. For each sample in the dataset, COPOD algorithm computes left-tail empirical copula from left-tail empirical cumulative distribution function, right-tail copula from right-tail empirical cumulative distribution function, and a skewness-corrected empirical copula using a skewness coefficient calculated from left-tail and right-tail empirical cumulative distribution functions. These three computed values are interpreted as left-tail, right-tail, and skewness-corrected probabilities, respectively. Lowest probability value results in largest negative-log value, which is the score assigned to the sample in the dataset. Samples with the highest scores in the dataset are tagged as anomalous.
 

sphinx/source/introduction/provdb.rst

@@ -2,12 +2,13 @@
 Provenance Database
 *******************
 
-The role of the provenance database is three-fold:
+The role of the provenance database is four-fold:
 
 - To store detailed information about anomalous function executions and also, for comparison, samples of normal executions. Stored in the **anomaly database**.
 - To store a wealth of metadata regarding the characteristics of the devices upon which the application is running. Stored in the **anomaly database**.
 - To store globally-aggregated function profile data and counter averages that reflect the overall running of the application. Stored in the **global database**.
-
+- To store the final AD model for each function. Stored in the **global database**.
+  
 The databases are implemented using `Sonata <https://xgitlab.cels.anl.gov/sds/sonata>`_ which implements a remote database built on top of `UnQLite <https://unqlite.org/>`_, a server-less JSON document store database. Sonata is capable of furnishing many clients and allows for arbitrarily complex document retrieval via `jx9 queries <https://unqlite.org/jx9.html>`_.
 
 In order to improve parallelizability the **anomaly database**, which stores the provenance data and metadata collected from the AD instances, is sharded (split over multiple independent instances), with the AD instances assigned to a shared in a round-robin fashion based on their rank index. The database shared are organized into three *collections*:
@@ -20,6 +21,7 @@ The **global database** exists as a single shard, and is written to at the very
 
 * **func_stats** : function profile information including statistics (mean, std. dev., etc) on the inclusive and exclusive runtimes as well as the number and frequency of anomalues.
 * **counter_stats** : averages of various counters collected over the run.
+* **ad_model** : the final AD model for each function.
 
 The schemata for the contents of both database components are described here: :ref:`io_schema/provdb_schema:Provenance Database Schema` 
 

sphinx/source/introduction/ps.rst

@@ -19,8 +19,9 @@ Design
 
 (**C**)lients (i.e. on-node AD modules) send requests with their locally-computed anomaly detection algorithm parameters to be aggregated with the global parameters and the updated parameters returned to the client. Network communication is performed using the `ZeroMQ <https://zeromq.org>`_ library and using `Cereal <https://uscilab.github.io/cereal/>`_ for data serialization.
 
-For handling requests from a large number of connected clients the parameter server uses a ROUTER-DEALER pattern whereby requests are sent to a **Frontend** router and distributed over thread (**W**)orkers
-via the **Backend** router in round-robin fashion. For the task of updating parameters, workers first acquire a global lock, and update the **in-mem DB**, and return the updated parameters (cf. :ref:`api/api_code:SstdParam`). Global statistics on anomalies and counters are compiled in a similar way using the data sent from the AD instances (cf. `global_anomaly_stats <../api/api_code.html#global-anomaly-stats>`__ and `global_counter_stats <../api/api_code.html#global-counter-stats>`__) and also stored in this database.
+For handling requests from a large number of connected clients the parameter server uses a ROUTER-DEALER pattern whereby requests are sent to a **Frontend** router and distributed over thread (**W**)orkers via the **Backend** router in round-robin fashion.
+
+For the task of updating parameters, scalability is ensured by having each worker write to a separate parameter object in the **In-Mem DB**. These are periodically aggregated by a background thread and the global model updated, typically once per second such that the AD instances receive the most up-to-date model as possible. Global statistics on anomalies and counters are compiled in a similar way using the data sent from the AD instances (cf. `global_anomaly_stats <../api/api_code.html#global-anomaly-stats>`__ and `global_counter_stats <../api/api_code.html#global-counter-stats>`__) and also stored in this database.
 
 A dedicated (**S**)treaming thread (cf. :ref:`api/api_code:PSstatSender`) is maintained that periodically sends the latest global statistics to the visualization server.
 

sphinx/source/io_schema/provdb_schema.rst

@@ -5,7 +5,7 @@ Provenance Database Schema
 Main database
 -------------
 
-Below we describe the JSON schema for the **anomalies** and **normalexecs** collections of the **main database** component of the provenance database.
+Below we describe the JSON schema for the **anomalies**, **normalexecs** and **metadata** collections of the **main database** component of the provenance database.
 
 Function event schema
 ^^^^^^^^^^^^^^^^^^^^^
@@ -114,8 +114,12 @@ For the SSTD (original) algorithm, the **algo_params** field has the following f
 For the HBOS and COPOD algorithms, the **algo_params** field has the following format:
 
 |    {
-|        **"Histogram Bin Counts"** : *the height of the histogram bin*,
-|        **"Histogram Bin Edges"** : *the edges of the bins starting with the lower edge of bin 0 followed by the upper edges of bins 0..N*
+|        **"histogram"**: *the histogram*,
+|        {
+|            **"Histogram Bin Counts"** : *the height of the histogram bin (array)* ,
+|            **"Histogram Bin Edges"** : *the edges of the bins starting with the lower edge of bin 0 followed by the upper edges of bins 0..N (array)*
+|        },
+|        **"internal_global_threshold"** : *a score threshold for anomaly detection used internally*
 |    }
 
 
@@ -180,7 +184,7 @@ Note that the **tid** (thread index) for metadata is usually 0, apart from for m
 Global database
 ---------------
 
-Below we describe the JSON schema for the **func_stats** and **counter_stats** collections of the **global database** component of the provenance database.
+Below we describe the JSON schema for the **func_stats**, **counter_stats** and **ad_model** collections of the **global database** component of the provenance database.
 
 Function profile statistics schema
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -247,3 +251,17 @@ The **counter_stats** collection has the following schema:
 |           }
 | }
 
+AD model schema
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The **ad_model** collection contains the final AD model for each function. It has the following schema:
+
+| {
+|   **"__id"**: *A unique record index*,
+|   **"pid"**: *The program index*,
+|   **"fid"**: *The function index*,
+|   **"func_name"**: *The function name*,
+|   **"model"** : *The model for this function, form dependent on algorithm used (object)*
+| }
+
+The **"model"** entry has the same form as the **"algo_params"** entry of the main database, and is documented above.