diff --git a/.buildinfo b/.buildinfo new file mode 100644 index 0000000..4b22c09 --- /dev/null +++ b/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: dffc969733c9266c84016d962e3ad9c6 +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/.nojekyll b/.nojekyll new file mode 100644 index 0000000..e69de29 diff --git a/_sources/api.rst.txt b/_sources/api.rst.txt new file mode 100644 index 0000000..12270c6 --- /dev/null +++ b/_sources/api.rst.txt @@ -0,0 +1,16 @@ +.. _API Documentation: + +API Documentation +================= + +Main Routine +------------ +.. autofunction:: thetis.thetis + +MLflow Format +------------- +.. autofunction:: thetis.thetis_mlflow + +Tensorboard Format +------------------ +Coming soon diff --git a/_sources/aspects.rst.txt b/_sources/aspects.rst.txt new file mode 100644 index 0000000..dce8795 --- /dev/null +++ b/_sources/aspects.rst.txt @@ -0,0 +1,18 @@ +.. _Background: + +Safety Aspects +============== + +In this section, we give a detailed overview about the safety aspects that are covered by Thetis. + +.. toctree:: + :maxdepth: 1 + + safety-aspects/uncertainty + safety-aspects/performance + safety-aspects/data_evaluation + safety-aspects/fairness + safety-aspects/robustness + safety-aspects/explainability + + diff --git a/_sources/compliance.rst.txt b/_sources/compliance.rst.txt new file mode 100644 index 0000000..b53591c --- /dev/null +++ b/_sources/compliance.rst.txt @@ -0,0 +1,5 @@ +ISO 26262 Compliance +==================== + +We are working to help users qualifying the Thetis toolbox according to ISO 26262 „Road vehicles – Functional safety“. +If you need further information, please contact us at `efs-techhub.com `__. \ No newline at end of file diff --git a/_sources/configuration.rst.txt b/_sources/configuration.rst.txt new file mode 100644 index 0000000..bd7058f --- /dev/null +++ b/_sources/configuration.rst.txt @@ -0,0 +1,224 @@ +.. _Configuration: + +Configuration +============= + +Thetis needs a YAML configuration file that specifies several aspects, e.g., the task, available classes, requested +evaluation aspects, etc. `Download an exemplary configuration file `__ or copy/paste the +following configuration file. An explanation for each configuration aspect can be found below. + + +Example Configuration File +-------------------------- + +An exemplary YAML configuration for Thetis must have the following form: + +.. code-block:: yaml + + # meta data of model predictions and data set + meta: + + model: + name: "" + revision: "" + + dataset: + name: "" + revision: "r1" + + + # examination task. Can be one of: "classification" (binary/multi-class classification), + # "detection" (image-based object detection) + task: "classification" + + # language of the final report. Can be one of: "en", "de" + language: "en" + + # list of distinct classes that can occur within the data set + distinct_classes: ["no person", "person"] + + # in binary classification (when 'distinct_classes' has length of 2), you must specify a positive label out of + # the list of available classes. This is important since you only give a single "confidence" for each prediction, + # targeting the probability of the positive class + binary_positive_label: "person" + + # you can specify some general settings here (atm only detection-specific settings) + task_settings: + + # bounding-box format. Can be one of: "xyxy" (xmin, ymin, xmax, ymax), "xywh" (xmin, ymin, width, height), + # or "cxcywh" (center x, center y, width, height). + detection_bbox_format: "xyxy" + + # list with IoU scores used for object detection evaluation + # Note: the IoU score "0.5" is always active for the evaluation. You can specify more IoU scores if you want + detection_bbox_ious: [0.75] + + # set to true if the bounding boxes are also inferred with a separate variance score (currently not supported) + detection_bbox_probabilistic: false + + # in detection mode, it is possible to set a confidence threshold + # to discard blurry predictions with low confidence + detection_confidence_thr: 0.2 + + # settings for the data evaluation routine + data_evaluation: + examine: true + + # settings for the AI baseline performance evaluation (which should be always performed!) + performance: + examine: true + + # settings for the evaluation of confidence calibration + uncertainty: + examine: true + + # number of bins used for ECE calculation, required for classification and detection evaluation + ece_bins : 20 + + # during ECE/D-ECE computation, bins with a number of samples less than this threshold are ignored + # required for classification and detection evaluation + ece_sample_threshold: 10 + + # number of bins used for D-ECE calculation (object detection), required for detection evaluation + dece_bins: 5 + + # settings for the evaluation of model fairness + fairness: + examine: true + + # specify sensitive attributes that are used for fairness evaluation. For each of these attributes, + # you need to specify the classes for which the attributes are actually valid (out of the labels + # within 'distinct_classes' list). You can also type "all" to mark validity for all classes. + gender: ["no person", "person"] + age: "all" + + +General Application Settings +---------------------------- + +In the following, we give a detailed overview about all possible general configuration settings. + +.. list-table:: Meta information settings describing the customer information, model properties, and used data set. + :widths: 35 10 55 + :header-rows: 1 + + * - Key/Specifier + - Dtype + - Description + * - :code:`meta/model/name` + - string + - Name of the AI model used to generate predictions. + * - :code:`meta/model/revision` + - string + - Revision of the AI model used to generate predictions. + * - :code:`meta/dataset/name` + - string + - Name of the data set holding the ground-truth information. + * - :code:`meta/dataset/revision` + - string + - Revision of the data set holding the ground-truth information. + + +.. list-table:: General application settings + :widths: 35 10 55 + :header-rows: 1 + + * - Key/Specifier + - Dtype + - Description + * - :code:`task` + - string + - Selection of the examination task. Can be one of: "classification" (binary/multi-class classification), "detection" (image-based object detection). + * - :code:`language` + - string + - Language of the final evaluation report. Can be one of: "en" (US English), "de" (German). + * - :code:`distinct_classes` + - list of int or string + - List of distinct classes that can occur within the data set. + * - :code:`binary_positive_label` + - int or string + - In binary classification (when 'distinct_classes' has length of 2), you must specify a positive label out of + the list of available classes. This is important since you only give a single "confidence" for each prediction, + targeting the probability of the positive class. + * - :code:`task_settings/detection_bbox_format` + - string + - Bounding-box format of the provided boxes in object detection mode. Can be one of: "xyxy" (xmin, ymin, xmax, ymax), + "xywh" (xmin, ymin, width, height), or "cxcywh" (center x, center y, width, height). + * - :code:`task_settings/detection_bbox_ious` + - list of float + - List with IoU scores (in [0, 1] interval) used for object detection evaluation. + Note: the IoU score "0.5" is always active for the evaluation. You can specify more IoU scores if you want. + * - :code:`task_settings/detection_bbox_probabilistic` + - boolean + - Currently not used. + * - :code:`task_settings/detection_confidence_thr` + - float + - In detection mode, it is possible to set a confidence threshold (in [0, 1] interval) to discard blurry predictions with low confidence. + +Configuration of Safety Evaluation +---------------------------------- + +.. list-table:: Configuration settings for data set evaluation. + :widths: 35 10 55 + :header-rows: 1 + + * - Key/Specifier + - Dtype + - Description + * - :code:`data_evaluation/examine` + - boolean + - Enables/disables the data evaluation for the final rating & reporting. + +.. list-table:: Configuration settings for AI performance evaluation. + :widths: 35 10 55 + :header-rows: 1 + + * - Key/Specifier + - Dtype + - Description + * - :code:`performance/examine` + - boolean + - Enables/disables the AI performance evaluation (e.g., accuracy, mAP, precision, recall, etc.) for the final reporting. + +.. list-table:: Configuration settings for uncertainty evaluation (uncertainty calibration). + :widths: 35 10 55 + :header-rows: 1 + + * - Key/Specifier + - Dtype + - Description + * - :code:`uncertainty/examine` + - boolean + - Enables/disables the uncertainty evaluation (uncertainty calibration, e.g., computation of the Expected Calibration Error (ECE)) for the final rating & reporting. + * - :code:`uncertainty/ece_bins` + - int + - Number of bins used for the computation of the Expected Calibration Error (ECE), Maximum Calibration Error (MCE), + and the respective reliability diagrams. The default value is 20. + * - :code:`uncertainty/ece_sample_threshold` + - int + - Sample threshold used for the computation of the ECE, MCE, D-ECE, and the respective reliability diagrams to discard + bins with an amount of samples below this threshold. Discarding bins with only a small amount of samples is + recommended to stabilize the ECE/MCE computations. The default value is 10. + * - :code:`uncertainty/dece_bins` + - int + - Number of bins used for the computation of the Decetion Expected Calibration Error (D-ECE) (object detection only) + and the respective reliability diagrams. The D-ECE is the counterpart of the ECE for position-dependent calibration + evaluation of object detection tasks. The default value is 5. + +.. list-table:: Configuration settings for AI fairness evaluation. + :widths: 35 10 55 + :header-rows: 1 + + * - Key/Specifier + - Dtype + - Description + * - :code:`fairness/examine` + - boolean + - Enables/disables the AI fairness evaluation for the final rating & reporting. + * - :code:`fairness/