Updating MLCube documentation [WIP] (#278)

MLCube documentation update.

  - Description for MLCube system settings configuration.
  - Explaining MLCube concepts such as platform, task and others.
  - Automatic documentation generation for MLCube command line interface. This is done using `mkdocs-click` extension (what implies that mkdocs requirements now include mlcube requirements).
  - Help messages use markdown syntax for making web-based documentation look nicer. This commit adds support to remove markdown syntax when printing help message on a console.
  - Updating help messages for MLCube commands and their options. In particular, adding custom epilogues to click commands that show usage examples.
  - Transitioning from arguments for configuration parameters to options (-P/-p).

.github/workflows/docs-site.yml

@@ -22,8 +22,8 @@ jobs:
       - name: Build Docs
         working-directory: './'
         run: |
-          mkdocs build --theme material \
-                --site-dir ../site/
+          export PYTHONPATH=$(pwd)/mlcube:${PYTHONPATH}
+          mkdocs build --theme material --site-dir ../site/
       - name: Deploy to GitHub Pages
         uses: peaceiris/[email protected]
         with:

docs/getting-started/cli.md

@@ -0,0 +1,9 @@
+# Command Line Interface
+
+::: mkdocs-click
+    :module: mlcube.__main__
+    :command: cli
+    :prog_name: mlcube
+    :style: table
+    :list_subcommands: true
+

docs/getting-started/concepts.md

@@ -0,0 +1,109 @@
+# MLCube concepts
+
+### Command Line Arguments
+MLCube runtime and MLCube runners accept multiple command line arguments. They can be classified into two categories:
+
+- Fixed command-specific parameters such as `--mlcube`, `--platform` and `--task` for the MLCube's `run` command, or
+  `create_platform` and `rename_platform` for the `config` command.
+- Parameters that override system settings or MLCube configuration. These parameters start with `-P` and should follow
+  [OmegaConf](https://omegaconf.readthedocs.io/)'s format (MLCube uses this library to manage its settings and 
+  configurations). For instance, to override docker runner settings and instruct it to always build MLCube images,
+  one should provide the following command line argument: `-Pdocker.build_strategy=always`.
+
+These command line arguments override system settings and MLCube configuration parameters.
+
+### Effective MLCube configuration
+`Effective MLCube configuration` is the actual configuration that MLCube runners use to run MLCubes. This effective
+configuration is built using the following algorithm. A platform configuration for a runner is retrieved from system 
+settings (users provide the platform name on a command line by passing `--platform=PLATFORM_NAME` argument). Then, 
+the MLCube configuration is loaded from the MLCube project to run, and this configuration can override default 
+configuration retrieved from system settings. The source for this configuration is specified by a user on a command line
+using `--mlcube=MLCUBE_PATH` argument. The third source of configuration is the command line. Users can provide 
+configuration parameters that override default behavior of a MLCube runner, or default parameters of the MLCube project.
+These parameters start with `-P`, for instance, `-Pdocker.build_strategy=always`. These parameters have the highest 
+priority and override any other parameters loaded so far.
+
+### MLCube Configuration
+`MLCube Configuration` provide MLCube-specific configuration, such as implemented `tasks` and, optionally, specific 
+platform (hardware) requirements, for instance, GPU and host memory required to run the tasks. This configuration 
+overrides system settings. For MLCubes that are distributed via GitHub (with source files), this configuration is
+stored in a YAML file with default location being `${MLCUBE_ROOT}/mlcube.yaml`.
+
+### MLCube Configuration Parameter
+`MLCube configuration paramter` is a configuration parameter for MLCube runners or MLCube projects that has (usually)
+a type from the following set: (integer, floating point number, string, bool). Every MLCube runner and all MLCube 
+projects have such parameters, usually organized in a hierarchy. Users can also provide these parameters on a command
+line when they interact with MLCube runtime to override default values for these parameters. MLCube uses 
+[OmegaConf](https://omegaconf.readthedocs.io/) library to manage its configuration. When users provide these parameters
+on a command line, they need to follow OmegaConf rules, in particular, nested parameters should use `.` symbol. Also,
+when providing these parameters on a command line, these parameters must have `-P` prefix. Several examples: 
+```shell
+# Overriding top level parameter. Here, the `description` is a parameter in a global namespace. 
+-Pdescription="MLCube project description"
+
+# Overriding nested parameter. Here, the `build_strategy` is a parameter defined in the `docker` namespace.   
+-Pdocker.build_strategy=always
+```
+
+### MLCube Home Directory
+`MLCube home directory` is the synonym for [MLCube Root Directory](#mlcube-root-directory).
+
+### MLCube Runtime
+The `MLCube Runtime` term is used to describe the core MLCube library with MLCube runners. MLCube runtime is responsible
+for managing MLCube system settings and MLCube configurations, and run MLCubes in various environments.
+
+### MLCube Root Directory
+`MLCube root directory` is the directory that contains MLCube configuration file (`mlcube.yaml`). This definition 
+applies to MLCubes that are distributed, for instance, via GitHub.
+
+### MLCubes
+The term `MLCubes` (or `MLCube project` in singular form) refers to Machine Learning projects packaged and distributed 
+using MLCube stack.
+
+### Platform
+A `platform` is a configured instance of a MLCube runner. Every runner has a default configured instance with the same
+name as runner. For instance, the MLCube docker runner (named `docker`) has a default platform named `docker` as well.
+Users may find it useful to have multiple configured instances (platforms) of one MLCube runner. For instance, if a user
+has personal and corporate accounts in some cloud provider, they can have multiple platforms - one for each account.
+Users directly interact with platforms via command line argument `--platform`. System settings provide platform 
+configurations, and users can manually edit system settings to add new platforms, or use MLCube's runtime `config` 
+command to perform basic operations with system settings, such as creating a new platform. See System Settings 
+description for more detailed explanation of MLCube runners and platforms, and how they relate to each other.
+
+### Runner
+`MLCube runners` are workhorses of MLCube runtime. They run MLCubes in different environments, such as docker and 
+singularity, remote on-prem or cloud compute nodes, and orchestration engines such as Kubernetes and KubeFlow. As part
+of MLCube ecosystem, we provide multiple MLCube reference runners. Users do not directly interact with MLCube runners,
+instead they interact with `platforms`, which are configured instances of MLCube runners.
+
+### System Settings
+MLCube `System Settings` configure MLCube and MLCube runners at a system level. The term `system level` here implies 
+that these settings are not tied to particular MLCubes (MLCube compliant ML projects). Instead, these settings are 
+used by MLCube runners on every machine where MLCube runtime is configured to use these settings. By default, system
+settings are stored in a YAML file with default location being `${MLCUBE_ROOT}/mlcube.yaml`. The location can be
+overriden by exporting the `MLCUBE_SYSTEM_SETTINGS` environment variable.
+
+### Task
+MLCube projects expose their functionality via `tasks`. A task implements one particular function, such as downloading
+a machine learning dataset, preprocessing this dataset, training a machine learning model, testing a machine learning 
+model or registering a machine learning model with the external model registry. It's up to developers to decide how they
+want to organize their projects into tasks. A close analogy would be machine learning pipelines composed of multiple
+steps organized into directed acyclic graph. Tasks in MLCubes are like steps on these pipelines, except that MLCube 
+runtime at this point is not aware about task dependencies, and so the MLCube task model could be described as 
+`bag of tasks` (similarly to `bag of words` term used in natural language processing to describe machine learning models
+that do not take into account positions of words in sentences).
+
+The MLCube examples [project](https://github.com/mlcommons/mlcube_examples) implements several MLCubes, including 
+[MNIST MLCube](https://github.com/mlcommons/mlcube_examples/blob/master/mnist/mlcube.yaml) that implements two tasks:
+`download` (download MNIST dataset) and `train` (train a simple classifier).
+
+When users run MLCubes, they can instruct MLCube runtime to execute a particular task by providing `--task` command
+line argument: `mlcube run --mlcube=. --task=download --platform=docker`. MLCube runtime can also accept 
+comma-separated list of tasks, in which case these tasks will be executed in the order users provided them on a command
+line: `mlcube run --mlcube=. --task=download,train --platform=docker`.
+
+### Workspace
+A `workspace` is a directory where input and output artifacts are stored. By default, its location is 
+`${MLCUBE_ROOT}/workspace`. Users can override this parameter on a command line by providing the `--workspace` argument.
+Users need to provide this parameter each time they run MLCube task, even when these tasks are logically grouped into 
+one execution. A better alternative would be to run multiple tasks at the same time (see task section).

docs/getting-started/system-settings.md

@@ -0,0 +1,103 @@
+# MLCube System Settings
+MLCube system settings configure MLCube and MLCube runners at a system level. The term `system level` here implies that
+these settings are not tied to particular MLCubes (MLCube compliant ML projects). Instead, these settings are used by
+MLCube runners on every machine where MLCube runtime is configured to use these settings.
+
+## Introduction
+When MLCube runners run MLCubes, they need to know not only the content of MLCubes (tasks that MLCubes provide), but 
+also non-standard or custom settings of a user environment. Effective MLCube configuration that MLCube runners end up
+using is constructed by merging configurations from the following sources:
+
+- `System settings` provide non-standard and/or user-specific parameters for MLCube runners. For instance, in system
+  settings users can indicate that they are required to use `sudo` to run docker. Or they can configure MLCube SSH or
+  GCP (Google Cloud Platform) runners with their credentials to be able to run MLCubes on specific remote servers, 
+  either on-prem, or in the cloud.
+- `MLCube configuration` provide MLCube-specific configuration, such as implemented tasks and, optionally, specific
+  platform (hardware) requirements, for instance, GPU and host memory required to run the tasks. This configuration
+  overrides system settings.
+- `Command line parameters` that users provide when they run MLCubes. These parameters have the highest priority and 
+  override system settings and MLCube configuration.
+
+## Location
+MLCube system settings are stored in a YAML file. Default location of this file is `${HOME}/mlcube.yaml`. This file is
+created or updated every time users run any of MLCube commands. When users install a new MLCube runner (for instance, 
+the singularity runner `pip install mlcube-singularity`), MLCube will update the system settings file with this new 
+runner next time MLCube runs. Users can directly modify this file. In addition, MLCube runtime provides `config` 
+command (`mlcube config --help`) to perform typical operations, such as creating a new MLCube runner configuration off 
+existing one. Users can override the location of this file by defining `MLCUBE_SYSTEM_SETTINGS` environment variable. 
+
+## MLCube System Settings 
+The MLCube system settings are stored in a YAML file. This file has the following schema:
+```yaml
+runners:
+  # A dictionary with metadata on installed MLCube runners. This section is updated (if 
+  # necessary) every time MLCube runs (this means that this section is not updated once 
+  # a new runner is installed). Every key in this dictionary is a runner name (must be 
+  # unique across all MLCube runners), and every value is (usually) a dictionary providing 
+  # runner metadata. In general, it is runner-specific. This section does not provide a 
+  # specific configuration for instances of MLCube runners, and users should not modify 
+  # content of this section - it is maintained automatically by MLCube runtime.
+  docker:
+    # MLCube provides several reference runners including docker and singularity runners. 
+    # All reference runners are implemented in Python and are distributed as separate python 
+    # packages on pypi (e.g., `pip install mlcube-docker`). All these reference runners use 
+    # the same metadata schema. Their metadata is a dictionary with just one field - `pkg`. 
+    # Names of MLCube runners in this section are not directly exposed to users via command 
+    # line API.
+    pkg: mlcube_docker
+    # All MLCube reference runners are described with a dictionary with one field (`pkg`) 
+    # that points to a Python package name.
+
+platforms:
+  # This section (a dictionary) configures instances of MLCube runners. Why there might be 
+  # more than one instance of a particular runner? For instance, users might have two Google 
+  # Cloud Platform accounts - personal and corporate. Or they might have access to a number 
+  # of on-prem compute nodes via ssh, and so they will have respective number of MLCube SSH 
+  # runner instances configured here. There is always a default MLCube runner instance that 
+  # has the same name as the runner itself (e.g., for Docker runner the name of a default 
+  # MLCube docker runner is `docker`). This default section is created automatically by 
+  # MLCube runtime if it does not exist.
+  # Every MLCube runner has its own schema (see MLCube runners documentation) with its own 
+  # unique set of configuration parameters.
+  # Names of MLCube runner instances defined here are directly exposed to users via command 
+  # line argument `--platform`:
+  #    - By default, a default MLCube runner instance is configured with the same name as 
+  #      its MLCube runner class name: docker, singularity etc.
+  #    - When users configure their own unique MLCube runner instances (either via `mlcube 
+  #      config create_platform` command, or manually modifying this file(*)), these instances 
+  #      become available to use, i.e., something like is possible:
+  #        $ mlcube run --mlcube=. --task=train --platform=my_mlcube_runner_instance_name
+  #    (*): To configure a new runner instance manually, duplicate default configuration with 
+  #         a new name and change its parameters.
+  singularity:
+    # This is the example of a default configuration for a MLCube reference singularity runner.
+    # This runner instance, as any other instance, defines the `runner` key which servers as 
+    # a foreign key to the `runners` section.
+    runner: singularity
+    # MLCube runner class (defined in `runners` section).
+    image: ${singularity.image}
+    # Image name, usually defined in MLCube configuration file.
+    image_dir: ${runtime.workspace}/.image
+    # Default build directory for MLCubes distributed with sources.
+    singularity: singularity
+    # Singularity executable.
+    build_args: --fakeroot
+    # Build arguments.
+    build_file: Singularity.recipe
+    # Default Singularity build file for MLCubes distributed with sources
+
+  singularity-1.5.3:
+    # This is an example of another MLCube singularity runner instance. Maybe, a user has 
+    # an outdated version that requires sudo and does not support --fakeroot argument.
+    # Then, this user use this name on a command line to run MLCubes with singularity runner:
+    #        $ mlcube run --mlcube=. --task=train --platform=singularity-1.5.3
+    # BTW, users can create this section by running the following command (they need to edit 
+    # it manually though anyway):
+    #        $ mlcube config create_platform singularity singularity-1.5.3 
+    runner: singularity
+    image: ${singularity.image}
+    image_dir: ${runtime.workspace}/.image
+    singularity: sudo singularity
+    build_args:
+    build_file: Singularity.recipe
+```

docs/mkdocs-requirements.txt

@@ -1,4 +1,6 @@
 flake8>=3.7.8
 mkdocs>=1.0.4
 mkdocs-material>=4.4.0
+mkdocs-click
 pymdown-extensions>=7.1
+-r ../mlcube/requirements.txt

mkdocs-requirements.txt

@@ -1,4 +1,6 @@
 flake8>=3.7.8
 mkdocs>=1.0.4
 mkdocs-material>=4.4.0
+mkdocs-click
 pymdown-extensions>=7.1
+-r ./mlcube/requirements.txt

mkdocs.yml

@@ -9,6 +9,9 @@ nav:
     - Installation: getting-started/index.md
     - Hello World: getting-started/hello-world.md
     - MNIST: getting-started/mnist.md
+    - MLCube concepts: getting-started/concepts.md
+    - System Settings: getting-started/system-settings.md
+    - Command Line Interface: getting-started/cli.md
   - Tutorials:
     - How to Create an MLCube: tutorials/create-mlcube.md
   - Runners:
@@ -40,5 +43,7 @@ markdown_extensions:
   - pymdownx.tasklist
   - pymdownx.details
   - pymdownx.superfences
+  - attr_list
+  - mkdocs-click
   - toc:
       permalink: True