Documentation update.

- Removing all references to "git clone" and starting to use mlcommons-box and runner packages from PyPi. - Updating docker runner documentation: adding information about `configure` phase being optional now. - Updating MNIST/Hello World examples and using box_examples repository.
mlcommons · Oct 9, 2020 · dbc55ea · dbc55ea
1 parent c7f09f3
commit dbc55ea
Show file tree

Hide file tree

Showing 5 changed files with 86 additions and 61 deletions.
diff --git a/docs/getting-started/hello-world.md b/docs/getting-started/hello-world.md
@@ -8,51 +8,42 @@ docker group, run the following:
 sudo groupadd docker             # Add the docker group if it doesn't already exist.
 sudo gpasswd -a ${USER} docker   # Add the connected user "${USER}" to the docker group. Change the user name to match your preferred user.
 sudo service docker restart      # Restart the Docker daemon.
-newgrp docker                    # Either do aÂ newgrp dockerÂ or log out/in to activate the changes to groups.
+newgrp docker                    # Either do a `newgrp docker` or log out/in to activate the changes to groups.
 ```
 
 
 ## Host python environment
-Hello World is an example of a simple python program distributed as an MLCommons-Box. This tutorial covers the case
-when the MLCommons-Box library and Hello World box are cloned from the GitHub repository:
+Hello World is an example of a simple python program distributed as an MLCommons-Box docker-based box. Follow the steps
+outlined in the `Introduction` section to create your Python virtual environment, download example MLCommons-Box boxes
+and install standard MLCommons-Box runners. Go to the folder containing MLCommons-Box example boxes and change directory
+to Hello World Box:
 ```
-git clone https://github.com/mlperf/mlbox ./mlbox
-cd ./mlbox
+cd ./hello_world
 ```
 
-Python >= 3.6 is required together with runners' python dependencies:
-```
-virtualenv -p python3.8 ./env
-source ./env/bin/activate
-pip install typer mlspeclib
-export PYTHONPATH=$(pwd)/mlcommons_box:$(pwd)/runners/mlcommons_box_singularity:$(pwd)/runners/mlcommons_box_docker:$(pwd)/runners/mlcommons_box_ssh
-```
-
-Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables.
-```
-export http_proxy=...
-export https_proxy=...
-```
 
 ## Configuring Hello World MLCommons-Box
 Boxes need to be configured before they can run. To do so, users need to run a MLCommons-Box runner with `configure` 
 command providing path to a box root directory and path to a platform configuration file. The Hello World box is a 
 docker-based box, so users provide path to a MLCommons-Box Docker platform configuration file that sets a number of
 parameters, including docker image name:
 ```
-python -m mlcommons_box_docker configure --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml
+mlcommons_box_docker configure --mlbox=. --platform=platforms/docker.yaml
 ```
-The Docker runner will build a docker image for the Hello World box.
+The Docker runner will build a docker image for the Hello World box. In general, this step is optional and is only
+required when MLCommons-Box needs to be rebuild. This can happen when users change implementation files and want to
+re-package their ML project into MLCommons-Box. In other situations, MLCommons-Box runners can auto-detect if
+`configure` command needs to be run before running a MLBox task.
 
 
 ## Running Hello World MLCommons-Box 
 In order to run the Hello World box, users need to provide the path to the root directory of the box, platform
 configuration file and path to a task definition file. Run the following two commands one at a time:
 ```
-python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/alice/hello.yaml
-python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/alice/bye.yaml
+mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/alice/hello.yaml
+mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/alice/bye.yaml
 ```
-Hello World creates a file `examples/hello_world/workspace/chats/chat_with_alice.txt` that contains the following:
+Hello World creates a file `workspace/chats/chat_with_alice.txt` that contains the following:
 ```
 [2020-09-03 09:13:14.236945]  Hi, Alice! Nice to meet you.
 [2020-09-03 09:13:20.749831]  Bye, Alice! It was great talking to you.
@@ -61,9 +52,9 @@ Hello World creates a file `examples/hello_world/workspace/chats/chat_with_alice
 ## Modifying MLCommons-Box
 
 ### Adding new user 
-Create a new file `examples/hello_world/workspace/names/donald.txt` with the following content: `Donald`.
+Create a new file `workspace/names/donald.txt` with the following content: `Donald`.
 
-Create a new file `examples/hello_world/run/donald/hello.yaml` with the following content:
+Create a new file `run/donald/hello.yaml` with the following content:
 ```yaml
 schema_type: mlbox_invoke
 schema_version: 1.0.0
@@ -77,7 +68,7 @@ output_binding:
         chat: $WORKSPACE/chats/chat_with_donald.txt
 ```
   
-Create a new file `examples/hello_world/run/donald/bye.yaml` with the following content:
+Create a new file `run/donald/bye.yaml` with the following content:
 ```yaml
 schema_type: mlbox_invoke
 schema_version: 1.0.0
@@ -93,10 +84,10 @@ output_binding:
 
 Run the following two commands one at a time:
 ```
-python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/donald/hello.yaml
-python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/donald/bye.yaml
+mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/donald/hello.yaml
+mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/donald/bye.yaml
 ```
-The Hello World box creates a file `examples/hello_world/workspace/chats/chat_with_donald.txt` that contains the
+The Hello World box creates a file `workspace/chats/chat_with_donald.txt` that contains the
 following:
 ```
 [2020-09-03 09:23:09.569558]  Hi, Donald! Nice to meet you.
@@ -109,7 +100,7 @@ Because how Hello World box was implemented, the greeting message is always the
 update the implementation so that if this is not the first time Alice says `hello`, the  MLBox will respond: `Nice to 
 see you again.`.
 
-Modify the file `examples/hello_world/build/hello_world.py`. Update the function named `get_greeting_message` on line
+Modify the file `build/hello_world.py`. Update the function named `get_greeting_message` on line
 14. It should have the following implementation:
 ```python
 def get_greeting_message(chat_file: str) -> str:
@@ -118,11 +109,11 @@ def get_greeting_message(chat_file: str) -> str:
 
 Since we updated a file in the `build` subdirectory, we need to re-configure the Hello World box:
 ```
-python -m mlcommons_box_docker configure --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml
+mlcommons_box_docker configure --mlbox=. --platform=platforms/docker.yaml
 ```
 Now, run two `hello` task again:
 ```
-python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/alice/hello.yaml
+mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/alice/hello.yaml
 ```
 The MLBox recognized it was not the first time it talked to Alice, and changed the greeting:
 ```

diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md
@@ -1 +1,33 @@
 # Getting Started
+
+This section describes some of the example MLCommon-Box boxes, in particular it covers the following topics:
+
+- Setting up python environment.
+- Running simple MLCommon-Box boxes.
+- Detailed description of the internal structure of MLCommons-Box boxes.
+
+
+## Repository of MLCommons-Box examples.
+MLCommons hosts a simple GitHub-based repository with example MLCommons-Box boxes. It is located [here](https://github.com/mlperf/box_examples).
+
+
+## Setting-up python environment
+In various tutorials we start with setting up Python environment and downloading MLCommons-Box boxes. Here is the step by step guide:
+```
+# Clone MLCommons-Box Examples
+git clone https://github.com/mlperf/box_examples.git && cd ./box_examples
+
+# Create Python Virtual Environment
+virtualenv -p python3 ./env && source ./env/bin/activate
+
+# Install MLCommons-Box Docker runner 
+pip install mlcommons-box-docker
+
+# Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables.
+# export http_proxy=...
+# export https_proxy=..
+
+# Optionally, install other runners
+# pip install mlcommons-box-singularity
+# pip install mlcommons-box-ssh
+``` 
diff --git a/docs/getting-started/mnist.md b/docs/getting-started/mnist.md
@@ -300,56 +300,50 @@ mnist/                                   # MLBox root directory.
 
 
 ## Running MNIST MLCommons-Box
-This tutorial covers the case when both MLCommons-Box library and the MNIST box are cloned from the GitHub repository:
-```
-git clone https://github.com/mlperf/mlbox ./mlbox
-cd ./mlbox
+We need to setup the Python virtual environment. These are the steps outlined in the `Introduction` section except we do
+not clone GitHub repository with the example MLCommons-Box boxes. 
 ```
+# Create Python Virtual Environment
+virtualenv -p python3 ./env && source ./env/bin/activate
 
-Python >= 3.6 is required together with runners' python dependencies:
-```
-virtualenv -p python3.8 ./env
-source ./env/bin/activate
-pip install typer mlspeclib
-export PYTHONPATH=$(pwd)/mlcommons_box:$(pwd)/runners/mlcommons_box_singularity:$(pwd)/runners/mlcommons_box_docker:$(pwd)/runners/mlcommons_box_ssh
-```
+# Install MLCommons-Box Docker and Singularity runners 
+pip install mlcommons-box-docker mlcommons-box-singularity
 
-Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables.
-```
-export http_proxy=...
-export https_proxy=...
-```
+# Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables.
+# export http_proxy=...
+# export https_proxy=..
+``` 
 
 > Before running MNIST box below, it is probably a good idea to remove tasks' outputs from previous runs that are
-> located in `examples/mnist/workspace`. All directories except `parameters` can be removed.
+> located in the `workspace` directory. All directories except `parameters` can be removed.
 
 
 ### Docker Runner
 Configure MNIST box:
 ```
-python -m mlcommons_box_docker configure --mlbox=examples/mnist --platform=examples/mnist/platforms/docker.yaml
+mlcommons_box_docker configure --mlbox=. --platform=platforms/docker.yaml
 ```
 
 Run two tasks - `download` (download data) and `train` (train tiny neural network):
 ```
-python -m mlcommons_box_docker run --mlbox=examples/mnist --platform=examples/mnist/platforms/docker.yaml --task=examples/mnist/run/download.yaml
-python -m mlcommons_box_docker run --mlbox=examples/mnist --platform=examples/mnist/platforms/docker.yaml --task=examples/mnist/run/train.yaml
+mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/download.yaml
+mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/train.yaml
 ```
 
 
 ### Singularity Runner
-Update path to store Singularity image. Open `examples/mnist/platforms/singularity.yaml` and update the `image` value
+Update path to store Singularity image. Open `platforms/singularity.yaml` and update the `image` value
 that is set by default to `/opt/singularity/mlperf_mlbox_mnist-0.01.simg` (relative paths are supported, they are
-relative to `examples/mnist/workspace`).  
+relative to `workspace`).  
 
 
 Configure MNIST box:
 ```
-python -m mlcommons_box_singularity configure --mlbox=examples/mnist --platform=examples/mnist/platforms/singularity.yaml
+mlcommons_box_singularity configure --mlbox=. --platform=platforms/singularity.yaml
 ```
 
 Run two tasks - `download` (download data) and `train` (train tiny neural network):
 ```
-python -m mlcommons_box_singularity run --mlbox=examples/mnist --platform=examples/mnist/platforms/singularity.yaml --task=examples/mnist/run/download.yaml
-python -m mlcommons_box_singularity run --mlbox=examples/mnist --platform=examples/mnist/platforms/singularity.yaml --task=examples/mnist/run/train.yaml
+mlcommons_box_singularity run --mlbox=. --platform=platforms/singularity.yaml --task=run/download.yaml
+mlcommons_box_singularity run --mlbox=. --platform=platforms/singularity.yaml --task=run/train.yaml
 ```
diff --git a/docs/runners/docker-runner.md b/docs/runners/docker-runner.md
@@ -21,13 +21,13 @@ configure and run phases:
 - __run__: `docker run ... -e http_proxy=${http_proxy} -e https_proxy=${https_proxy} ...`  
 
 
-## Build command
+## Configuring MLBoxes
 Docker runner uses `{MLCOMMONS_BOX_ROOT}/build` directory as the build context directory. This implies that all files
 that must be packaged in a docker image, must be located in that directory, including source files, python requirements,
 resource files, ML models etc. The docker recipe must have the standard name `Dockerfile`.
 
-In current implementation, only docker `build` is supported (i.e., Dockerfile must present). In future releases, Docker
-runner will support docker `pull` as well.
+If `Dockerfile` file exists in `{MLCOMMONS_BOX_ROOT}/build`, the Docker runner assumes that it needs to `build` a docker
+image. If that file does not exists, the Docker runner will try to `pull` image with the specified name.
 
 Docker runner under the hood runs the following command line:  
 ```
@@ -39,8 +39,15 @@ where:
   supported.  
 -  `{image_name}` is the image name defined in the platform configuration file.  
 
+> The `configure` command is optional and users do not necessarily need to be aware about it. The Docker runner
+> auto-detects if docker image exists before running a task, and if it does not exist, the docker runner runs the 
+> `configure` command. During the `configure` phase, docker runner does not check if docker image exists. This means the
+> following. If some of the implementation files have been modified, to rebuild the docker image users need to run
+> the `configure` command explicitly.
 
-## Run command
+
+
+## Running MLBoxes
 Docker runner runs the following command:    
 ```
 {docker_runtime} run --rm --net=host --privileged=true {volumes} {env_args} {image_name} {args}

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -6,6 +6,7 @@ docs_dir: 'docs/'
 nav:
   - Home: index.md
   - Getting Started:
+    - Introduction: getting-started/index.md
     - Hello World: getting-started/hello-world.md
     - MNIST: getting-started/mnist.md
   - Runners: