diff --git a/docs/getting-started/hello-world.md b/docs/getting-started/hello-world.md index 9e7208c6..7113f889 100644 --- a/docs/getting-started/hello-world.md +++ b/docs/getting-started/hello-world.md @@ -8,31 +8,19 @@ docker group, run the following: sudo groupadd docker # Add the docker group if it doesn't already exist. sudo gpasswd -a ${USER} docker # Add the connected user "${USER}" to the docker group. Change the user name to match your preferred user. sudo service docker restart # Restart the Docker daemon. -newgrp docker # Either do a newgrp docker or log out/in to activate the changes to groups. +newgrp docker # Either do a `newgrp docker` or log out/in to activate the changes to groups. ``` ## Host python environment -Hello World is an example of a simple python program distributed as an MLCommons-Box. This tutorial covers the case -when the MLCommons-Box library and Hello World box are cloned from the GitHub repository: +Hello World is an example of a simple python program distributed as an MLCommons-Box docker-based box. Follow the steps +outlined in the `Introduction` section to create your Python virtual environment, download example MLCommons-Box boxes +and install standard MLCommons-Box runners. Go to the folder containing MLCommons-Box example boxes and change directory +to Hello World Box: ``` -git clone https://github.com/mlperf/mlbox ./mlbox -cd ./mlbox +cd ./hello_world ``` -Python >= 3.6 is required together with runners' python dependencies: -``` -virtualenv -p python3.8 ./env -source ./env/bin/activate -pip install typer mlspeclib -export PYTHONPATH=$(pwd)/mlcommons_box:$(pwd)/runners/mlcommons_box_singularity:$(pwd)/runners/mlcommons_box_docker:$(pwd)/runners/mlcommons_box_ssh -``` - -Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables. -``` -export http_proxy=... -export https_proxy=... -``` ## Configuring Hello World MLCommons-Box Boxes need to be configured before they can run. To do so, users need to run a MLCommons-Box runner with `configure` @@ -40,19 +28,22 @@ command providing path to a box root directory and path to a platform configurat docker-based box, so users provide path to a MLCommons-Box Docker platform configuration file that sets a number of parameters, including docker image name: ``` -python -m mlcommons_box_docker configure --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml +mlcommons_box_docker configure --mlbox=. --platform=platforms/docker.yaml ``` -The Docker runner will build a docker image for the Hello World box. +The Docker runner will build a docker image for the Hello World box. In general, this step is optional and is only +required when MLCommons-Box needs to be rebuild. This can happen when users change implementation files and want to +re-package their ML project into MLCommons-Box. In other situations, MLCommons-Box runners can auto-detect if +`configure` command needs to be run before running a MLBox task. ## Running Hello World MLCommons-Box In order to run the Hello World box, users need to provide the path to the root directory of the box, platform configuration file and path to a task definition file. Run the following two commands one at a time: ``` -python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/alice/hello.yaml -python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/alice/bye.yaml +mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/alice/hello.yaml +mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/alice/bye.yaml ``` -Hello World creates a file `examples/hello_world/workspace/chats/chat_with_alice.txt` that contains the following: +Hello World creates a file `workspace/chats/chat_with_alice.txt` that contains the following: ``` [2020-09-03 09:13:14.236945] Hi, Alice! Nice to meet you. [2020-09-03 09:13:20.749831] Bye, Alice! It was great talking to you. @@ -61,9 +52,9 @@ Hello World creates a file `examples/hello_world/workspace/chats/chat_with_alice ## Modifying MLCommons-Box ### Adding new user -Create a new file `examples/hello_world/workspace/names/donald.txt` with the following content: `Donald`. +Create a new file `workspace/names/donald.txt` with the following content: `Donald`. -Create a new file `examples/hello_world/run/donald/hello.yaml` with the following content: +Create a new file `run/donald/hello.yaml` with the following content: ```yaml schema_type: mlbox_invoke schema_version: 1.0.0 @@ -77,7 +68,7 @@ output_binding: chat: $WORKSPACE/chats/chat_with_donald.txt ``` -Create a new file `examples/hello_world/run/donald/bye.yaml` with the following content: +Create a new file `run/donald/bye.yaml` with the following content: ```yaml schema_type: mlbox_invoke schema_version: 1.0.0 @@ -93,10 +84,10 @@ output_binding: Run the following two commands one at a time: ``` -python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/donald/hello.yaml -python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/donald/bye.yaml +mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/donald/hello.yaml +mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/donald/bye.yaml ``` -The Hello World box creates a file `examples/hello_world/workspace/chats/chat_with_donald.txt` that contains the +The Hello World box creates a file `workspace/chats/chat_with_donald.txt` that contains the following: ``` [2020-09-03 09:23:09.569558] Hi, Donald! Nice to meet you. @@ -109,7 +100,7 @@ Because how Hello World box was implemented, the greeting message is always the update the implementation so that if this is not the first time Alice says `hello`, the MLBox will respond: `Nice to see you again.`. -Modify the file `examples/hello_world/build/hello_world.py`. Update the function named `get_greeting_message` on line +Modify the file `build/hello_world.py`. Update the function named `get_greeting_message` on line 14. It should have the following implementation: ```python def get_greeting_message(chat_file: str) -> str: @@ -118,11 +109,11 @@ def get_greeting_message(chat_file: str) -> str: Since we updated a file in the `build` subdirectory, we need to re-configure the Hello World box: ``` -python -m mlcommons_box_docker configure --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml +mlcommons_box_docker configure --mlbox=. --platform=platforms/docker.yaml ``` Now, run two `hello` task again: ``` -python -m mlcommons_box_docker run --mlbox=examples/hello_world --platform=examples/hello_world/platforms/docker.yaml --task=examples/hello_world/run/alice/hello.yaml +mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/alice/hello.yaml ``` The MLBox recognized it was not the first time it talked to Alice, and changed the greeting: ``` diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md index bad55622..8a843e09 100644 --- a/docs/getting-started/index.md +++ b/docs/getting-started/index.md @@ -1 +1,33 @@ # Getting Started + +This section describes some of the example MLCommon-Box boxes, in particular it covers the following topics: + +- Setting up python environment. +- Running simple MLCommon-Box boxes. +- Detailed description of the internal structure of MLCommons-Box boxes. + + +## Repository of MLCommons-Box examples. +MLCommons hosts a simple GitHub-based repository with example MLCommons-Box boxes. It is located [here](https://github.com/mlperf/box_examples). + + +## Setting-up python environment +In various tutorials we start with setting up Python environment and downloading MLCommons-Box boxes. Here is the step by step guide: +``` +# Clone MLCommons-Box Examples +git clone https://github.com/mlperf/box_examples.git && cd ./box_examples + +# Create Python Virtual Environment +virtualenv -p python3 ./env && source ./env/bin/activate + +# Install MLCommons-Box Docker runner +pip install mlcommons-box-docker + +# Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables. +# export http_proxy=... +# export https_proxy=.. + +# Optionally, install other runners +# pip install mlcommons-box-singularity +# pip install mlcommons-box-ssh +``` diff --git a/docs/getting-started/mnist.md b/docs/getting-started/mnist.md index 14b1008b..221e51ca 100644 --- a/docs/getting-started/mnist.md +++ b/docs/getting-started/mnist.md @@ -300,56 +300,50 @@ mnist/ # MLBox root directory. ## Running MNIST MLCommons-Box -This tutorial covers the case when both MLCommons-Box library and the MNIST box are cloned from the GitHub repository: -``` -git clone https://github.com/mlperf/mlbox ./mlbox -cd ./mlbox +We need to setup the Python virtual environment. These are the steps outlined in the `Introduction` section except we do +not clone GitHub repository with the example MLCommons-Box boxes. ``` +# Create Python Virtual Environment +virtualenv -p python3 ./env && source ./env/bin/activate -Python >= 3.6 is required together with runners' python dependencies: -``` -virtualenv -p python3.8 ./env -source ./env/bin/activate -pip install typer mlspeclib -export PYTHONPATH=$(pwd)/mlcommons_box:$(pwd)/runners/mlcommons_box_singularity:$(pwd)/runners/mlcommons_box_docker:$(pwd)/runners/mlcommons_box_ssh -``` +# Install MLCommons-Box Docker and Singularity runners +pip install mlcommons-box-docker mlcommons-box-singularity -Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables. -``` -export http_proxy=... -export https_proxy=... -``` +# Optionally, setup host environment by providing the correct `http_proxy` and `https_proxy` environmental variables. +# export http_proxy=... +# export https_proxy=.. +``` > Before running MNIST box below, it is probably a good idea to remove tasks' outputs from previous runs that are -> located in `examples/mnist/workspace`. All directories except `parameters` can be removed. +> located in the `workspace` directory. All directories except `parameters` can be removed. ### Docker Runner Configure MNIST box: ``` -python -m mlcommons_box_docker configure --mlbox=examples/mnist --platform=examples/mnist/platforms/docker.yaml +mlcommons_box_docker configure --mlbox=. --platform=platforms/docker.yaml ``` Run two tasks - `download` (download data) and `train` (train tiny neural network): ``` -python -m mlcommons_box_docker run --mlbox=examples/mnist --platform=examples/mnist/platforms/docker.yaml --task=examples/mnist/run/download.yaml -python -m mlcommons_box_docker run --mlbox=examples/mnist --platform=examples/mnist/platforms/docker.yaml --task=examples/mnist/run/train.yaml +mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/download.yaml +mlcommons_box_docker run --mlbox=. --platform=platforms/docker.yaml --task=run/train.yaml ``` ### Singularity Runner -Update path to store Singularity image. Open `examples/mnist/platforms/singularity.yaml` and update the `image` value +Update path to store Singularity image. Open `platforms/singularity.yaml` and update the `image` value that is set by default to `/opt/singularity/mlperf_mlbox_mnist-0.01.simg` (relative paths are supported, they are -relative to `examples/mnist/workspace`). +relative to `workspace`). Configure MNIST box: ``` -python -m mlcommons_box_singularity configure --mlbox=examples/mnist --platform=examples/mnist/platforms/singularity.yaml +mlcommons_box_singularity configure --mlbox=. --platform=platforms/singularity.yaml ``` Run two tasks - `download` (download data) and `train` (train tiny neural network): ``` -python -m mlcommons_box_singularity run --mlbox=examples/mnist --platform=examples/mnist/platforms/singularity.yaml --task=examples/mnist/run/download.yaml -python -m mlcommons_box_singularity run --mlbox=examples/mnist --platform=examples/mnist/platforms/singularity.yaml --task=examples/mnist/run/train.yaml +mlcommons_box_singularity run --mlbox=. --platform=platforms/singularity.yaml --task=run/download.yaml +mlcommons_box_singularity run --mlbox=. --platform=platforms/singularity.yaml --task=run/train.yaml ``` diff --git a/docs/runners/docker-runner.md b/docs/runners/docker-runner.md index 324040d6..51247edd 100644 --- a/docs/runners/docker-runner.md +++ b/docs/runners/docker-runner.md @@ -21,13 +21,13 @@ configure and run phases: - __run__: `docker run ... -e http_proxy=${http_proxy} -e https_proxy=${https_proxy} ...` -## Build command +## Configuring MLBoxes Docker runner uses `{MLCOMMONS_BOX_ROOT}/build` directory as the build context directory. This implies that all files that must be packaged in a docker image, must be located in that directory, including source files, python requirements, resource files, ML models etc. The docker recipe must have the standard name `Dockerfile`. -In current implementation, only docker `build` is supported (i.e., Dockerfile must present). In future releases, Docker -runner will support docker `pull` as well. +If `Dockerfile` file exists in `{MLCOMMONS_BOX_ROOT}/build`, the Docker runner assumes that it needs to `build` a docker +image. If that file does not exists, the Docker runner will try to `pull` image with the specified name. Docker runner under the hood runs the following command line: ``` @@ -39,8 +39,15 @@ where: supported. - `{image_name}` is the image name defined in the platform configuration file. +> The `configure` command is optional and users do not necessarily need to be aware about it. The Docker runner +> auto-detects if docker image exists before running a task, and if it does not exist, the docker runner runs the +> `configure` command. During the `configure` phase, docker runner does not check if docker image exists. This means the +> following. If some of the implementation files have been modified, to rebuild the docker image users need to run +> the `configure` command explicitly. -## Run command + + +## Running MLBoxes Docker runner runs the following command: ``` {docker_runtime} run --rm --net=host --privileged=true {volumes} {env_args} {image_name} {args} diff --git a/mkdocs.yml b/mkdocs.yml index 0f3687aa..565f9d9b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -6,6 +6,7 @@ docs_dir: 'docs/' nav: - Home: index.md - Getting Started: + - Introduction: getting-started/index.md - Hello World: getting-started/hello-world.md - MNIST: getting-started/mnist.md - Runners: