Skip to content

Commit

Permalink
Merge pull request #176 from mlcommons/dev
Browse files Browse the repository at this point in the history 
Sync Dev
  • Loading branch information
@anandhu-eng
anandhu-eng authored Feb 3, 2025
2 parents def3863 + 6a91792 commit 2ac0b7b
Show file tree
Hide file tree
Showing 298 changed files with 122 additions and 33,640 deletions.
5 changes: 1 addition & 4 deletions .github/workflows/test-mlperf-inference-mixtral.yml
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: MLPerf inference MIXTRAL-8x7B

on:
schedule:
- cron: "59 19 * * *" # 30th minute and 20th hour => 20:30 UTC => 2 AM IST
- cron: "59 23 * * */5" # 30th minute and 20th hour => 20:30 UTC => 2 AM IST

jobs:
build_reference:
Expand Down
7 changes: 6 additions & 1 deletion .github/workflows/test-nvidia-mlperf-inference-implementations.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ jobs:
gpu_name=rtx_4090
docker_string=" --docker"
fi
if [ "${{ matrix.model }}" = "bert-99" ] || [ "${{ matrix.model }}" = "bert-99.9" ]; then
category="edge"
else
category="datacenter,edge"
fi
if [ -f "gh_action/bin/deactivate" ]; then source gh_action/bin/deactivate; fi
python3 -m venv gh_action
Expand All @@ -51,6 +56,6 @@ jobs:
pip install --upgrade mlcflow
mlc pull repo mlcommons@mlperf-automations --branch=dev
mlcr --tags=run-mlperf,inference,_all-scenarios,_submission,_full,_r5.0-dev --preprocess_submission=yes --pull_changes=yes --pull_inference_changes=yes --execution_mode=valid --gpu_name=$gpu_name --pull_changes=yes --pull_inference_changes=yes --model=${{ matrix.model }} --submitter="MLCommons" --hw_name=$hw_name --implementation=nvidia --backend=tensorrt --category=datacenter,edge --division=closed --docker_dt --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --adr.compiler.tags=gcc --device=cuda --use_model_from_host=yes --use_dataset_from_host=yes --results_dir=$HOME/gh_action_results --submission_dir=$HOME/gh_action_submissions --clean $docker_string --quiet
mlcr --tags=run-mlperf,inference,_all-scenarios,_submission,_full,_r5.0-dev --preprocess_submission=yes --pull_changes=yes --pull_inference_changes=yes --execution_mode=valid --gpu_name=$gpu_name --pull_changes=yes --pull_inference_changes=yes --model=${{ matrix.model }} --submitter="MLCommons" --hw_name=$hw_name --implementation=nvidia --backend=tensorrt --category=$category --division=closed --docker_dt --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --adr.compiler.tags=gcc --device=cuda --use_model_from_host=yes --use_dataset_from_host=yes --results_dir=$HOME/gh_action_results --submission_dir=$HOME/gh_action_submissions --clean $docker_string --quiet
mlcr --tags=push,github,mlperf,inference,submission --repo_url=https://github.com/mlcommons/mlperf_inference_unofficial_submissions_v5.0 --repo_branch=auto-update --commit_message="Results from GH action on NVIDIA_$hw_name" --quiet --submission_dir=$HOME/gh_action_submissions --hw_name=$hw_name
46 changes: 0 additions & 46 deletions docs/cm-yaml-guide.md
View file

This file was deleted.

179 changes: 73 additions & 106 deletions docs/getting-started.md
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,71 @@
# Getting Started with MLC Script Automation

# Getting Started with CM Script Automation
## Running MLC Scripts

## Running CM Scripts

To execute a simple script in CM that captures OS details, use the following command:
To execute a simple script in MLC that captures OS details, use the following command:

```bash
cm run script --tags=detect,os -j
mlcr detect,os -j
```
* Here, `mlcr` is a shortform for `mlc run script --tags=`

This command gathers details about the system on which it's run, such as:

```json
{
"CM_HOST_OS_TYPE": "linux",
"CM_HOST_OS_BITS": "64",
"CM_HOST_OS_FLAVOR": "ubuntu",
"CM_HOST_OS_FLAVOR_LIKE": "debian",
"CM_HOST_OS_VERSION": "24.04",
"CM_HOST_OS_KERNEL_VERSION": "6.8.0-45-generic",
"CM_HOST_OS_GLIBC_VERSION": "2.39",
"CM_HOST_OS_MACHINE": "x86_64",
"CM_HOST_OS_PACKAGE_MANAGER": "apt",
"CM_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
"CM_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
"+CM_HOST_OS_DEFAULT_LIBRARY_PATH": [
$ mlcr detect,os -j
[2025-02-03 04:57:23,449 main.py:694 INFO] - Repos path for Index: /home/arjun/MLC/repos
[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for script saved to /home/arjun/MLC/repos/index_script.json.
[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for cache saved to /home/arjun/MLC/repos/index_cache.json.
[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for experiment saved to /home/arjun/MLC/repos/index_experiment.json.
[2025-02-03 04:57:24,210 module.py:574 INFO] - * mlcr detect,os
[2025-02-03 04:57:24,213 module.py:5354 INFO] - ! cd /mnt/arjun/MLC/repos/gateoverflow@mlperf-automations
[2025-02-03 04:57:24,213 module.py:5355 INFO] - ! call /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-03 04:57:24,245 module.py:5501 INFO] - ! call "postprocess" from /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os/customize.py
[2025-02-03 04:57:24,254 module.py:2195 INFO] - {
"return": 0,
"env": {
"MLC_HOST_OS_TYPE": "linux",
"MLC_HOST_OS_BITS": "64",
"MLC_HOST_OS_FLAVOR": "ubuntu",
"MLC_HOST_OS_FLAVOR_LIKE": "debian",
"MLC_HOST_OS_VERSION": "24.04",
"MLC_HOST_OS_KERNEL_VERSION": "6.8.0-52-generic",
"MLC_HOST_OS_GLIBC_VERSION": "2.39",
"MLC_HOST_OS_MACHINE": "x86_64",
"MLC_HOST_OS_PACKAGE_MANAGER": "apt",
"MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
"MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
"+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
"/usr/local/lib/x86_64-linux-gnu",
"/lib/x86_64-linux-gnu",
"/usr/lib/x86_64-linux-gnu",
"/usr/lib/x86_64-linux-gnu64",
"/usr/local/lib64",
"/lib64",
"/usr/lib64",
"/usr/local/lib",
"/lib",
"/usr/lib",
"/usr/x86_64-linux-gnu/lib64",
"/usr/x86_64-linux-gnu/lib"
],
"MLC_HOST_PLATFORM_FLAVOR": "x86_64",
"MLC_HOST_PYTHON_BITS": "64",
"MLC_HOST_SYSTEM_NAME": "arjun-spr"
},
"new_env": {
"MLC_HOST_OS_TYPE": "linux",
"MLC_HOST_OS_BITS": "64",
"MLC_HOST_OS_FLAVOR": "ubuntu",
"MLC_HOST_OS_FLAVOR_LIKE": "debian",
"MLC_HOST_OS_VERSION": "24.04",
"MLC_HOST_OS_KERNEL_VERSION": "6.8.0-52-generic",
"MLC_HOST_OS_GLIBC_VERSION": "2.39",
"MLC_HOST_OS_MACHINE": "x86_64",
"MLC_HOST_OS_PACKAGE_MANAGER": "apt",
"MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
"MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
"+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
"/usr/local/lib/x86_64-linux-gnu",
"/lib/x86_64-linux-gnu",
"/usr/lib/x86_64-linux-gnu",
Expand All @@ -38,98 +79,24 @@ This command gathers details about the system on which it's run, such as:
"/usr/x86_64-linux-gnu/lib64",
"/usr/x86_64-linux-gnu/lib"
],
"CM_HOST_PLATFORM_FLAVOR": "x86_64",
"CM_HOST_PYTHON_BITS": "64",
"CM_HOST_SYSTEM_NAME": "intel-spr-i9"
"MLC_HOST_PLATFORM_FLAVOR": "x86_64",
"MLC_HOST_PYTHON_BITS": "64",
"MLC_HOST_SYSTEM_NAME": "arjun-spr"
},
"state": {
"os_uname_machine": "x86_64",
"os_uname_all": "Linux arjun-spr 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux"
},
"new_state": {
"os_uname_machine": "x86_64",
"os_uname_all": "Linux arjun-spr 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux"
},
"deps": []
}
```

For more details on CM scripts, see the [CM documentation](index.md).

### Adding New CM Scripts
For more details on MLC scripts, see the [MLC documentation](index.md).

CM aims to provide lightweight connectors between existing automation scripts and tools without substituting them. You can add your own scripts to CM with the following command, which creates a script named `hello-world`:

```bash
cm add script hello-world --tags=hello-world,display,test
```

This command initializes a CM script in the local repository with the following structure:

```
└── CM
├── index.json
├── repos
│ ├── local
│ │ ├── cfg
│ │ ├── cache
│ │ ├── cmr.yaml
│ │ └── script
│ │ └── hello-world
│ │ ├── _cm.yaml
│ │ ├── customize.py
│ │ ├── README-extra.md
│ │ ├── run.bat
│ │ └── run.sh
│ └── mlcommons@cm4mlops
└── repos.json
```

You can also execute the script from Python as follows:

```python
import cmind
output = cmind.access({'action':'run', 'automation':'script', 'tags':'hello-world,display,test'})
if output['return'] == 0:
print(output)
```

If you discover that your new script is similar to an existing script in any CM repository, you can clone an existing script using the following command:

```bash
cm copy script <source_script> .:<target_script>
```

Here, `<source_script>` is the name of the existing script, and `<target_script>` is the name of the new script you're creating. Existing script names in the `cm4mlops` repository can be found [here](https://github.com/mlcommons/cm4mlops/tree/mlperf-inference/script).

## Caching and Reusing CM Script Outputs

By default, CM scripts run in the current directory and record all new files there. For example, a universal download script might download an image to the current directory:

```bash
cm run script --tags=download,file,_wget --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e
```

To cache and reuse the output of scripts, CM offers a `cache` automation feature similar to `script`. When `"cache":true` is specified in a script's metadata, CM will create a `cache` directory in `$HOME/CM/repos/local` with a unique ID and the same tags as `script`, and execute the script there.

Subsequent executions of the same script will reuse files from the cache, avoiding redundancy. This is especially useful for large files or data sets.

You can manage cache entries and find specific ones using commands like:

```bash
cm show cache
cm show cache --tags=get,ml-model,resnet50,_onnx
cm find cache --tags=download,file,ml-model,resnet50,_onnx
cm info cache --tags=download,file,ml-model,resnet50,_onnx
```

To clean cache entries:

```bash
cm rm cache --tags=ml-model,resnet50
cm rm cache -f # Clean all entries
```

You can completely reset the CM framework by removing the `$HOME/CM` directory, which deletes all downloaded repositories and cached entries.

## Integration with Containers

CM scripts are designed to run natively or inside containers with the same commands. You can substitute `cm run script` with `cm docker script` to execute a script inside an automatically-generated container:

```bash
cm docker script --tags=python,app,image-classification,onnx,_cpu
```

CM automatically handles the generation of Dockerfiles, building of containers, and execution within containers, providing a seamless experience whether running scripts natively or in containers.

This approach simplifies the development process by eliminating the need for separate Dockerfile maintenance and allows for the use of native scripts and workflows directly within containers.
Loading

0 comments on commit 2ac0b7b

Please sign in to comment.