Skip to content

Commit

Permalink
Results from self hosted Github actions - NVIDIARTX4090
Browse files Browse the repository at this point in the history
  • Loading branch information
arjunsuresh committed Nov 28, 2024
1 parent d54b5a1 commit 4c7a622
Show file tree
Hide file tree
Showing 35 changed files with 10,493 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
|---------------------|------------|----------------------|--------------|-------------------|
| stable-diffusion-xl | offline | (16.3689, 237.82579) | 0.384 | - |
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
This experiment is generated using the [MLCommons Collective Mind automation framework (CM)](https://github.com/mlcommons/cm4mlops).

*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*

## Host platform

* OS version: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
* CPU version: x86_64
* Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]
* MLCommons CM version: 3.3.4

## CM Run Command

See [CM installation guide](https://docs.mlcommons.org/inference/install/).

```bash
pip install -U cmind

cm rm cache -f

cm pull repo gateoverflow@cm4mlops --checkout=6a09ef4596d537e45b4a4772112284fca73347af

cm run script \
--tags=app,mlperf,inference,generic,_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline \
--quiet=true \
--env.CM_MLPERF_MODEL_SDXL_DOWNLOAD_TO_HOST=yes \
--env.CM_QUIET=yes \
--env.CM_MLPERF_IMPLEMENTATION=reference \
--env.CM_MLPERF_MODEL=sdxl \
--env.CM_MLPERF_RUN_STYLE=test \
--env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=False \
--env.CM_DOCKER_PRIVILEGED_MODE=True \
--env.CM_MLPERF_BACKEND=pytorch \
--env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=datacenter \
--env.CM_MLPERF_CLEAN_ALL=True \
--env.CM_MLPERF_DEVICE=cuda \
--env.CM_MLPERF_USE_DOCKER=True \
--env.CM_MLPERF_MODEL_PRECISION=float16 \
--env.OUTPUT_BASE_DIR=/home/arjun/scc_gh_action_results \
--env.CM_MLPERF_LOADGEN_SCENARIO=Offline \
--env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/home/arjun/scc_gh_action_submissions \
--env.CM_MLPERF_INFERENCE_VERSION=4.1-dev \
--env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1-dev_default \
--env.CM_MLPERF_SUBMISSION_DIVISION=open \
--env.CM_RUN_MLPERF_SUBMISSION_PREPROCESSOR=False \
--env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=short \
--env.CM_MLPERF_SUT_NAME_RUN_CONFIG_SUFFIX4=scc24-base \
--env.CM_DOCKER_IMAGE_NAME=scc24-reference \
--env.CM_MLPERF_INFERENCE_MIN_QUERY_COUNT=50 \
--env.CM_MLPERF_LOADGEN_ALL_MODES=yes \
--env.CM_MLPERF_INFERENCE_SOURCE_VERSION=4.1.23 \
--env.CM_MLPERF_LAST_RELEASE=v4.1 \
--env.CM_TMP_CURRENT_PATH=/home/arjun/actions-runner/_work/cm4mlops/cm4mlops \
--env.CM_TMP_PIP_VERSION_STRING= \
--env.CM_MODEL=sdxl \
--env.CM_MLPERF_LOADGEN_COMPLIANCE=no \
--env.CM_MLPERF_CLEAN_SUBMISSION_DIR=yes \
--env.CM_RERUN=yes \
--env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= \
--env.CM_MLPERF_LOADGEN_MODE=performance \
--env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline \
--env.CM_MLPERF_LOADGEN_MODES,=performance,accuracy \
--env.CM_OUTPUT_FOLDER_NAME=test_results \
--env.CM_DOCKER_REUSE_EXISTING_CONTAINER=no \
--env.CM_DOCKER_DETACHED_MODE=yes \
--add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r4_1-dev \
--add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r4_1-dev \
--add_deps_recursive.mlperf-inference-nvidia-scratch-space.tags=_version.r4_1-dev \
--add_deps_recursive.submission-checker.tags=_short-run \
--add_deps_recursive.coco2014-preprocessed.tags=_size.50,_with-sample-ids \
--add_deps_recursive.coco2014-dataset.tags=_size.50,_with-sample-ids \
--add_deps_recursive.nvidia-preprocess-data.extra_cache_tags=scc24-base \
--v=False \
--print_env=False \
--print_deps=False \
--dump_version_info=True \
--env.OUTPUT_BASE_DIR=/cm-mount/home/arjun/scc_gh_action_results \
--env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/cm-mount/home/arjun/scc_gh_action_submissions \
--env.SDXL_CHECKPOINT_PATH=/home/cmuser/CM/repos/local/cache/6be1f30ecbde4c4e/stable_diffusion_fp16
```
*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
you should simply reload gateoverflow@cm4mlops without checkout and clean CM cache as follows:*

```bash
cm rm repo gateoverflow@cm4mlops
cm pull repo gateoverflow@cm4mlops
cm rm cache -f

```

## Results

Platform: ca3caa61abb5-reference-gpu-pytorch-v2.5.1-scc24-base_cu124

Model Precision: fp32

### Accuracy Results
`CLIP_SCORE`: `16.3689`, Required accuracy for closed division `>= 31.68632` and `<= 31.81332`
`FID_SCORE`: `237.82579`, Required accuracy for closed division `>= 23.01086` and `<= 23.95008`

### Performance Results
`Samples per second`: `0.383978`

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"starting_weights_filename": "https://github.com/mlcommons/inference/tree/master/text_to_image#download-model",
"retraining": "no",
"input_data_types": "fp32",
"weight_data_types": "fp32",
"weight_transformations": "no"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
graph TD
app-mlperf-inference,d775cac873ee4231_(_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline_) --> detect,os
app-mlperf-inference,d775cac873ee4231_(_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline_) --> get,sys-utils-cm
app-mlperf-inference,d775cac873ee4231_(_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline_) --> get,python
get-mlperf-inference-src,4b57186581024797 --> detect,os
get-mlperf-inference-src,4b57186581024797 --> get,python3
get-git-repo,ed603e7292974f10_(_branch.master,_repo.https://github.com/mlcommons/inference_) --> detect,os
get-mlperf-inference-src,4b57186581024797 --> get,git,repo,_branch.master,_repo.https://github.com/mlcommons/inference
app-mlperf-inference,d775cac873ee4231_(_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline_) --> get,mlcommons,inference,src
get-mlperf-inference-utils,e341e5f86d8342e5 --> get,mlperf,inference,src
app-mlperf-inference,d775cac873ee4231_(_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline_) --> get,mlperf,inference,utils
get-cuda,46d133d9ef92422d_(_toolkit_) --> detect,os
get-cuda-devices,7a3ede4d3558427a_(_with-pycuda_) --> get,cuda,_toolkit
get-cuda-devices,7a3ede4d3558427a_(_with-pycuda_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.pycuda_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.pycuda_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.pycuda_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_pip_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.pycuda_) --> get,generic-python-lib,_pip
get-cuda-devices,7a3ede4d3558427a_(_with-pycuda_) --> get,generic-python-lib,_package.pycuda
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> get,generic-python-lib,_pip
get-cuda-devices,7a3ede4d3558427a_(_with-pycuda_) --> get,generic-python-lib,_package.numpy
app-mlperf-inference,d775cac873ee4231_(_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline_) --> get,cuda-devices,_with-pycuda
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> detect,cpu
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,sys-utils-cm
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,python
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,cuda,_cudnn
get-generic-python-lib,94b62a682bc44791_(_torch_cuda_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_torch_cuda_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_torch_cuda_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_torch_cuda_) --> get,generic-python-lib,_pip
get-generic-python-lib,94b62a682bc44791_(_torch_cuda_) --> get,cuda
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_torch_cuda
get-generic-python-lib,94b62a682bc44791_(_torchvision_cuda_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_torchvision_cuda_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_torchvision_cuda_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_torchvision_cuda_) --> get,generic-python-lib,_pip
get-generic-python-lib,94b62a682bc44791_(_torchvision_cuda_) --> get,cuda
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_torchvision_cuda
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,ml-model,stable-diffusion,text-to-image,sdxl,raw,_pytorch,_fp16
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,dataset,coco2014,_validation,_size.50,_with-sample-ids
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,cpu
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,python
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,mlcommons,inference,src
get-mlperf-inference-sut-configs,c2fbf72009e2445b --> get,cache,dir,_name.mlperf-inference-sut-configs
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,sut,configs
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> generate,user-conf,mlperf,inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,loadgen
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,mlcommons,inference,src
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.psutil
get-generic-python-lib,94b62a682bc44791_(_package.diffusers_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.diffusers_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.diffusers_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.diffusers_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.diffusers
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.transformers
get-generic-python-lib,94b62a682bc44791_(_package.torchvision_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.torchvision_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.torchvision_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.torchvision_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.torchvision
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.accelerate
get-generic-python-lib,94b62a682bc44791_(_package.torchmetrics_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.torchmetrics_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.torchmetrics_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.torchmetrics_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.torchmetrics
get-generic-python-lib,94b62a682bc44791_(_package.torch-fidelity_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.torch-fidelity_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.torch-fidelity_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.torch-fidelity_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.torch-fidelity
get-generic-python-lib,94b62a682bc44791_(_package.open_clip_torch_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.open_clip_torch_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.open_clip_torch_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.open_clip_torch_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.open_clip_torch
get-generic-python-lib,94b62a682bc44791_(_package.opencv-python_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.opencv-python_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.opencv-python_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.opencv-python_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.opencv-python
get-generic-python-lib,94b62a682bc44791_(_package.scipy_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
get-generic-python-lib,94b62a682bc44791_(_package.scipy_) --> detect,cpu
get-generic-python-lib,94b62a682bc44791_(_package.scipy_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_pip_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.scipy_) --> get,generic-python-lib,_pip
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> get,generic-python-lib,_package.scipy
detect-cpu,586c8a43320142f7 --> detect,os
benchmark-program,19f369ef47084895 --> detect,cpu
benchmark-program-mlperf,cfff0132a8aa4018 --> benchmark-program,program
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_offline,_sdxl,_pytorch,_float16_) --> benchmark-mlperf
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 4c7a622

Please sign in to comment.