From 7d75ee44e8711717821ef05d0612e09794feee75 Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Sun, 7 Jul 2024 19:20:56 +0200 Subject: [PATCH] minor docs clean up --- CONTRIBUTING.md | 10 +++--- README.md | 21 ++++++------ cm/CHANGES.md | 3 ++ cm/README.md | 78 +++++++++++++++++++++++++++----------------- cm/cmind/__init__.py | 2 +- cm4mlperf/README.md | 8 +++++ docs/installation.md | 1 + 7 files changed, 76 insertions(+), 47 deletions(-) create mode 100644 cm4mlperf/README.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c9fb4c153..28293626e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,11 +14,11 @@ Modify the project in your own fork and issue a pull request once you want other to take a look at what you have done and discuss the proposed changes. Ensure that cla-bot and other checks pass for your Pull requests. -## Original author (CK, CM, CM4MLOPS, CM4ABTF, CM4MLPERF) - -* Grigori Fursin ([cKnowledge.org](https://cKnowledge.org) and [cTuning.org](https://cTuning.org)) - -## CM contributors in alphabetical order (suggestions, feedback, scripts) +Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin), +sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. +Since then, this open-source technology (CM, CM4MLOps, CM4ABTF, CM4Research, etc) +is being developed as a community effort thanks to all our volunteers, collaborators and contributors +listed here in alphabetical order: * Resmi Arjun * Alice Cheng (Nvidia) diff --git a/README.md b/README.md index b273ed501..076379901 100755 --- a/README.md +++ b/README.md @@ -65,10 +65,6 @@ CK consists of several ongoing sub-projects: [Apache 2.0](LICENSE.md) -### Citing this project - -Please use this [BibTex file](https://github.com/mlcommons/ck/blob/master/citation.bib). - ### Documentation **MLCommons is updating the CM documentation based on user feedback - please stay tuned for more details**. @@ -86,15 +82,18 @@ Please use this [BibTex file](https://github.com/mlcommons/ck/blob/master/citati ### Acknowledgments -This open-source technology was originally developed by [Grigori Fursin](https://cKnowledge.org/gfursin) -and donated to MLCommons to benefit everyone. You can learn more about the motivation behind these projects from the following articles and presentations: +Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin), +sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. +Since then, this open-source technology (CM, CM4MLOps, CM4ABTF, CM4Research, etc) +is being developed as a community effort thanks to all our +[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! + +You can learn more about the motivation behind these projects from the following articles and presentations: * "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] * ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] -We would like to thank all our great -[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) -for their support, fruitful discussions, and useful feedback! -We thank the [cTuning foundation](https://cTuning.org), [cKnowledge.org](https://cKnowledge.org) -and [MLCommons](https://mlcommons.org) for sponsoring this project! +### Citing this project + +Please use this [BibTex file](https://github.com/mlcommons/ck/blob/master/citation.bib). diff --git a/cm/CHANGES.md b/cm/CHANGES.md index f75df4a58..02200b517 100644 --- a/cm/CHANGES.md +++ b/cm/CHANGES.md @@ -1,3 +1,6 @@ +## V2.3.3 + - minor documentation update for MLPerf inference v4.1 + ## V2.3.2 - fixed "cm pull repo --branch={BRANCH NAME}" behavior for all OS - added GitHub tests for Windows diff --git a/cm/README.md b/cm/README.md index 6a7d94fea..d13c18671 100644 --- a/cm/README.md +++ b/cm/README.md @@ -3,6 +3,7 @@ [![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md) [![Downloads](https://static.pepy.tech/badge/cmind)](https://pepy.tech/project/cmind) +[![arXiv](https://img.shields.io/badge/arXiv-2406.16791-b31b1b.svg)](https://arxiv.org/abs/2406.16791) [![CM test](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml) [![CM script automation features test](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml) @@ -11,15 +12,14 @@ Collective Mind (CM) is a collection of portable, extensible, technology-agnostic and ready-to-use automation recipes with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on any platform with any software and hardware: see [CM4MLOps online catalog](https://access.cknowledge.org/playground/?action=scripts), -[source code](https://github.com/mlcommons/ck/blob/master/cm-mlops/script), [ArXiv project article](https://arxiv.org/abs/2406.16791). +[source code](https://github.com/mlcommons/ck/blob/master/cm-mlops/script), [ArXiv white paper]( https://arxiv.org/abs/2406.16791 ). CM scripts require Python 3.7+ with minimal dependencies and are [continuously extended by the community and MLCommons members](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux and any other operating system, in a cloud or inside automatically generated containers while keeping backward compatibility - please don't hesitate -to report encountered issues [here](https://github.com/mlcommons/ck/issues) -and contact us via [public Discord Server](https://discord.gg/JjWNWXKxwT) +to report encountered issues [here](https://github.com/mlcommons/ck/issues) to help this collaborative engineering effort! CM scripts were originally developed based on the following requirements from the @@ -47,9 +47,11 @@ using a few CM commands: ```bash -pip install cm4mlperf -U +pip install cmind -U + +cm pull repo cknowledge@cm4mlops --branch=mlperf-inference -cm run script "run-mlperf-inference _r4.0 _accuracy-only _short" \ +cm run script "run-mlperf-inference _r4.1 _accuracy-only _short" \ --device=cpu \ --model=resnet50 \ --precision=float32 \ @@ -60,7 +62,7 @@ cm run script "run-mlperf-inference _r4.0 _accuracy-only _short" \ --quiet \ --time -cm run script "run-mlperf-inference _r4.0 _submission _short" \ +cm run script "run-mlperf-inference _r4.1 _submission _short" \ --device=cpu \ --model=resnet50 \ --precision=float32 \ @@ -73,16 +75,36 @@ cm run script "run-mlperf-inference _r4.0 _submission _short" \ ... -+----------+----------+----------+--------+-----------------+---------------------------------+ -| Model | Scenario | Accuracy | QPS | Latency (in ms) | Power Efficiency (in samples/J) | -+----------+----------+----------+--------+-----------------+---------------------------------+ -| resnet50 | Offline | 80.0 | 27.371 | - | | -+----------+----------+----------+--------+-----------------+---------------------------------+ + 0 +Organization CTuning +Availability available +Division open +SystemType edge +SystemName ip_172_31_87_92 +Platform ip_172_31_87_92-reference-cpu-onnxruntime-v1.1... +Model resnet50 +MlperfModel resnet +Scenario Offline +Result 14.3978 +Accuracy 80.0 +number_of_nodes 1 +host_processor_model_name Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz +host_processors_per_node 1 +host_processor_core_count 2 +accelerator_model_name NaN +accelerators_per_node 0 +Location open/CTuning/results/ip_172_31_87_92-reference... +framework onnxruntime v1.18.1 +operating_system Ubuntu 24.04 (linux-6.8.0-1009-aws-glibc2.39) +notes Automated by MLCommons CM v2.3.2. +compliance 1 +errors 0 +version v4.1 +inferred 0 +has_power False +Units Samples/s -The MLPerf inference results are stored at /home/gfursin/CM/repos/local/cache/a504fb143e97452f/test_results -Path to the MLPerf inference benchmark reference sources: /home/gfursin/CM/repos/local/cache/8061c243b8b54a3b/inference -Path to the MLPerf inference reference configuration file: /home/gfursin/CM/repos/local/cache/8061c243b8b54a3b/inference/mlperf.conf ``` You can also run the same commands using a unified CM Python API: @@ -91,7 +113,7 @@ You can also run the same commands using a unified CM Python API: import cmind output=cmind.access({ 'action':'run', 'automation':'script', - 'tags':'run-mlperf-inference,_r4.0,_performance-only,_short', + 'tags':'run-mlperf-inference,_r4.1,_performance-only,_short', 'device':'cpu', 'model':'resnet50', 'precision':'float32', @@ -117,7 +139,7 @@ See more examples of CM scripts and workflows to download Stable Diffusion, GPT- ```bash pip install cmind -U -cm pull repo mlcommons@cm4mlops --branch=dev +cm pull repo mlcommons@cm4mlops --branch=mlperf-inference cm show repo @@ -216,24 +238,20 @@ and how to implement and share new automations in your public or private project [Apache 2.0](LICENSE.md) -### Citing this project - -Please use this [BibTex file](https://github.com/mlcommons/ck/blob/master/citation.bib). - ### Acknowledgments -[Collective Knowledge automation framework (deprecated CK v1 and v2)](https://github.com/mlcommons/ck/tree/master/ck), -[Collective Mind automation framework (CM)](https://github.com/mlcommons/ck/tree/master/cm), -[CM4MLOPS](https://github.com/mlcommons/cm4mlops) and -[CM4ABTF](https://github.com/mlcommons/cm4abtf) were originally developed by [Grigori Fursin](https://cKnowledge.org/gfursin) -and donated to MLCommons to benefit everyone. You can learn more about the motivation behind these projects from the following articles and presentations: +Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin), +sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. +Since then, this open-source technology (CM, CM4MLOps, CM4ABTF, CM4Research, etc) +is being developed as a community effort thanks to all our +[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! + +You can learn more about the motivation behind these projects from the following articles and presentations: * "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] * ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] -We would like to thank all our great -[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) -for their support, fruitful discussions, and useful feedback! -We thank the [cTuning foundation](https://cTuning.org), [cKnowledge.org](https://cKnowledge.org) -and [MLCommons](https://mlcommons.org) for sponsoring this project! +### Citing this project + +Please use this [BibTex file](https://github.com/mlcommons/ck/blob/master/citation.bib). diff --git a/cm/cmind/__init__.py b/cm/cmind/__init__.py index 62a14f051..d4170cd14 100644 --- a/cm/cmind/__init__.py +++ b/cm/cmind/__init__.py @@ -2,7 +2,7 @@ # # Written by Grigori Fursin -__version__ = "2.3.2" +__version__ = "2.3.3" from cmind.core import access from cmind.core import error diff --git a/cm4mlperf/README.md b/cm4mlperf/README.md new file mode 100644 index 000000000..03aa82a1a --- /dev/null +++ b/cm4mlperf/README.md @@ -0,0 +1,8 @@ +# CM4MLPerf: CM automation for MLPerf benchmarks + +Unified [CM interface](https://arxiv.org/abs/2406.16791) +with portable automations for MLPerf benchmarks based on [CM4MLOps](https://github.com/mlcommons/cm4mlops). + +# Stable versions + +TBD diff --git a/docs/installation.md b/docs/installation.md index f6457128b..c27db8c72 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -55,6 +55,7 @@ You can reuse misc CM utils listed [here](#misc-cm-utils). sudo apt update && sudo apt upgrade sudo apt install python3 python3-pip python3-venv git wget curl +sudo apt install libgl1-mesa-dev ``` **Note that you must set up virtual env on Ubuntu 23+ before using any Python project:**