Skip to content

Commit b607edb

Browse files
ctuning-adminctuning-admin
authored
CM/CMX v3.5.2 (#1377)
- added `-i` flag to print memory and disk use before running CM/CMX commands: #1375 - added utils.get_disk_use - added utils.get_memory_use - formatted Python modules from the internal repository using autopep8
2 parents 7f66e24 + 2dd17a6 commit b607edb

File tree

10 files changed

+236
-14
lines changed

10 files changed

+236
-14
lines changed

Diff for: ‎.github/workflows/test-cm.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
strategy:
1717
fail-fast: false
1818
matrix:
19-
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
19+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
2020
on: [ubuntu-latest, windows-latest, macos-latest]
2121
exclude:
2222
- python-version: "3.7"

Diff for: ‎HISTORY.CM.md

+91
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
This document narrates the history of the creation and design of CM and CM4MLOps (also known as CK2)
2+
by [Grigori Fursin](https://cKnowledge.org/gfursin). It also highlights the donation of this open-source technology to MLCommons,
3+
aimed at benefiting the broader community and fostering its ongoing development as a collaborative, community-driven initiative:
4+
5+
* Jan 28, 2021: After delivering an invited ACM TechTalk'21 about the Collective Knowledge framework (CK1)
6+
and reproducibility initiatives for conferences, as well as CK-MLOps and MLPerf automations,
7+
Grigori received lots of positive feedback and suggestions for improvements to workflow automations:
8+
https://learning.acm.org/techtalks/reproducibility.
9+
10+
Following this, Grigori began prototyping CK2 (later CM) to streamline CK1, CK-MLOps and MLPerf benchmarking.
11+
The goal was to dramatically simplify CK1 workflows by introducing just a few core and portable automations,
12+
which eventually evolved into `CM script` and `CM cache`.
13+
14+
At that time, the cTuning foundation hosted CK1 and all the prototypes for the CM framework at https://github.com/ctuning/ck:
15+
[ref1](https://github.com/mlcommons/ck/commit/9e57934f4999db23052531e92160772ab831463a),
16+
[ref2](https://github.com/mlcommons/ck/tree/9e57934f4999db23052531e92160772ab831463a),
17+
[ref3](https://github.com/mlcommons/ck/tree/9e57934f4999db23052531e92160772ab831463a/incubator).
18+
19+
* Sep 23, 2021: donated CK1, CK-MLOps, MLPerf automations and early prototypes of CM from the cTuning repository to MLCommons:
20+
[ref1](https://web.archive.org/web/20240803140223/https://octo.ai/blog/octoml-joins-the-community-effort-to-democratize-mlperf-inference-benchmarking),
21+
[ref2](https://github.com/mlcommons/ck/tree/228f80b0bf44610c8244ff0c3f6bec5bbd25aa6c/incubator),
22+
[ref3](https://github.com/mlcommons/ck/tree/695c3843fd8121bbdde6c453cd6ec9503986b0c6?tab=readme-ov-file#author-and-coordinator),
23+
[ref4](https://github.com/mlcommons/ck/tree/master/ck),
24+
[ref5](https://github.com/mlcommons/ck-mlops).
25+
26+
* Mar 1, 2022: started developing cm-mlops: [ref](https://github.com/octoml/cm-mlops/commit/0ae94736a420dfa84f7417fc62d323303b8760c6).
27+
28+
* Mar 24, 2022: after successfully stabilizing the initial prototype of CM, donated it to MLCommons to benefit the entire community:
29+
[ref1](https://github.com/mlcommons/ck/tree/c7918ad544f26b6c499c2fc9c07431a9640fca5a/ck2),
30+
[ref2](https://github.com/mlcommons/ck/tree/c7918ad544f26b6c499c2fc9c07431a9640fca5a/ck2#coordinators),
31+
[ref3](https://github.com/mlcommons/ck/commit/3c146cb3c75a015363f7a96758adf6dcc43032d6),
32+
[ref4](https://github.com/mlcommons/ck/commit/3c146cb3c75a015363f7a96758adf6dcc43032d6#diff-d97f0f6f5a32f16d6ed18b9600ffc650f7b25512685f7a2373436c492c6b52b3R48).
33+
34+
* Apr 6, 2022: started transitioning previous MLOps and MLPerf automations from the mlcommons/ck-mlops format
35+
to the new CM format using the cm-mlops repository (will be later renamed to cm4mlops):
36+
[ref1](https://github.com/octoml/cm-mlops/commit/d1efdc30fb535ce144020d4e88f3ed768c933176),
37+
[ref2](https://github.com/octoml/cm-mlops/blob/d1efdc30fb535ce144020d4e88f3ed768c933176/CONTRIBUTIONS).
38+
39+
* Apr 22, 2022: began architecting "Intelligent Components" in the CM-MLOps repository,
40+
which will be renamed to `CM Script` at a later stage:
41+
[ref1](https://github.com/octoml/cm-mlops/commit/b335c609c47d2c547afe174d9df232652d57f4f8),
42+
[ref2](https://github.com/octoml/cm-mlops/tree/b335c609c47d2c547afe174d9df232652d57f4f8),
43+
[ref3](https://github.com/octoml/cm-mlops/blob/b335c609c47d2c547afe174d9df232652d57f4f8/CONTRIBUTIONS).
44+
45+
At the same time, prototyped other core CM automations, including IC, Docker, and Experiment:
46+
[ref1](https://github.com/octoml/cm-mlops/tree/b335c609c47d2c547afe174d9df232652d57f4f8/automation),
47+
[ref2](https://github.com/mlcommons/ck/commits/master/?before=7f66e2438bfe21b4ce2d08326a5168bb9e3132f6+7001).
48+
49+
* Apr 28, 2022: donated CM-MLOps to MLCommons, which was later renamed to CM4MLOps:
50+
[ref](https://github.com/mlcommons/ck/commit/456e4861056c0e39c4d689c03da91f90a44be058).
51+
52+
* May 9, 2022: developed the initial set of core IC automations for MLOps (aka CM scripts):
53+
[ref1](https://github.com/octoml/cm-mlops/commit/4a4a027f4088ce7e7abcec29c39d98981bf09d4c),
54+
[ref2](https://github.com/octoml/cm-mlops/tree/4a4a027f4088ce7e7abcec29c39d98981bf09d4c),
55+
[ref3](https://github.com/octoml/cm-mlops/blob/7692240becd6397a96c3975388913ea082002e7a/CONTRIBUTIONS).
56+
57+
* May 11, 2022: After successfully prototyping CM and CM-MLOps, deprecated the CK1 framework in favor of CM.
58+
Welcomed Arjun as a maintainer and tester for CM and CM-MLOps:
59+
[ref](https://github.com/octoml/cm-mlops/blob/17405833665bc1e93820f9ff76deb28a0f543bdb/CONTRIBUTIONS).
60+
61+
Created a [file](https://github.com/mlcommons/ck/blob/master/cm-mlops/CHANGES.md)
62+
to document and track our public developments at MLCommons.
63+
64+
* Jun 8, 2022: renamed the 'IC' automation to the more intuitive 'CM script' automation.
65+
[ref1](https://github.com/mlcommons/ck/tree/5ca4e2c33e58a660ac20a545d8aa5143ab6e8e81/cm-devops/automation/script),
66+
[ref2](https://github.com/mlcommons/ck/tree/5ca4e2c33e58a660ac20a545d8aa5143ab6e8e81),
67+
[ref3](https://github.com/octoml/cm-mlops/commit/7910fb7ffc62a617d987d2f887d6f9981ff80187).
68+
69+
* Jun 16, 2022: prototyped the `CM cache` automation to facilitate caching and reuse of the outputs from CM scripts:
70+
[ref1](https://github.com/mlcommons/ck/commit/1f81aae8cebd5567ec4ca55f693beaf32b49fb48),
71+
[ref2](https://github.com/mlcommons/ck/tree/1f81aae8cebd5567ec4ca55f693beaf32b49fb48),
72+
[ref3](https://github.com/mlcommons/ck/tree/1f81aae8cebd5567ec4ca55f693beaf32b49fb48?tab=readme-ov-file#contacts).
73+
74+
* Sep 6, 2022: delivered CM demo to run MLPerf while deprecating CK1 automations for MLPerf:
75+
[ref1](https://github.com/mlcommons/ck/commit/2c5d5c5c944ae5f252113c62af457c7a4c5e877a#diff-faac2c4ecfd0bfb928dafc938d3dad5651762fbb504a2544752a337294ee2573R224),
76+
[ref2](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#author-and-coordinator).
77+
78+
Welcomed Arjun Suresh as a contributor to CM: [ref](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#contributors-in-alphabetical-order).
79+
80+
* From September 2022: coordinated community development of CM and CM4MLOps
81+
to [modularize and automate MLPerf](https://docs.mlcommons.org/inference)
82+
and support [reproducibility initiatives at ML and Systems conferences](https://cTuning.or/ae)
83+
through the MLCommons Task Force on Automation and Reproducibility.
84+
85+
* Starting in April 2024, began the gradual transfer of ongoing maintenance and enhancement
86+
responsibilities for CM and CM4MLOps, including MLPerf automations, to MLCommons.
87+
Welcomed Anandhu Sooraj as a maintainer and contributor to CM4MLOps with MLPerf automations.
88+
89+
For more details, please refer to this [white paper](https://arxiv.org/abs/2406.16791)
90+
and [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339).
91+

Diff for: ‎cm/CHANGES.md

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1-
## V3.5.1.1
1+
## V3.5.2
2+
- added `-i` flag to print memory and disk use before running CM/CMX commands:
3+
https://github.com/mlcommons/ck/issues/1375
4+
- added utils.get_disk_use
25
- added utils.get_memory_use
36
- formatted Python modules from the internal repository using autopep8
47

Diff for: ‎cm/HISTORY.md

+91
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
This document narrates the history of the creation and design of CM and CM4MLOps (also known as CK2)
2+
by [Grigori Fursin](https://cKnowledge.org/gfursin). It also highlights the donation of this open-source technology to MLCommons,
3+
aimed at benefiting the broader community and fostering its ongoing development as a collaborative, community-driven initiative:
4+
5+
* Jan 28, 2021: After delivering an invited ACM TechTalk'21 about the Collective Knowledge framework (CK1)
6+
and reproducibility initiatives for conferences, as well as CK-MLOps and MLPerf automations,
7+
Grigori received lots of positive feedback and suggestions for improvements to workflow automations:
8+
https://learning.acm.org/techtalks/reproducibility.
9+
10+
Following this, Grigori began prototyping CK2 (later CM) to streamline CK1, CK-MLOps and MLPerf benchmarking.
11+
The goal was to dramatically simplify CK1 workflows by introducing just a few core and portable automations,
12+
which eventually evolved into `CM script` and `CM cache`.
13+
14+
At that time, the cTuning foundation hosted CK1 and all the prototypes for the CM framework at https://github.com/ctuning/ck:
15+
[ref1](https://github.com/mlcommons/ck/commit/9e57934f4999db23052531e92160772ab831463a),
16+
[ref2](https://github.com/mlcommons/ck/tree/9e57934f4999db23052531e92160772ab831463a),
17+
[ref3](https://github.com/mlcommons/ck/tree/9e57934f4999db23052531e92160772ab831463a/incubator).
18+
19+
* Sep 23, 2021: donated CK1, CK-MLOps, MLPerf automations and early prototypes of CM from the cTuning repository to MLCommons:
20+
[ref1](https://web.archive.org/web/20240803140223/https://octo.ai/blog/octoml-joins-the-community-effort-to-democratize-mlperf-inference-benchmarking),
21+
[ref2](https://github.com/mlcommons/ck/tree/228f80b0bf44610c8244ff0c3f6bec5bbd25aa6c/incubator),
22+
[ref3](https://github.com/mlcommons/ck/tree/695c3843fd8121bbdde6c453cd6ec9503986b0c6?tab=readme-ov-file#author-and-coordinator),
23+
[ref4](https://github.com/mlcommons/ck/tree/master/ck),
24+
[ref5](https://github.com/mlcommons/ck-mlops).
25+
26+
* Mar 1, 2022: started developing cm-mlops: [ref](https://github.com/octoml/cm-mlops/commit/0ae94736a420dfa84f7417fc62d323303b8760c6).
27+
28+
* Mar 24, 2022: after successfully stabilizing the initial prototype of CM, donated it to MLCommons to benefit the entire community:
29+
[ref1](https://github.com/mlcommons/ck/tree/c7918ad544f26b6c499c2fc9c07431a9640fca5a/ck2),
30+
[ref2](https://github.com/mlcommons/ck/tree/c7918ad544f26b6c499c2fc9c07431a9640fca5a/ck2#coordinators),
31+
[ref3](https://github.com/mlcommons/ck/commit/3c146cb3c75a015363f7a96758adf6dcc43032d6),
32+
[ref4](https://github.com/mlcommons/ck/commit/3c146cb3c75a015363f7a96758adf6dcc43032d6#diff-d97f0f6f5a32f16d6ed18b9600ffc650f7b25512685f7a2373436c492c6b52b3R48).
33+
34+
* Apr 6, 2022: started transitioning previous MLOps and MLPerf automations from the mlcommons/ck-mlops format
35+
to the new CM format using the cm-mlops repository (will be later renamed to cm4mlops):
36+
[ref1](https://github.com/octoml/cm-mlops/commit/d1efdc30fb535ce144020d4e88f3ed768c933176),
37+
[ref2](https://github.com/octoml/cm-mlops/blob/d1efdc30fb535ce144020d4e88f3ed768c933176/CONTRIBUTIONS).
38+
39+
* Apr 22, 2022: began architecting "Intelligent Components" in the CM-MLOps repository,
40+
which will be renamed to `CM Script` at a later stage:
41+
[ref1](https://github.com/octoml/cm-mlops/commit/b335c609c47d2c547afe174d9df232652d57f4f8),
42+
[ref2](https://github.com/octoml/cm-mlops/tree/b335c609c47d2c547afe174d9df232652d57f4f8),
43+
[ref3](https://github.com/octoml/cm-mlops/blob/b335c609c47d2c547afe174d9df232652d57f4f8/CONTRIBUTIONS).
44+
45+
At the same time, prototyped other core CM automations, including IC, Docker, and Experiment:
46+
[ref1](https://github.com/octoml/cm-mlops/tree/b335c609c47d2c547afe174d9df232652d57f4f8/automation),
47+
[ref2](https://github.com/mlcommons/ck/commits/master/?before=7f66e2438bfe21b4ce2d08326a5168bb9e3132f6+7001).
48+
49+
* Apr 28, 2022: donated CM-MLOps to MLCommons, which was later renamed to CM4MLOps:
50+
[ref](https://github.com/mlcommons/ck/commit/456e4861056c0e39c4d689c03da91f90a44be058).
51+
52+
* May 9, 2022: developed the initial set of core IC automations for MLOps (aka CM scripts):
53+
[ref1](https://github.com/octoml/cm-mlops/commit/4a4a027f4088ce7e7abcec29c39d98981bf09d4c),
54+
[ref2](https://github.com/octoml/cm-mlops/tree/4a4a027f4088ce7e7abcec29c39d98981bf09d4c),
55+
[ref3](https://github.com/octoml/cm-mlops/blob/7692240becd6397a96c3975388913ea082002e7a/CONTRIBUTIONS).
56+
57+
* May 11, 2022: After successfully prototyping CM and CM-MLOps, deprecated the CK1 framework in favor of CM.
58+
Welcomed Arjun as a maintainer and tester for CM and CM-MLOps:
59+
[ref](https://github.com/octoml/cm-mlops/blob/17405833665bc1e93820f9ff76deb28a0f543bdb/CONTRIBUTIONS).
60+
61+
Created a [file](https://github.com/mlcommons/ck/blob/master/cm-mlops/CHANGES.md)
62+
to document and track our public developments at MLCommons.
63+
64+
* Jun 8, 2022: renamed the 'IC' automation to the more intuitive 'CM script' automation.
65+
[ref1](https://github.com/mlcommons/ck/tree/5ca4e2c33e58a660ac20a545d8aa5143ab6e8e81/cm-devops/automation/script),
66+
[ref2](https://github.com/mlcommons/ck/tree/5ca4e2c33e58a660ac20a545d8aa5143ab6e8e81),
67+
[ref3](https://github.com/octoml/cm-mlops/commit/7910fb7ffc62a617d987d2f887d6f9981ff80187).
68+
69+
* Jun 16, 2022: prototyped the `CM cache` automation to facilitate caching and reuse of the outputs from CM scripts:
70+
[ref1](https://github.com/mlcommons/ck/commit/1f81aae8cebd5567ec4ca55f693beaf32b49fb48),
71+
[ref2](https://github.com/mlcommons/ck/tree/1f81aae8cebd5567ec4ca55f693beaf32b49fb48),
72+
[ref3](https://github.com/mlcommons/ck/tree/1f81aae8cebd5567ec4ca55f693beaf32b49fb48?tab=readme-ov-file#contacts).
73+
74+
* Sep 6, 2022: delivered CM demo to run MLPerf while deprecating CK1 automations for MLPerf:
75+
[ref1](https://github.com/mlcommons/ck/commit/2c5d5c5c944ae5f252113c62af457c7a4c5e877a#diff-faac2c4ecfd0bfb928dafc938d3dad5651762fbb504a2544752a337294ee2573R224),
76+
[ref2](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#author-and-coordinator).
77+
78+
Welcomed Arjun Suresh as a contributor to CM: [ref](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#contributors-in-alphabetical-order).
79+
80+
* From September 2022: coordinated community development of CM and CM4MLOps
81+
to [modularize and automate MLPerf](https://docs.mlcommons.org/inference)
82+
and support [reproducibility initiatives at ML and Systems conferences](https://cTuning.or/ae)
83+
through the MLCommons Task Force on Automation and Reproducibility.
84+
85+
* Starting in April 2024, began the gradual transfer of ongoing maintenance and enhancement
86+
responsibilities for CM and CM4MLOps, including MLPerf automations, to MLCommons.
87+
Welcomed Anandhu Sooraj as a maintainer and contributor to CM4MLOps with MLPerf automations.
88+
89+
For more details, please refer to this [white paper](https://arxiv.org/abs/2406.16791)
90+
and [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339).
91+

Diff for: ‎cm/cmind/__init__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
#
33
# Written by Grigori Fursin
44

5-
__version__ = "3.5.1.1"
5+
__version__ = "3.5.2"
66

77
from cmind.core import access
88
from cmind.core import x

Diff for: ‎cm/cmind/core.py

+10-1
Original file line numberDiff line numberDiff line change
@@ -858,7 +858,7 @@ def x(self, i, out = None):
858858
'h', 'help', 'version', 'out', 'j', 'json',
859859
'save_to_json_file', 'save_to_yaml_file', 'common',
860860
'ignore_inheritance', 'log', 'logfile', 'raise', 'repro',
861-
'f', 'time', 'profile']]
861+
'i', 'f', 'time', 'profile']]
862862

863863
delayed_error = ''
864864

@@ -882,6 +882,8 @@ def x(self, i, out = None):
882882

883883
self_profile = control.get('profile', False)
884884

885+
self_info = control.get('i', False)
886+
885887
# Check repro
886888
use_log = str(control_flags.pop('log', '')).strip().lower()
887889
log_file = control_flags.pop('logfile', '')
@@ -955,6 +957,13 @@ def x(self, i, out = None):
955957
self.log(f"x input: {spaces} ({i})", "debug")
956958

957959
# Call access helper
960+
if not x_was_called and self_info:
961+
utils.get_memory_use(True)
962+
print ('')
963+
utils.get_disk_use('/', True)
964+
print ('')
965+
966+
958967
if not x_was_called and self_profile:
959968
# https://docs.python.org/3/library/profile.html#module-cProfile
960969
import cProfile, pstats, io

Diff for: ‎cm/cmind/utils.py

+38
Original file line numberDiff line numberDiff line change
@@ -2219,3 +2219,41 @@ def get_memory_use(console = False):
22192219
'total_memory': total_memory,
22202220
'total_memory_gb': total_memory_gb}
22212221

2222+
##############################################################################
2223+
def get_disk_use(path = '/', console = False):
2224+
"""
2225+
Get disk space
2226+
2227+
Args:
2228+
console (bool): if True, print to console
2229+
2230+
Returns:
2231+
total (int)
2232+
total_gb (float)
2233+
used (int)
2234+
used_gb (float)
2235+
free (int)
2236+
free_gb (float)
2237+
2238+
"""
2239+
2240+
import shutil
2241+
2242+
total, used, free = shutil.disk_usage(path)
2243+
2244+
total_gb = total / 1e9
2245+
used_gb = used / 1e9
2246+
free_gb = free / 1e9
2247+
2248+
if console:
2249+
print(f"Total disk space: {total_gb:.2f} GB")
2250+
print(f"Used disk space: {used_gb:.2f} GB")
2251+
print(f"Free disk space: {free_gb:.2f} GB")
2252+
2253+
return {'return':0,
2254+
'total': total,
2255+
'total_gb': total_gb,
2256+
'used': used,
2257+
'used_gb': used_gb,
2258+
'free': free,
2259+
'free_gb': free_gb}

Diff for: ‎cm4abtf/README.md

-1
This file was deleted.

Diff for: ‎cm4mlops/README.md

-5
This file was deleted.

Diff for: ‎cm4mlperf/README.md

-4
This file was deleted.

0 commit comments

Comments
 (0)