cmind 1.5.3

cmind

About

The MLCommons Task Force on Automation and Reproducibility, cTuning foundation and cKnowledge.org have developed a non-intrusive, technology-agnostic and plugin-based Collective Mind automation language (CM) to enable collaborative, reproducible, automated and unified benchmarking, optimization and comparison of computer systems access diverse and rapidly evolving models, data sets, software and hardware from different vendors.

The CM automation language was successfully validated by the community and cTuning foundation via public optimization and reproducibility challenges at the CK playground to automate > 90% of all MLPerf inference v3.1 results and cross 10000 submissions in one round for the first time - please read this HPC Wire article about the cTuning's community submission and check the list of the new CM/CK capabilities available to everyone to prepare and automate their future MLPerf submissions.

See related ACM REP'23 keynote, ACM Tech Talk and MLPerf submitters orientation to learn more about our open-source developments and long-term vision.

Join our Discord server to ask questions, provide feedback and participate in collaborative developments.

Documentation

• Table of contents

Some practical use cases

• CM installation
• All CM tutorials

Run Python Hello World app

python3 -m pip install cmind
# restart bash to add cm and cmr binaries to PATH

cm pull repo mlcommons@ck
cm run script --tags=print,python,hello-world
cmr "print python hello-world"

This CM script is a simple wrapper to native scripts and tools described by a simple declarative YAML configuration file specifying inputs, environment variables and dependencies on other portable and shared CM scripts:

alias: print-hello-world-py
uid: d83274c7eb754d90

automation_alias: script
automation_uid: 5b4e0237da074764

deps:
- tags: detect,os
- tags: get,sys-utils-cm
- names:
  - python
  tags: get,python3

tags:
- print
- hello-world
- python

Our goal is to let the community start using CM within minutes!

Run MLPerf benchmarks out-of-the-box

• CM automation for the new MLPerf submitters
• MLPerf inference automation
• Visualization of MLPerf results

Participate in reproducible AI/ML Systems optimization challenges

We invite the community to participate in collaborative benchmarking and optimization of AI/ML systems:

• Community challenges (reproducibility, extension, benchmarking, optimization)
• Shared benchmarking results for AI/ML Systems (performance, accuracy, power consumption, costs)
• Leaderboard

Reproduce results from ACM/IEEE/NeurIPS papers

• CM automation to reproduce results from ACM/IEEE MICRO'23 papers
• CM automation to support Student Cluster Competition at SuperComputing'23
• CM automation to reproduce IPOL paper

Project coordinators

• Grigori Fursin
• Arjun Suresh

Copyright

2021-2023 MLCommons

License

Apache 2.0

Acknowledgments

Collective Mind automation language was developed from scratch by Grigori Fursin and Arjun Suresh in 2022-2023 within the MLCommons Task Force on Automation and Reproducibility and with many great contributions from the community.

This project is supported by MLCommons, cTuning foundation, cKnowledge.org, and individual contributors. We thank HiPEAC and OctoML for sponsoring initial development.