Collective Knowledge (CK) in an educational project to help researchers and engineers automate their repetitive, tedious and time-consuming tasks to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data, software and hardware.

CK consists of several sub-projects:

• Collective Mind framework (CM) - a very lightweight Python-based framework with minimal dependencies to help users implement, share and reuse cross-platform automation recipes to build, benchmark and optimize applications on any platform with any software and hardware.

  • CM interface to run MLPerf inference benchmarks

  • CM4MLOPS - a collection of portable, extensible and technology-agnostic automation recipes with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on diverse platforms with any software and hardware: see online catalog at CK playground, online MLCommons catalog

  • CM4ABTF - a unified CM interface and automation recipes to run automotive benchmark across different models, data sets, software and hardware from different vendors.

• CMX (the next generation of CM) - we are developing the next generation of CM to make it simpler and more flexible based on user feedback. Please follow this project here.

• Collective Knowledge Playground - a unified platform to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results in a reproducible format with CM workflows, and organize public optimization challenges and reproducibility initiatives to co-design more efficient and cost-effiective software and hardware for emerging workloads.

• Artifact Evaluation - automating artifact evaluation and reproducibility initiatives at ML and systems conferences.

Apache 2.0

• Copyright (c) 2021-2024 MLCommons
• Copyright (c) 2014-2021 cTuning foundation

• CM/CMX/CM4Research: Grigori Fursin
• CM4MLOps: Arjun Suresh and Anandhu Sooraj

You can learn more about the motivation behind these projects from the following articles and presentations:

• "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ ArXiv ]
• ACM REP'23 keynote about the MLCommons CM automation framework: [ slides ]
• ACM TechTalk'21 about automating research projects: [ YouTube ] [ slides ]

• CM installation GUI
• CM Getting Started Guide and FAQ
  • Common CM interface to run MLPerf inference benchmarks
  • Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23
  • CM automation recipes for MLOps and DevOps
  • Other CM tutorials
• Full documentation
• CM development tasks
• CM and CK history

The Collective Mind automation framework (CM) was created by Grigori Fursin, sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. This open-source technology (CM, CM4MLOps, CM4MLPerf, CM4ABTF, CM4Research, etc) is being developed as a community effort thanks to all our fantastic volunteers, collaborators and contributors!