Collective Knowledge - lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows
Project description
Collective Knowledge Framework (CK)
Linux/MacOS: Windows: Coverage:
Introduction
We have developed the Collective Knowledge framework (CK) to help researchers share their projects and artifacts (code, data, models, scripts, experiments, papers) in a common format as a human-readable database with a standardized API, CLI and JSON input/output/meta descriptions
CK is a small, cross-platform, CLI-based and community-driven Python framework to add, share and reuse automation actions for repetitive, tedious, and time-consuming R&D tasks in a non-intrusive way along with existing research projects.
CK also helps to convert all ad-hoc artifacts into standardized and reusable CK components with a common API and JSON meta description. For example, CK features software detection plugins (CK "soft" component), meta packages (CK "package" component) and OS descriptions to automate the detection and installation of all the dependencies required by a given research project to run on any target platform.
Such CK actions and components can be connected into platform-agnostic, portable, customizable, reusable and reproducible workflows that can be easily integrated with CI tools, existing/legacy projects, and production systems.
The stable components are published at the open CodeReef platform similar to PyPI along with auto-generated "live" papers and portable workflows to help the community participate in collaborative validation of research results across diverse hardware, datasets and models similar to SETI@home.
Our long-term goal is to enable collaborative, reproducible, sustainable and production-ready research based on DevOps principles.
Learn more about our long-term vision in the following white papers and presentations: MLOps@MLSys'20, FOSDEM'19, CNRS'17, DATE'16, CTI'09.
Documentation
- CK wiki - a major revision is planned in 2020!
- CK basics
- CK Getting Started Guide
Sphinx-based documentation is under preparation
Important features
-
CK actions can be shared and reused across research projects: see the list of available actions and modules.
-
Standardized CK APIs and meta-descriptions help users to easily connect actions into automated, portable and customizable workflows, and quickly integrate them with practically all major tools, frameworks and Continuous Integration Services: see the list of shared repositories with CK workflows and actions.
-
CK helps to automate Artifact Evaluation, perform reproducible experiments and generate papers with reusable research components: see the list of articles with CK workflows and the CK-based interactive report with the Raspberry Pi foundation.
-
CK is used in reproducible optimization competitions to co-design efficient software and hardware for emerging AI, ML and quantum computing workloads in terms of speed, accuracy, energy, and costs: see the live CK dashboard with results from different Hackathons, tournaments and crowd-tuning campaigns.
CK-based projects
- Research projects in the CK format from our partners.
- Research papers from ML and systems conferences shared with artifacts and workflows in the CK format.
- MLPerf benchmark automation demo
- GitHub repositories in the CK format
- R&D automation actions
- Software detection plugins
- Meta-packages
- Artifact abstractions (CK modules)
Installation
You can install the Collective Knowledge framework on most platforms using PIP as follows:
pip install ck
You can also install CK using a specific Python version (for example, Python 3.6 or for Python 2.7):
python3.6 -m pip install ck
or
python2.7 -m pip install ck
You may need to add flag "--user" to install the client in your user space:
pip install ck --user
python3.6 -m pip install ck --user
You should now be able to run CK using one of the following alternative commands:
ck
python3.6 -m ck
If the installation is successful, you will see some internal information about the CK installation and a Python version used:
CK version: 1.11.4.1
Python executable used by CK: c:\Users\fgg\Anaconda3\python.exe
Python version used by CK: 3.6.3 |Anaconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)]
Path to the default repo: D:\Work1\CK\ck\ck\repo
Path to the local repo: D:\Work1\CK\ck-repos\local
Path to CK repositories: D:\Work1\CK\ck-repos
Documentation: https://github.com/ctuning/ck/wiki
CK Google group: https://bit.ly/ck-google-group
CK Slack channel: https://cKnowledge.org/join-slack
Prerequisites
The CK framework requires minimal dependencies: Python 2.7+ or 3.x, PIP and Git.
CK supports the following platforms:
As a host platform | As a target platform | |
---|---|---|
Generic Linux | ✓ | ✓ |
Linux (Arm) | ✓ | ✓ |
Raspberry Pi | ✓ | ✓ |
MacOS | ✓ | ✓ |
Windows | ✓ | ✓ |
Android | partially | ✓ |
iOS | TBD | TBD |
Linux
You need to have the following packages installed (Ubuntu example):
sudo apt-get install python3 python3-pip git wget
MacOS
brew install python3 python3-pip git wget
Windows
- Download and install Git from git-for-windows.github.io.
- Download and install any Python from www.python.org/downloads/windows.
Android (Linux host)
These dependencies are needed to cross-compile for Android (tested on Ubuntu 18.04 including Docker and Windows 10 Subsystem for Linux).
sudo apt update
sudo apt install git wget libz-dev curl cmake
sudo apt install gcc g++ autoconf autogen libtool
sudo apt install android-sdk
sudo apt install google-android-ndk-installer
Docker
We prepared several Docker images with the CK framework and AI/ML CK workflows at the cTuning Docker hub. Select the most relevant image and run it as follows:
docker run -p 3344:3344 -it {Docker image name from the above list} /bin/bash
Next steps
Based on user feedback we plan the following activities:
- Standardization of CK actions, APIs and meta descriptions
- Better documentation
- GUI to create, test and interconnect CK actions
- GUI to assemble portable workflows
- GUI to automate MLPerf submissions
Get involved
Please follow this guide to add your workflows and components. Note that we plan to rewrite it and add tutorials as soon as we have more resources!
Provide your suggestions using our public mailing list and the Slack channel!
Help the community to improve the existing CK components (actions, modules, packages, software plugins, workflows), when they fail on new platforms or miss some functionality, share the new ones and fix buges - you can provide your feedback and report bugs in the respective CK development repositories or using the CodeReef platform!
Consider sponsoring the cTuning foundation to support our community activities.
Contact Grigori Fursin (the CK author) about our long-term vision and development plans.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.