Skip to main content

Collective Knowledge - a lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows

Project description

Collective Knowledge framework (CK)

Downloads PyPI version Python Version License

Linux/MacOS: Build Status Windows: Windows Build status

Documentation Status Coverage Status

News

  • We are very excited to join forces with OctoML.ai! Contact Grigori Fursin for more details!
  • See our ACM TechTalk about the CK technology, reproducible research, FAIR principles and MLPerf (February 2021).
  • The overview of the CK technology has appeared in the Philosophical Transactions A, the world's longest-running journal where Newton published: DOI, ArXiv.
  • We are preparing a major redesign of the CK framework to make it more Pythonic and easier to use (CK2) - feel free to contact Grigori Fursin for more details and subscribe for our newsletter!
  • We have finished prototyping the CK concept and successfully used it to automate MLPerf inference benchmark submissions for edge devices and the design space exploration of efficient ML/SW/HW stacks in terms of speed, accuracy, energy and costs.

Overview

Collective Knowledge framework (CK) helps to organize software projects as a database of reusable components with common automation actions and extensible meta descriptions based on FAIR principles (findability, accessibility, interoperability and reusability) as described in this article.

Our goal is to help researchers and practitioners share, reuse and extend their knowledge in the form of portable workflows, automation actions and reusable artifacts with a common API, CLI, and meta description. See how CK supports collaborative and reproducible research:

Documentation

Installation

Follow this guide to install CK framework on your platform.

CK supports the following platforms:

As a host platform As a target platform
Generic Linux
Linux (Arm)
Raspberry Pi
MacOS ±
Windows
Android ±
iOS TBD TBD
Bare-metal (edge devices) - ±

Example (without Docker)

Here we show how to pull a GitHub repo in the CK format and use a unified CK interface to compile and run any program (image corner detection in our case) with any compatible data set on any compatible platform:

python3 -m pip install ck

ck pull repo --url=https://github.com/ctuning/ck-crowdtuning

ck ls program:*susan*

ck search dataset --tags=jpeg

ck detect soft --tags=compiler,gcc
ck detect soft --tags=compiler,llvm

ck show env --tags=compiler

ck compile program:cbench-automotive-susan --speed

ck run program:cbench-automotive-susan --cmd_key=corners --repeat=1 --env.MY_ENV=123 --env.TEST=xyz

You can check output of this program in the following directory:

cd `ck find program:cbench-automotive-susan`/tmp
ls -l

tmp-output.tmp - image with detected corners (rename to ppm to view it)

Check CK docs for further details.

Example (with Docker)

We have prepared a CK container with AI and ML components: [Docker], [CK meta]

You can start it as follows:

docker run --rm -it ctuning/ck-ai:ubuntu-20.04

You can then prepare and run portable AI/ML workflows and program pipelines.

More examples of CK workflows, automation actions and reusable artifacts for

CK portal

cKnowledge.io: organizing ML and Systems knowledge in the form of portable CK workflows, automation actions and reusable components:

Contributions

Users can extend the CK functionality via CK modules or external GitHub reposities in the CK format as described here.

Please check this documentation if you want to extend the CK core functionality and modules.

Note, that we plan to redesign the CK core to be more pythonic (we wrote the first prototype without OO to be able to port to bare-metal devices in C but we decided not to do it at the end). We also plan to relicense the framework to Apache 2.0.

Author

Acknowledgments

We would like to thank all contributors and collaborators for their support, fruitful discussions, and useful feedback! See more acknowledgments in the CK journal article.

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ck-1.55.5.tar.gz (1.4 MB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page