Skip to main content

Collective Knowledge - a lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows

Project description

Collective Knowledge SDK

Downloads PyPI version Python Version DOI License

Linux/MacOS: Build Status Windows: Windows Build status Coverage: Coverage Status

Introduction

Developing complex computational systems for emerging workloads (ML, AI, Quantum, IoT) and moving them to production is a very tedious, ad-hoc and time consuming process due to continuously changing software, hardware, models, data sets and research techniques.

The Collective Knowledge project (CK) is our attempt to solve these problems with a common SDK and an open repository of knowledge to decompose complex systems and research projects into reusable, portable, customizable and non-virtualized CK components with the unified automation actions, Python APIs, CLI and JSON meta description. We can then use the DevOps methodology to connect such components together into portable CK solutions that can automatically adapt to evolving models, data sets and bare-metal platforms.

Our goal is to help researchers and practitioners to collaboratively validate innovative techniques, benchmark and optimize novel computational systems, and find the most efficient ones in terms of speed, accuracy, energy, size and different costs using a public repository of knowledge with live SOTA scoreboards and reproducible papers.

We hope that such approach will make it possible to better understand what is happening inside complex and "black box" computational systems, integrate them with production and legacy systems, use them inside Docker and Kubernetes, share them along with published papers, and apply the DevOps methodology in deep tech research and computational science. We also hope that the CK technology will complement existing reproducibility efforts such as PapersWithCode, ACM and artifact evaluation.

Please check our MLPerf automation demo and feel free to get in touch if you have any feedback and suggestions! You can learn more about our project in the CK documentation and the following presentations and white papers: 2019, 2018, 2017, 2009.

Even though the CK technology is used in production for more than 5 years, it is still a proof-of-concept prototype requiring further improvements and standardization. Depending on the available resources, we plan to develop a new, backward-compatible and more user-friendly version - please get in touch if you are interested to know more!

Open knowledge portal

CK use cases

Documentation

Older wiki-based documentation (we gradually move it to Sphinx-based docs above):

Installation

Follow this guide to install the CK SDK on your platform.

Check CK-based demo to participate in reproducible MLPerf benchmarking with a public SOTA scoreboard.

CK supports the following platforms:

As a host platform As a target platform
Generic Linux
Linux (Arm)
Raspberry Pi
MacOS ±
Windows
Android ±
iOS TBD TBD

Get involved

  • This is an ongoing community project and there is a lot to be improved - don't hesitate to get in touch using our slack, mailing list, twitter and email if you have any feedback or would like to collaborate.

  • Check this outdated guide to add your workflows and components. We are gradually rewriting this guide with our limited resources and developing a CK wrapper (cBrain) to simplify the user experience and provide a CK GUI at the open cKnowledge.io platform.

Acknowledgments

We would like to thank all CK users and partners for fruitful discussions and feedback!

Copyright 2015-2020 Grigori Fursin and the cTuning foundation

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ck-1.15.0.tar.gz (133.5 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page