Skip to main content

Collective Knowledge - lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows

Project description

Collective Knowledge Framework (CK)

Downloads PyPI version Python Version DOI License

Linux/MacOS: Build Status Windows: Windows Build status Coverage: Coverage Status

Introduction

We have developed the Collective Knowledge framework (CK) to help researchers share their projects and artifacts (code, data, models, scripts, experiments, papers) in a common format as a human-readable database with a standardized API, CLI and JSON input/output/meta descriptions

CK is a small, cross-platform, CLI-based and community-driven Python framework to add, share and reuse automation actions for repetitive, tedious, and time-consuming R&D tasks in a non-intrusive way along with existing research projects.

CK also helps to convert all ad-hoc artifacts into standardized and reusable CK components with a common API and JSON meta description. For example, CK features software detection plugins (CK "soft" component), meta packages (CK "package" component) and OS descriptions to automate the detection and installation of all the dependencies required by a given research project to run on any target platform.

Such CK actions and components can be connected into platform-agnostic, portable, customizable, reusable and reproducible workflows that can be easily integrated with CI tools, existing/legacy projects, and production systems.

The stable components are published at the open CodeReef platform similar to PyPI along with auto-generated "live" papers and portable workflows to help the community participate in collaborative validation of research results across diverse hardware, datasets and models similar to SETI@home.

Our long-term goal is to enable collaborative, reproducible, sustainable and production-ready research based on DevOps principles.

Learn more about our long-term vision in the following white papers and presentations: MLOps@MLSys'20, FOSDEM'19, CNRS'17, DATE'16, CTI'09.

Documentation

Sphinx-based documentation is under preparation

Important features

CK-based projects

Installation

You can install the Collective Knowledge framework on most platforms using PIP as follows:

pip install ck

You can also install CK using a specific Python version (for example, Python 3.6 or for Python 2.7):

python3.6 -m pip install ck

or

python2.7 -m pip install ck

You may need to add flag "--user" to install the client in your user space:

pip install ck --user
python3.6 -m pip install ck --user

You should now be able to run CK using one of the following alternative commands:

ck

python3.6 -m ck

If the installation is successful, you will see some internal information about the CK installation and a Python version used:

CK version: 1.11.4.1

Python executable used by CK: c:\Users\fgg\Anaconda3\python.exe

Python version used by CK: 3.6.3 |Anaconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)]

Path to the default repo: D:\Work1\CK\ck\ck\repo
Path to the local repo:   D:\Work1\CK\ck-repos\local
Path to CK repositories:  D:\Work1\CK\ck-repos

Documentation:    https://github.com/ctuning/ck/wiki
CK Google group:  https://bit.ly/ck-google-group
CK Slack channel: https://cKnowledge.org/join-slack

Prerequisites

The CK framework requires minimal dependencies: Python 2.7+ or 3.x, PIP and Git.

CK supports the following platforms:

As a host platform As a target platform
Generic Linux
Linux (Arm)
Raspberry Pi
MacOS
Windows
Android partially
iOS TBD TBD

Linux

You need to have the following packages installed (Ubuntu example):

sudo apt-get install python3 python3-pip git wget

MacOS

brew install python3 python3-pip git wget

Windows

Android (Linux host)

These dependencies are needed to cross-compile for Android (tested on Ubuntu 18.04 including Docker and Windows 10 Subsystem for Linux).

 sudo apt update
 sudo apt install git wget libz-dev curl cmake
 sudo apt install gcc g++ autoconf autogen libtool
 sudo apt install android-sdk
 sudo apt install google-android-ndk-installer

Docker

We prepared several Docker images with the CK framework and AI/ML CK workflows at the cTuning Docker hub. Select the most relevant image and run it as follows:

docker run -p 3344:3344 -it {Docker image name from the above list} /bin/bash

Next steps

Based on user feedback we plan the following activities:

  • Standardization of CK actions, APIs and meta descriptions
  • Better documentation
  • GUI to create, test and interconnect CK actions
  • GUI to assemble portable workflows
  • GUI to automate MLPerf submissions

Get involved

Please follow this guide to add your workflows and components. Note that we plan to rewrite it and add tutorials as soon as we have more resources!

Provide your suggestions using our public mailing list and the Slack channel!

Help the community to improve the existing CK components (actions, modules, packages, software plugins, workflows), when they fail on new platforms or miss some functionality, share the new ones and fix buges - you can provide your feedback and report bugs in the respective CK development repositories or using the CodeReef platform!

Consider sponsoring the cTuning foundation to support our community activities.

Contact Grigori Fursin (the CK author) about our long-term vision and development plans.

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ck-1.12.3.tar.gz (135.0 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page