learnware 0.2.0.7

learnware market project

Learnware is a model sharing platform, which give a basic implementation of the learnware paradigm. A learnware is a well-performed trained machine learning model with a specification that enables it to be adequately identified to reuse according to the requirement of future users who may know nothing about the learnware in advance. The learnware paradigm can solve entangled problems in the current machine learning paradigm, like continual learning and catastrophic forgetting. It also reduces resources for training a well-performed model.

Introduction

Framework

Machine learning, especially the prevailing big model paradigm, has achieved great success in natural language processing and computer vision applications. However, it still faces challenges such as the requirement of a large amount of labeled training data, difficulty in adapting to changing environments, and catastrophic forgetting when refining trained models incrementally. These big models, while useful in their targeted tasks, often fail to address the above issues and struggle to generalize beyond their specific purposes.

The learnware paradigm introduces the concept of a well-performed, trained machine learning model with a specification that allows future users, who have no prior knowledge of the learnware, to reuse it based on their requirements.

Developers or owners of trained machine learning models can submit their models to a learnware market. If accepted, the market assigns a specification to the model and accommodates it. The learnware market could host thousands or millions of well-performed models from different developers, for various tasks, using diverse data, and optimizing different objectives.

Instead of building a model from scratch, users can submit their requirements to the learnware market, which then identifies and deploys helpful learnware(s) based on the specifications. Users can apply the learnware directly, adapt it using their data, or exploit it in other ways to improve their model. This process is more efficient and less expensive than building a model from scratch.

Benefits of the Learnware Paradigm

Benefit	Description
Lack of training data	Strong models can be built with small data by adapting well-performed learnwares.
Lack of training skills	Ordinary users can obtain strong models by leveraging well-performed learnwares instead of building models from scratch.
Catastrophic forgetting	Accepted learnwares are always stored in the learnware market, retaining old knowledge.
Continual learning	The learnware market continually enriches its knowledge with constant submissions of well-performed learnwares.
Data privacy/ proprietary	Developers only submit models, not data, preserving data privacy/proprietary.
Unplanned tasks	Open to all legal developers, the learnware market can accommodate helpful learnwares for various tasks.
Carbon emission	Assembling small models may offer good-enough performance, reducing interest in training large models and the carbon footprint.

Quick Start

Installation

Learnware is currently hosted on PyPI. You can easily intsall Learnware according to the following steps:

• For Windows and Linux users:

  pip install learnware
  

• For macOS users:

  conda install -c pytorch faiss
  pip install learnware
  

Prepare Learnware

The Learnware Market consists of a wide range of learnwares. A valid learnware is a zipfile which is composed of the following four parts.

• __init__.py

  A python file offering interfaces for your model's fitting, predicting and fine-tuning.

• rkme.json

  A json file containing the statistical specification of your data.

• learnware.yaml

  A config file describing your model class name, type of statistical specification(e.g. Reduced Kernel Mean Embedding, RKMETableSpecification), and the file name of your statistical specification file.

• environment.yaml

  A Conda environment configuration file for running the model (if the model environment is incompatible, you can rely on this for manual configuration). You can generate this file according to the following steps:

  • Create env config for conda:

    conda env export | grep -v "^prefix: " > environment.yaml
    

  • Recover env from config:

    conda env create -f environment.yaml
    

We also demonstrate the detail format of learnware zipfile in [DOC link], and also please refer to Examples for concrete learnware zipfile example.

Learnware Market Workflow

Users can start an Learnware workflow according to the following steps:

Initialize a Learware Market

The EasyMarket class implements the most basic set of functions in a Learnware. You can use the following code snippet to initialize a basic Learnware named "demo":

import learnware
from learnware.market import EasyMarket

learnware.init()
easy_market = EasyMarket(market_id="demo", rebuild=True)

Upload Leanwares

Before uploading your learnware into the Learnware, create a semantic specification semantic_spec by selecting or filling in values for the predefined semantic tags to describe the features of your task and model.

For example, the following code snippet demonstrates the semantic specification of a Scikit-Learn type model, which is designed for business scenario and performs classification on tabular data:

semantic_spec = {
    "Data": {"Values": ["Tabular"], "Type": "Class"},
    "Task": {"Values": ["Classification"], "Type": "Class"},
    "Library": {"Values": ["Scikit-learn"], "Type": "Class"},
    "Scenario": {"Values": ["Business"], "Type": "Tag"},
    "Description": {"Values": "", "Type": "String"},
    "Name": {"Values": "demo_learnware", "Type": "String"},
}

Once the semantic specification is defined, you can easily upload your learnware with a single line of code:

easy_market.add_learnware(zip_path, semantic_spec) 

Here, zip_path is the directory of your learnware zipfile.

Semantic Specification Search

To search for learnwares that fit your task purpose, you should also provide a semantic specification user_semantic that describes the characteristics of your task. The Learnware will perform a first-stage search based on user_semantic, identifying potentially helpful leranwares whose models solve tasks similar to your requirements.

# construct user_info which includes semantic specification for searching learnware
user_info = BaseUserInfo(id="user", semantic_spec=semantic_spec)

# search_learnware performs semantic specification search if user_info doesn't include a statistical specification
_, single_learnware_list, _ = easy_market.search_learnware(user_info) 

# single_learnware_list is the learnware list by semantic specification searching
print(single_learnware_list)

Statistical Specification Search

If you choose to porvide your own statistical specification file stat.json, the Learnware can perform a more accurate leanware selection from the learnwares returned by the previous step. This second-stage search is based on statistical information and returns one or more learnwares that are most likely to be helpful for your task.

For example, the following code is designed to work with Reduced Set Kernel Embedding as a statistical specification:

import learnware.specification as specification

user_spec = specification.RKMETableSpecification()
user_spec.load(os.path.join(unzip_path, "rkme.json"))
user_info = BaseUserInfo(
    semantic_spec=user_semantic, stat_info={"RKMETableSpecification": user_spec}
)
(sorted_score_list, single_learnware_list,
    mixture_score, mixture_learnware_list) = easy_market.search_learnware(user_info)

# sorted_score_list is the learnware scores based on MMD distances, sorted in descending order
print(sorted_score_list) 

# single_learnware_list is the learnwares sorted in descending order based on their scores
print(single_learnware_list)

# mixture_learnware_list is the learnwares whose mixture is helpful for your task
print(mixture_learnware_list) 

# mixture_score is the score of the mixture of learnwares
print(mixture_score)

Reuse Learnwares

Based on the returned list of learnwares mixture_learnware_list in the previous step, you can easily reuse them to make predictions your own data, instead of training a model from scratch. We provide two baseline methods for reusing a given list of learnwares, namely JobSelectorReuser and AveragingReuser. Simply replace test_x in the code snippet below with your own testing data and start reusing learnwares!

# using jobselector reuser to reuse the searched learnwares to make prediction
reuse_job_selector = JobSelectorReuser(learnware_list=mixture_learnware_list)
job_selector_predict_y = reuse_job_selector.predict(user_data=test_x)

# using averaging ensemble reuser to reuse the searched learnwares to make prediction
reuse_ensemble = AveragingReuser(learnware_list=mixture_learnware_list)
ensemble_predict_y = reuse_ensemble.predict(user_data=test_x)

Auto Workflow Example

Learnware also provides an auto workflow example, which includes preparing learnwares, upload and delete learnware from markets, search learnware with semantic specifications and statistical specifications. The users can run examples/workflow_by_code.py to try the basic workflow of Learnware.

Experiments and Examples

Environment

For all experiments, we used a single linux server. Details on the specifications are listed in the table below. All processors were used for training and evaluating.

System	GPU	CPU
Ubuntu 20.04.4 LTS	Nvidia Tesla V100S	Intel(R) Xeon(R) Gold 6240R

Datasets

We designed experiments on three publicly available datasets, namely Prediction Future Sales (PFS), M5 Forecasting (M5) and CIFAR 10. For the two sales forecasting data sets of PFS and M5, we divide the user data according to different stores, and train the Ridge model and LightGBM model on the corresponding data respectively. For the CIFAR10 image classification task, we first randomly pick 6 to 10 categories, and randomly select 800 to 2000 samples from each category from the categories corresponding to the training set, constituting a total of 50 different uploaders. For test users, we first randomly pick 3 to 6 categories, and randomly select 150 to 350 samples from each category from the corresponding categories from the test set, constituting a total of 20 different users.

We tested the efficiency of the specification generation and the accuracy of the search and reuse model respectively. The evaluation index on PFS and M5 data is RMSE, and the evaluation index on CIFAR10 classification task is classification accuracy

Results

The time-consuming specification generation is shown in the table below:

Dataset	Data Dimensions	Specification Generation Time (s)
PFS	8714274*31	< 1.5
M5	46027957*82	9~15
CIFAR 10	9000*3*32*32	7~10

The accuracy of search and reuse is shown in the table below:

Dataset	Top-1 Performance	Job Selector Reuse	Average Ensemble Reuse
PFS	1.955 +/- 2.866	2.175 +/- 2.847	1.950 +/- 2.888
M5	2.066 +/- 0.424	2.116 +/- 0.472	2.512 +/- 0.573
CIFAR 10	0.619 +/- 0.138	0.585 +/- 0.056	.715 +/- 0.075

About

Contributor

We appreciate all contributions and thank all the contributors!

TODO: Here paste the github API after publishing:

Pic after publish

About us

Visit LAMDA's official website.