dlk 0.1.0

dlk: Deep Learning Kit

Deep Learning toolKit (dlk)

English | 简体中文

• Install
• Demo
  • Grid Search
  • Task Demo
• Usage and Features
  • Usage
  • Module Registration
  • Some Built-in Modules Introduction
    • Callbacks
    • Adversarial Training
    • Complex Training Control
    • Text Generation
  • More Documentation

In our work, we often need to conduct rapid experiments on deep neural network models, search for optimal structures and parameters, and deploy the optimal models. Sometimes, we also need to create demos for validation.

The core code for training, prediction, deployment, and demo steps is the same, but implementations require certain modifications. This makes the development process very fragmented, and the resulting code redundancy is disastrous for long-term code maintenance.

DLK is a toolkit that uses lightning's Trainer and intc as the configuration management system, integrating model training, parameter (architecture) search, model prediction, model deployment, and demo functionalities into one. For implementing these functions for the same model, only one set of code is required, greatly reducing development and maintenance costs.

At the same time, as a universal training framework, DLK's various training techniques and enhancement methods can also be conveniently used for different models. For this reason, DLK includes many useful components.

In addition to the basic components, DLK also provides rich examples for major tasks, with more examples gradually being added.

Install

pip install dlk == 0.1.0

# or clone this repo and cd to the project root dir
pip install .

Demo

Below are some examples developed based on dlk:

NOTE: Since I currently only have a personal PC with one AMD Radeon VII 16G GPU and 32GB of memory, the computing power is very limited. Therefore, many parameters in the examples here have not been optimized to SOTA.

Grid Search

dlk, based on intc, also provides parameter search capabilities. And intc's _search is not limited to numerical parameter search; it can also search the entire module. Therefore, dlk actually also has module-level architecture search capabilities.

An example of searching for hyperparameters is provided in ./examples/grid_search_exp.

Task Demo

Usage and Features

Usage

Generally, a common dlk development task consists of two pipelines: data preprocessing pipeline and model inference pipeline. In fact, these two steps can be placed in the same pipeline. However, most tasks in the current examples require reuse of preprocessed data, so two pipelines are used.

The built-in entry point for the data preprocessing pipeline is dlk.preprocess.Process. We need to write a process.jsonc config file to configure the preprocessing process (the training, inference, and deploy processes all reuse the same file, so there are different settings for different stages in the configuration file) and initialize Process to input the data and execute run to output the preprocessed data as required.

The built-in entry point for the model training pipeline is dlk.train.Train. We need to write a fit.jsonc config file to configure model training (the inference and deploy processes also reuse this file), use the configuration file to initialize Train, and then execute run to obtain the trained model.

For demos, we only need to import the same process.jsonc and fit.jsonc used in the training process, along with the trained model (saved by the checkpoint callback component).

Model deployment only requires instantiating dlk.server.Server, distributing it to the corresponding server, and accepting single or batch data through Server.fit (TODO: Example).

Module Registration

DLK depends on two registration systems: one is the config registration cregister of intc, and the other is the module registration of DLK itself. The registration principles are consistent, registering a module with module_type and module_name as the key into the registry. The reason for choosing a two-layer naming scheme as the key is because it is more convenient to distinguish different module types.

Taking dlk.nn.layer.embedding.static as an example, we register StaticEmbeddingConfig as StaticEmbedding's config with ("embedding", "static") as the key into intc's cregister, and we register StaticEmbedding with the same key into DLK's module registry register.

The advantage of using registries is that we don't need to be concerned about where a specific class is implemented. We just need to know the registered name to directly obtain this class, which makes it very convenient to extend embedding types in any location. This is important for extending DLK in our own projects, and module registration is equally important for intc. Given the registered name of StaticEmbedding, obtaining this module is very simple; you can just use register.get("embedding", "static"), without needing to know its actual storage location (cregister also has the same function).

Some Built-in Modules Introduction

Callbacks

dlk's Trainer is implemented based on lightning.Trainer. Therefore, dlk can also use the callbacks provided by lightning. dlk.callback contains some commonly used callbacks.

Adversarial Training

Adversarial Training is a common technique for improving model performance. DLK includes some commonly used adv methods for embedding (dlk.adv_method). ./examples/adv_exp is an example of usage.

Complex Training Control

dlk.scheduler module provides various training schedulers, and multi_loss in dlk.nn.loss module also provides the ability to freely control various losses for multiple losses.

Text Generation

DLK also implements various token_sample methods, inspired by fairseq, providing powerful control capabilities for text generation.

More Documentation

TODO