flagai 1.0.0b1

FlagAI aims to help researchers and developers to freely train and test large-scale models for NLP tasks.

简体中文

---

FlagAI aims to help researchers and developers to freely train and test large-scale models for NLP tasks.

• Now it supports GLM, Bert, RoBerta, GPT2, T5 and models from Huggingface Transformers.

• It provides APIs to quickly download and use those pretrained models on a given text, fine-tune them on your own datasets and then share them with the community on our model hub.

• These models can be applied on Text, for tasks like text classification, information extraction, question answering, summarization, text generation, especially in Chinese.

• FlagAI is backed by the three most popular data/model parallel libraries — PyTorch/Deepspeed/Megatron-LM — with a seamless integration between them. Your can paralle your training/testing process with less than ten lines of code.

The code is partially based on Transformers and DeepSpeedExamples.

• Requirements and Installation
• Quick Started
  • Load model and tokenizer
  • Predictor
  • NER task
  • Title generation task
  • Semantic matching task
• Pretrained Models and examples
• Tutorials
• Learn More About FlagAI
• Contributing
• License

Requirements and Installation

• PyTorch version >= 1.8.0
• Python version >= 3.8
• For training/testing models on GPUs, you'll also need install CUDA and NCCL

To install FlagAI and develop locally:

git clone https://github.com/BAAI-WuDao/Sailing.git
python setup.py install

• [Optional] For faster training install NVIDIA's apex

git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

• [Optional] For ZeRO optimizers install DEEPSPEED

git clone https://github.com/microsoft/DeepSpeed
cd DeepSpeed
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 pip install -e .
ds_report # check the deespeed status

• [Tips] For single-node docker enviroments, we need to setup ports for your ssh. e.g., root@127.0.0.1 with port 7110

>>> vim ~/.ssh/config
Host 127.0.0.1
    Hostname 127.0.0.1
    Port 7110
    User root

• [Tips] For multi-node docker enviroments, generate ssh keys and copy the public key to all nodes (in ~/.ssh/)

>>> ssh-keygen -t rsa -C "[e-mail]"

Quick Start

We provide many models which are trained to perform different tasks. You can load these models by AutoLoader to make prediction.

Load model and tokenizer

We provide the AutoLoad class to load the model and tokenizer quickly, for example:

from flagai.auto_model.auto_loader import AutoLoader

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

auto_loader = AutoLoader(
    task_name="seq2seq",
    model_name="bert_title_generation_en"
)
model = auto_loader.get_model()
tokenizer = auto_loader.get_tokenizer()

This example is for the title_generation task, and you can also model other tasks by modifying the task_name. Then you can use the model and tokenizer to finetune or test.

Predictor

We provide the Predictor class to predict for different tasks, for example:

import torch
from flagai.model.predictor.predictor import Predictor
predictor = Predictor(model, tokenizer)
test_data = [
    "Four minutes after the red card, Emerson Royal nodded a corner into the path of the unmarked Kane at the far post, who nudged the ball in for his 12th goal in 17 North London derby appearances. Arteta's misery was compounded two minutes after half-time when Kane held the ball up in front of goal and teed up Son to smash a shot beyond a crowd of defenders to make it 3-0.The goal moved the South Korea talisman a goal behind Premier League top scorer Mohamed Salah on 21 for the season, and he looked perturbed when he was hauled off with 18 minutes remaining, receiving words of consolation from Pierre-Emile Hojbjerg.Once his frustrations have eased, Son and Spurs will look ahead to two final games in which they only need a point more than Arsenal to finish fourth.",
]

for text in test_data:
    print(
        predictor.predict_generate_beamsearch(text,
                                              out_max_length=50,
                                              beam_size=3))

Pretrained Models and examples

• Poetry generation with glm_large_ch
• Title Generation with RoBerta-WWM
• Semantic Matching with RoBerta-WWM
• NER with RoBerta-WWM
• Writing with GPT-2
• Title generation with T5
• Supported tasks

This session explains how the base NLP classes work, how you can load pre-trained models to tag your text, how you can embed your text with different word or document embeddings, and how you can train your own language models, sequence labeling models, and text classification models. Let us know if anything is unclear.

Tutorials

We provide a set of quick tutorials to get you started with the library:

• Tutorial 1: Basics
• Tutorial 2: Project structure
• Tutorial 3: Supported tokenizers
• Tutorial 4: Supported datasets
• Tutorial 5: Supported models
• Tutorial 6: Training a model
• Tutorial 7: AutoLoader
• Tutorial 8: Predictor

Learn More About FlagAI

• Datasets: supported datasets & PET integration.
• Setup enviroments for data/model parallel
• Three types of generation
• Using huggingface's t5-3b & tricks
• Transform a model into Megatron-LM version

Contributing

Thanks for your interest in contributing! There are many ways to get involved; start with our contributor guidelines and then check these open issues for specific tasks.

License

Copyright [2022] [BAAI]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.