nlpaug
This python library helps you with augmenting nlp for your machine learning projects. Visit this introduction to understand about Data Augmentation in NLP. Augmenter is the basic element of augmentation while Flow is a pipeline to orchestra multi augmenter together.
Features
- Generate synthetic data for improving model performance without manual effort
- Simple, easy-to-use and lightweight library. Augment data in 3 lines of code
- Plug and play to any neural network frameworks (e.g. PyTorch, TensorFlow)
- Support textual and audio input
Textual Data Augmentation Example
Acoustic Data Augmentation Example
| Section | Description |
|---|---|
| Quick Demo | How to use this library |
| Augmenter | Introduce all available augmentation methods |
| Installation | How to install this library |
| Recent Changes | Latest enhancement |
| Extension Reading | More real life examples or researchs |
| Reference | Refernce of external resources such as data or model |
Quick Demo
- Example of Augmentation for Textual Inputs
- Example of Augmentation for Spectrogram Inputs
- Example of Augmentation for Audio Inputs
- Example of Orchestra Multiple Augmenters
- How to train TF-IDF model
- How to create custom augmentation
- API Documentation
Augmenter
| Augmenter | Target | Augmenter | Action | Description |
|---|---|---|---|---|
| Textual | Character | KeyboardAug | substitute | Simulate keyboard distance error |
| Textual | OcrAug | substitute | Simulate OCR engine error | |
| Textual | RandomAug | insert, substitute, swap, delete | Apply augmentation randomly | |
| Textual | Word | AntonymAug | substitute | Substitute opposite meaning word according to WordNet antonym |
| Textual | ContextualWordEmbsAug | insert, substitute | Feeding surroundings word to BERT, DistilBERT, RoBERTa or XLNet language model to find out the most suitlabe word for augmentation | |
| Textual | RandomWordAug | swap, delete | Apply augmentation randomly | |
| Textual | SpellingAug | substitute | Substitute word according to spelling mistake dictionary | |
| Textual | SplitAug | split | Split one word to two words randomly | |
| Textual | SynonymAug | substitute | Substitute similar word according to WordNet/ PPDB synonym | |
| Textual | TfIdfAug | insert, substitute | Use TF-IDF to find out how word should be augmented | |
| Textual | WordEmbsAug | insert, substitute | Leverage word2vec, GloVe or fasttext embeddings to apply augmentation | |
| Textual | Sentence | ContextualWordEmbsForSentenceAug | insert | Insert sentence according to XLNet, GPT2 or DistilGPT2 prediction |
| Signal | Audio | CropAug | delete | Delete audio's segment |
| Signal | LoudnessAug | substitute | Adjust audio's volume | |
| Signal | MaskAug | substitute | Mask audio's segment | |
| Signal | NoiseAug | substitute | Inject noise | |
| Signal | PitchAug | substitute | Adjust audio's pitch | |
| Signal | ShiftAug | substitute | Shift time dimension forward/ backward | |
| Signal | SpeedAug | substitute | Adjust audio's speed | |
| Signal | VtlpAug | substitute | Change vocal tract | |
| Signal | Spectrogram | FrequencyMaskingAug | substitute | Set block of values to zero according to frequency dimension |
| Signal | TimeMaskingAug | substitute | Set block of values to zero according to time dimension |
Flow
| Augmenter | Augmenter | Description |
|---|---|---|
| Pipeline | Sequential | Apply list of augmentation functions sequentially |
| Pipeline | Sometimes | Apply some augmentation functions randomly |
Installation
The library supports python 3.5+ in linux and window platform.
To install the library:
pip install nlpaug numpy matplotlib python-dotenv
or install the latest version (include BETA features) from github directly
pip install git+https://github.com/makcedward/nlpaug.git numpy matplotlib python-dotenv
If you use ContextualWordEmbsAug or ContextualWordEmbsForSentenceAug, install the following dependencies as well
pip install torch>=1.2.0 transformers>=2.0.0
If you use AntonymAug, SynonymAug, install the following dependencies as well
pip install nltk>=3.4.5
If you use WordEmbsAug (word2vec, glove or fasttext), downloading pre-trained model first
from nlpaug.util.file.download import DownloadUtil
DownloadUtil.download_word2vec(dest_dir='.') # Download word2vec model
DownloadUtil.download_glove(model_name='glove.6B', dest_dir='.') # Download GloVe model
DownloadUtil.download_fasttext(model_name='wiki-news-300d-1M', dest_dir='.') # Download fasttext model
If you use any one of audio augmenter, install the following dependencies as well
pip install librosa>=0.7.1
Recent Changes
**0.0.11 Dec, 2019
- Support color noise (pink, blue, red and violet noise) in audio's NoiseAug
- Support given background noise in audio's NoiseAug
- Support inject noise to portion of audio only in audio's NoiseAug
- Introduce
zone,coverageto all audio augmenter. Support only augmented portion of audio input - Add VTLP augmentation methods (Audio's augmenter)
- Adopt latest transformer's interface #59
- Support RoBERTa (including DistilRoBERTa) and DistilBERT (ContextualWordEmbsAug)
- Support DistilGPT2 (ContextualWordEmbsForSentenceAug)
- Fix librosa hard dependency #62
- Introduce
optimizeattribute ContextualWordEmbsForSentenceAug #63 - Optimize word selection for ContextualWordEmbsAug and ContextualWordEmbsForSentenceAug (Speed up around 30%)
- Add retry mechanism into ContextualWordEmbsAug insert action #68
See changelog for more details.
Extension Reading
- Data Augmentation library for Text
- Does your NLP model able to prevent adversarial attack?
- How does Data Noising Help to Improve your NLP Model?
- Data Augmentation library for Speech Recognition
- Data Augmentation library for Audio
- Unsupervied Data Augmentation
Reference
This library uses data (e.g. capturing from internet), research (e.g. following augmenter idea), model (e.g. using pre-trained model) See data source for more details.
1 maintainer
Metadata
1 maintainer
Release files for nlpaug 0.0.11
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| nlpaug-0.0.11.tar.gz | 43.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| nlpaug-0.0.11-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 137.3 kB
Release files / nlpaug-0.0.11.tar.gz
| Download URL | nlpaug-0.0.11.tar.gz |
|---|---|
| Size | 43.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
142a862debb8d628a1e892b4570e8a6c5dab22679b871a6c3348ef427e97d053
|
|
BLAKE2b-256 checksum How to use checksums |
3c54d47b83216c39b28b914cbbe8c6b458daadd09a4491189d7a75d064b60340
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.6.4
|
Release files / nlpaug-0.0.11-py3-none-any.whl
| Download URL | nlpaug-0.0.11-py3-none-any.whl |
|---|---|
| Size | 94.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
10134ea8ea963be6319561d4822245123e2e3e5eaf972e4b3a568f0290c3e61e
|
|
BLAKE2b-256 checksum How to use checksums |
f4379765ab8ddc9aa9364d1bbbdf87166e1861c02f0d0ad716ca31514d6d0123
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.6.4
|
