Sentence Segmentation with sequece tagging
Project description
DeepSegment: A sentence segmenter that actually works!
Note: For the original implementation please use the "master" branch of this repo.
The Demo for deepsegment (en) + deeppunct is available at http://bpraneeth.com/projects/deeppunct
Installation:
pip install --upgrade deepsegment
Supported languages:
en - english (Trained on data from various sources)
fr - french (Only Tatoeba data)
it - italian (Only Tatoeba data)
Usage:
from deepsegment import DeepSegment
# The default language is 'en'
segmenter = DeepSegment('en')
segmenter.segment('I am Batman i live in gotham')
# ['I am Batman', 'i live in gotham']
Using with tf serving docker image
docker pull bedapudi6788/deepsegment_en:v2
docker run -d -p 8500:8500 bedapudi6788/deepsegment_en:v2
from deepsegment import DeepSegment
# The default language is 'en'
segmenter = DeepSegment('en', tf_serving=True)
segmenter.segment('I am Batman i live in gotham')
# ['I am Batman', 'i live in gotham']
Finetuning DeepSegment
Since one-size will never fit all, finetuning deepsegment's default models with your own data is encouraged.
from deepsegment import finetune, generate_data
x, y = generate_data(['my name', 'is batman', 'who are', 'you'], n_examples=10000)
vx, vy = generate_data(['my name', 'is batman'])
# NOTE: name, epochs, batch_size, lr are optional arguments.
finetune('en', x, y, vx, vy, name='finetuned_model_name', epochs=number_of_epochs, batch_size=batch_size, lr=learning_rate)
Using with a finetuned checkpoint
from deepsegment import DeepSegment
segmenter = DeepSegment('en', checkpoint_name='finetuned_model_name')
Training deepsegment on custom data: https://colab.research.google.com/drive/1CjYbdbDHX1UmIyvn7nDW2ClQPnnNeA_m
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
deepsegment-2.3.0.tar.gz
(7.2 kB
view hashes)
Built Distribution
Close
Hashes for deepsegment-2.3.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd46f0f89873c5119c60484731f1ba40a7e5726223313dea670425a6f740b0ef |
|
MD5 | e9302504730dee69538c9a24cc8cc4ad |
|
BLAKE2b-256 | a3e0bb67170d127970e94aee8924009b70cad03899654a0d0ccab9038db0d19b |