Vietnamese NLP Toolkit
Project description
Under The Sea - Vietnamese NLP Toolkit
underthesea is a suite of open source Python modules, data sets and tutorials supporting research and development in Vietnamese Natural Language Processing.
Free software: GNU General Public License v3
Documentation: https://underthesea.readthedocs.io
Live demo: underthesea app
Facebook Page: https://www.facebook.com/undertheseanlp/
Installation
Install dependencies
$ pip install Cython numpy scipy sklearn fasttext python-crfsuite
To install underthesea, simply:
$ pip install underthesea
$ underthesea data
✨🍰✨
Satisfaction, guaranteed.
Usage
1. Corpus
Collection of Vietnamese corpus
2. Word Segmentation
Vietnamese Word Segmentation using Conditional Random Fields
>>> # -*- coding: utf-8 -*-
>>> from underthesea import word_sent
>>> sentence = u"Chúng ta thường nói đến Rau sạch, Rau an toàn để phân biệt với các rau bình thường bán ngoài chợ."
>>> word_sent(sentence)
[u"Chúng ta", u"thường", u"nói", u"đến", u"Rau sạch", u",", u"Rau", u"an toàn", u"để", u"phân biệt", u"với",
u"các", u"rau", u"bình thường", u"bán", u"ngoài", u"chợ", u"."]
>>> word_sent(sentence, format="text")
u'Chúng_ta thường nói đến Rau_sạch , Rau an_toàn để phân_biệt với các rau bình_thường bán ngoài chợ .'
3. POS Tagging
Vietnamese Part of Speech Tagging using Conditional Random Fields
>>> # -*- coding: utf-8 -*-
>>> from underthesea import pos_tag
>>> text = u"Chợ thịt chó nổi tiếng ở TP Hồ Chí Minh bị truy quét"
>>> pos_tag(text)
[(u'Chợ', 'N'),
(u'thịt', 'N'),
(u'chó', 'N'),
(u'nổi tiếng', 'A'),
(u'ở', 'E'),
(u'TP HCM', 'Np'),
(u'bị', 'V'),
(u'truy quét', 'V')]
4. Chunking
Vietnamese Chunking using Conditional Random Fields
>>> # -*- coding: utf-8 -*-
>>> from underthesea import chunk
>>> text = u"Bác sĩ bây giờ có thể thản nhiên báo tin bệnh nhân bị ung thư?"
>>> chunk(text)
[(u'Bác sĩ', 'N', 'B-NP'),
(u'bây giờ', 'P', 'I-NP'),
(u'có thể', 'R', 'B-VP'),
(u'thản nhiên', 'V', 'I-VP'),
(u'báo tin', 'N', 'B-NP'),
(u'bệnh nhân', 'N', 'I-NP'),
(u'bị', 'V', 'B-VP'),
(u'ung thư', 'N', 'I-VP'),
(u'?', 'CH', 'O')]
5. Text Classification
Vietnamese Text Classification using Fasttext
>>> # -*- coding: utf-8 -*-
>>> from underthesea import classify
>>> classify("HLV đầu tiên ở Premier League bị sa thải sau 4 vòng đấu")
['The thao']
>>> classify("Hội đồng tư vấn kinh doanh Asean vinh danh giải thưởng quốc tế")
['Kinh doanh']
>>> classify("Đánh giá “rạp hát tại gia” Samsung Soundbar Sound+ MS750")
['Vi tinh']
Up Coming Features
Sentiment Analysis
Word Representation (Word Representation Experiments)
Dependency Parsing
Named Entity Recognition
Contributing
Do you want to contribute with underthesea development? Great! Please read more details at CONTRIBUTING.rst.
History
1.1.4 (2017-09-12)
New feature: text classification
[bug] Fix Text error
[doc] Add facebook link
1.1.3 (2017-08-30)
Add live demo: https://underthesea.herokuapp.com/
1.1.2 (2017-08-22)
Add dictionary
1.1.1 (2017-07-05)
Support Python 3
Refactor feature_engineering code
1.1.0 (2017-05-30)
Add chunking feature
Add pos_tag feature
Add word_sent feature, fix performance
Add Corpus class
Add Transformer classes
Integrated with dictionary of Ho Ngoc Duc
Add travis-CI, auto build with PyPI
1.0.0 (2017-03-01)
First release on PyPI.
First release on Readthedocs
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for underthesea-1.1.4rc2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ff64a094bb0e29ffdb5540c6628280c90817ac5de281f4469602decb4375a8d |
|
MD5 | 594e3a48636ac5c7f63c5362a9d4c7b0 |
|
BLAKE2b-256 | d2e39d0e99f84acd7ab1d41f40d1d4e72262882f3b4c4de905b5586c8a6724e8 |