Skip to main content

Ckip Segmenter

A Python client for the Chinese Word Segmentation System (see ckipsvr.iis.sinica.edu.tw) provided by Academia Sinica Chinese Knowledge and Information Processing (CKIP) Group. The core was adapted from amigcamel/PyCCS.

Installation

Simply run the following command:

pip install ckip-segmenter

If pip is not available, you can also download it manually from PyPI.

Note: Currently only Python 3+ is supported.

Usage

Summon a CkipSegmenter

In[1]:

from ckip import CkipSegmenter
segmenter = CkipSegmenter()

text = '詞是最小有意義且可以自由使用的語言單位。任何語言處理的系統都必須先能分辨文本中的詞才能進行進一步的處理'
corpus = [
    '詞是最小有意義且可以自由使用的語言單位',
    '任何語言處理的系統都必須先能分辨文本中的詞才能進行進一步的處理',
    '例如機器翻譯、語言分析、語言了解、資訊抽取',
    '因此中文自動分詞的工作成了語言處理不可或缺的技術',
    '基本上自動分詞多利用詞典中收錄的詞和文本做比對',
    '找出可能包含的詞,由於存在歧義的切分結果',
    '因此多數的中文分詞程式多討論如何解決分詞歧義的問題',
    '而較少討論如何處理詞典中未收錄的詞出現的問題(新詞如何辨認)',
]

The result object contains res, tok and pos

In[2]:

result = segmenter.seg(text)
# result.res is a list of tuples contain a token and its pos-tag.
print('result.res: {}\n'.format(result.res))

# result.tok and result.pos contains only tokens and pos-tags respectively.
print('result.tok: {}\n'.format(result.tok))
print('result.pos: {}\n'.format(result.pos))

Out[2]:

result.res: [('詞', 'Na'), ('是', 'SHI'), ('最', 'Dfa'), ('小', 'VH'), ('有', 'V_2'), ('意義', 'Na'), ('且', 'Cbb'), ('可以', 'D'), ('自由', 'VH'), ('使用', 'VC'), ('的', 'DE'), ('語言', 'Na'), ('單位', 'Na'), ('。', 'PERIODCATEGORY'), ('任何', 'Neqa'), ('語言', 'Na'), ('處理', 'VC'), ('的', 'DE'), ('系統', 'Na'), ('都', 'D'), ('必須', 'D'), ('先能', 'Nb'), ('分辨', 'VE'), ('文本', 'Nb'), ('中', 'Ng'), ('的', 'DE'), ('詞', 'Na'), ('才能', 'Na'), ('進行', 'VC'), ('進一步', 'D'), ('的', 'DE'), ('處理', 'VC')]

result.tok: ['詞', '是', '最', '小', '有', '意義', '且', '可以', '自由', '使用', '的', '語言', '單位', '。', '任何', '語言', '處理', '的', '系統', '都', '必須', '先能', '分辨', '文本', '中', '的', '詞', '才能', '進行', '進一步', '的', '處理']

result.pos: ['Na', 'SHI', 'Dfa', 'VH', 'V_2', 'Na', 'Cbb', 'D', 'VH', 'VC', 'DE', 'Na', 'Na', 'PERIODCATEGORY', 'Neqa', 'Na', 'VC', 'DE', 'Na', 'D', 'D', 'Nb', 'VE', 'Nb', 'Ng', 'DE', 'Na', 'Na', 'VC', 'D', 'DE', 'VC']

Using batch_seg for a list of text would be slightly faster

In[3]:

segmenter.batch_seg(corpus)

Out[3]:

8/8 collected.

[<SegResult  詞(Na) 是(SHI) 最(Dfa) 小(VH) 有(V_2) 意義(Na) 且(Cbb) 可以 ...>,
 <SegResult  任何(Neqa) 語言(Na) 處理(VC) 的(DE) 系統(Na) 都(D) 必須(D) 先能 ...>,
 <SegResult  例如(P) 機器(Na) 翻譯(VC) 、(PAUSECATEGORY) 語言(Na) 分析(VC ...>,
 <SegResult  因此(Cbb) 中文(Na) 自動(VH) 分詞(Na) 的(DE) 工作(Na) 成(VG) 了 ...>,
 <SegResult  基本(Na) 上(Ncd) 自動(VH) 分詞(Na) 多(D) 利用(VC) 詞典(Na) 中( ...>,
 <SegResult  找出(VC) 可能(D) 包含(VJ) 的(DE) 詞(Na) ,(COMMACATEGORY)  ...>,
 <SegResult  因此(Cbb) 多數(Neqa) 的(DE) 中文(Na) 分詞(Na) 程式(Na) 多(D)  ...>,
 <SegResult  而(Cbb) 較少(D) 討論(VE) 如何(D) 處理(VC) 詞典(Na) 中(Ng) 未(D ...>]

Metadata

Release files for ckip-segmenter 1.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for ckip-segmenter 1.0.2
File Interpreter ABI Platform Actions
ckip_segmenter-1.0.2-py3-none-any.whl Python 3 none any Details for ckip_segmenter-1.0.2-py3-none-any.whl

Release files / ckip_segmenter-1.0.2-py3-none-any.whl

Download URL ckip_segmenter-1.0.2-py3-none-any.whl
Size 7.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
932f2e297e85e9ddaa5a4510ed574a2da5db20faf58bae2f652169171c13f468
BLAKE2b-256 checksum
How to use checksums
f6b6f32a83f415581ae6b901bdcef0d9560e6b20f8f4fceb64bec7e1700f370c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for ckip-segmenter 1.0.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.2 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page