Skip to main content

Tokenizer POS-tagger and Dependency-parser for Classical Chinese

Project description

Current PyPI packages

UD-Kanbun

Tokenizer, POS-Tagger, and Dependency-Parser for Classical Chinese Texts (漢文/文言文), working on Universal Dependencies.

Basic usage

>>> import udkanbun
>>> lzh=udkanbun.load()
>>> s=lzh("不入虎穴不得虎子")
>>> print(s)
# text = 不入虎穴不得虎子
1			ADV	v,副詞,否定,無界	Polarity=Neg	2	advmod	_	Gloss=not|SpaceAfter=No
2			VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
3			NOUN	n,名詞,主体,動物	_	4	nmod	_	Gloss=tiger|SpaceAfter=No
4			NOUN	n,名詞,固定物,地形	Case=Loc	2	obj	_	Gloss=cave|SpaceAfter=No
5			ADV	v,副詞,否定,無界	Polarity=Neg	6	advmod	_	Gloss=not|SpaceAfter=No
6			VERB	v,動詞,行為,得失	_	2	parataxis	_	Gloss=get|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	8	nmod	_	Gloss=tiger|SpaceAfter=No
8			NOUN	n,名詞,,関係	_	6	obj	_	Gloss=child|SpaceAfter=No

>>> t=s[1]
>>> print(t.id,t.form,t.lemma,t.upos,t.xpos,t.feats,t.head.id,t.deprel,t.deps,t.misc)
1   ADV v,副詞,否定,無界 Polarity=Neg 2 advmod _ Gloss=not|SpaceAfter=No

>>> print(s.kaeriten())
虎穴虎子

>>> print(s.to_tree())
 <════╗   advmod
 ═══╗═╝═╗ root
 <     nmod
 ═╝<    obj
 <════╗  advmod
 ═══╗═╝< parataxis
 <      nmod
 ═╝<     obj

>>> f=open("trial.svg","w")
>>> f.write(s.to_svg())
>>> f.close()

trial.svg udkanbun.load() has three options udkanbun.load(MeCab=True,Danku=False). By default, the UD-Kanbun pipeline uses MeCab for tokenizer and POS-tagger, then uses UDPipe for dependency-parser. With the option MeCab=False the pipeline uses UDPipe for all through the processing. With the option Danku=True the pipeline tries to segment sentences automatically.

udkanbun.UDKanbunEntry.to_tree() has an option to_tree(BoxDrawingWidth=2) for old terminals, whose Box Drawing characters are "fullwidth". to_tree(kaeriten=True,Japanese=True) is convenient for Japanese users.

You can simply use udkanbun on the command line:

echo 不入虎穴不得虎子 | udkanbun

Usage via spaCy

If you have already installed spaCy 2.1.0 or later, you can use UD-Kanbun via spaCy Language pipeline.

>>> import udkanbun.spacy
>>> lzh=udkanbun.spacy.load()
>>> d=lzh("不入虎穴不得虎子")
>>> print(type(d))
<class 'spacy.tokens.doc.Doc'>
>>> print(udkanbun.spacy.to_conllu(d))
# text = 不入虎穴不得虎子
1			ADV	v,副詞,否定,無界	_	2	advmod	_	Gloss=not|SpaceAfter=No
2			VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
3			NOUN	n,名詞,主体,動物	_	4	nmod	_	Gloss=tiger|SpaceAfter=No
4			NOUN	n,名詞,固定物,地形	_	2	obj	_	Gloss=cave|SpaceAfter=No
5			ADV	v,副詞,否定,無界	_	6	advmod	_	Gloss=not|SpaceAfter=No
6			VERB	v,動詞,行為,得失	_	2	parataxis	_	Gloss=get|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	8	nmod	_	Gloss=tiger|SpaceAfter=No
8			NOUN	n,名詞,,関係	_	6	obj	_	Gloss=child|SpaceAfter=No

>>> t=d[0]
>>> print(t.i+1,t.orth_,t.lemma_,t.pos_,t.tag_,t.head.i+1,t.dep_,t.whitespace_,t.norm_)
1   ADV v,副詞,否定,無界 2 advmod  not

Installation for Linux

Tar-ball is available for Linux, and is installed by default when you use pip:

pip install udkanbun

Installation for Cygwin

Make sure to get gcc-g++ python37-pip python37-devel packages, and then:

pip3.7 install udkanbun

Use python3.7 command in Cygwin instead of python.

Installation for Jupyter Notebook (Google Colaboratory)

!pip install udkanbun

Try notebook for Google Colaboratory.

Author

Koichi Yasuoka (安岡孝一)

References

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for udkanbun, version 3.2.9
Filename, size File type Python version Upload date Hashes
Filename, size udkanbun-3.2.9.tar.gz (16.3 MB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page