Skip to main content

Tokenizer POS-tagger and Dependency-parser for Classical Chinese

Project description

Current PyPI packages

UD-Kanbun

Tokenizer, POS-Tagger, and Dependency-Parser for Classical Chinese Texts (漢文/文言文), working on Universal Dependencies.

Basic usage

>>> import udkanbun
>>> lzh=udkanbun.load()
>>> s=lzh("不入虎穴不得虎子")
>>> print(s)
# text = 不入虎穴不得虎子
1			ADV	v,副詞,否定,無界	Polarity=Neg	2	advmod	_	Gloss=not|SpaceAfter=No
2			VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
3			NOUN	n,名詞,主体,動物	_	4	nmod	_	Gloss=tiger|SpaceAfter=No
4			NOUN	n,名詞,固定物,地形	Case=Loc	2	obj	_	Gloss=cave|SpaceAfter=No
5			ADV	v,副詞,否定,無界	Polarity=Neg	6	advmod	_	Gloss=not|SpaceAfter=No
6			VERB	v,動詞,行為,得失	_	2	parataxis	_	Gloss=get|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	8	nmod	_	Gloss=tiger|SpaceAfter=No
8			NOUN	n,名詞,,関係	_	6	obj	_	Gloss=child|SpaceAfter=No

>>> t=s[1]
>>> print(t.id,t.form,t.lemma,t.upos,t.xpos,t.feats,t.head.id,t.deprel,t.deps,t.misc)
1   ADV v,副詞,否定,無界 Polarity=Neg 2 advmod _ Gloss=not|SpaceAfter=No

>>> print(s.kaeriten())
虎穴虎子

>>> print(s.to_tree())
 <════╗   advmod
 ═══╗═╝═╗ root
 <     nmod
 ═╝<    obj
 <════╗  advmod
 ═══╗═╝< parataxis
 <      nmod
 ═╝<     obj

>>> f=open("trial.svg","w")
>>> f.write(s.to_svg())
>>> f.close()

trial.svg udkanbun.load() has three options udkanbun.load(MeCab=True,Danku=False). By default, the UD-Kanbun pipeline uses MeCab for tokenizer and POS-tagger, then uses UDPipe for dependency-parser. With the option MeCab=False the pipeline uses UDPipe for all through the processing. With the option Danku=True the pipeline tries to segment sentences automatically.

udkanbun.UDKanbunEntry.to_tree() has an option to_tree(BoxDrawingWidth=2) for old terminals, whose Box Drawing characters are "fullwidth". to_tree(kaeriten=True,Japanese=True) is convenient for Japanese users.

You can simply use udkanbun on the command line:

echo 不入虎穴不得虎子 | udkanbun

Usage via spaCy

If you have already installed spaCy 2.1.0 or later, you can use UD-Kanbun via spaCy Language pipeline.

>>> import udkanbun.spacy
>>> lzh=udkanbun.spacy.load()
>>> d=lzh("不入虎穴不得虎子")
>>> print(type(d))
<class 'spacy.tokens.doc.Doc'>
>>> print(udkanbun.spacy.to_conllu(d))
# text = 不入虎穴不得虎子
1			ADV	v,副詞,否定,無界	_	2	advmod	_	Gloss=not|SpaceAfter=No
2			VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
3			NOUN	n,名詞,主体,動物	_	4	nmod	_	Gloss=tiger|SpaceAfter=No
4			NOUN	n,名詞,固定物,地形	_	2	obj	_	Gloss=cave|SpaceAfter=No
5			ADV	v,副詞,否定,無界	_	6	advmod	_	Gloss=not|SpaceAfter=No
6			VERB	v,動詞,行為,得失	_	2	parataxis	_	Gloss=get|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	8	nmod	_	Gloss=tiger|SpaceAfter=No
8			NOUN	n,名詞,,関係	_	6	obj	_	Gloss=child|SpaceAfter=No

>>> t=d[0]
>>> print(t.i+1,t.orth_,t.lemma_,t.pos_,t.tag_,t.head.i+1,t.dep_,t.whitespace_,t.norm_)
1   ADV v,副詞,否定,無界 2 advmod  not

Installation for Linux

Tar-ball is available for Linux, and is installed by default when you use pip:

pip install udkanbun

Installation for Cygwin

Make sure to get gcc-g++ python37-pip python37-devel packages, and then:

pip3.7 install udkanbun

Use python3.7 command in Cygwin instead of python.

Installation for Jupyter Notebook (Google Colaboratory)

!pip install udkanbun

Try notebook for Google Colaboratory.

Author

Koichi Yasuoka (安岡孝一)

References

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

udkanbun-3.4.5.tar.gz (19.8 MB view details)

Uploaded Source

File details

Details for the file udkanbun-3.4.5.tar.gz.

File metadata

  • Download URL: udkanbun-3.4.5.tar.gz
  • Upload date:
  • Size: 19.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.2

File hashes

Hashes for udkanbun-3.4.5.tar.gz
Algorithm Hash digest
SHA256 c39a491f9a9ed3de3c797ce729ae09937c57570a70715d92d90600a7fb1d0f7e
MD5 f2bf663110b9c46c26b7d9b238513616
BLAKE2b-256 b728f6c233db1274c7b3ecd9104bdbbe23d1ea86e1fc076f205216324728bfd6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page