zword

安装方法如下，请用 python3

pip install zword

使用有问题请到 gitee.com/znlp/zword/issues 发帖。

从维基百科抽取中文语料

维基百科语料下载地址 : dumps.wikimedia.org/zhwiki

有很多链接，下载比如 https://dumps.wikimedia.org/zhwiki/20200701/zhwiki-20200701-pages-articles.xml.bz2

下载后运行类似如下的命令来抽取中文语料

wiki_txt /share/wiki/zhwiki-20200701-pages-articles.xml.bz2

小技巧：维基百科打包打包很大，但是不需要完全下载也可以运行以上命令（会报错，但能部分输出，方便开发）

会在bz2的同目录输出两个文件

条目正文：zhwiki-20200701-pages-articles.title.txt.zd
条目标题：zhwiki-20200701-pages-articles.txt.zd

这两个文件是Zstandard压缩后的纯文本文件 ( 参见 Zstandard：一种新的无损压缩算法 )

使用本软件包附带的 zdcat 命令可以查看，比如：

zdcat /share/wiki/zhwiki-20200701-pages-articles.title.txt.zd

在条目正文中，条目的标题以 "➜ " 开头。

在程序中读取zd文件，可用如下方法

from zword import zd

with zd.open(
  "/share/wiki/zhwiki-20200701-pages-articles.txt.zd"
) as f:
  for i in f:
    print(i)

特别感谢

代码改编自《获取并处理中文维基百科语料 - 科学空间|Scientific Spaces》

Metadata

Release files for zword 0.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for zword 0.0.4
File	Size	Uploaded	Provenance
zword-0.0.4.tar.gz	3.2 kB	Jul 21, 2020	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for zword 0.0.4
File	Interpreter	ABI	Platform	Reset
zword-0.0.4-py3-none-any.whl 9.5 kB Jul 21, 2020 Without provenance	Python 3	none	any	Details

Total release size: 12.6 kB

Release files / zword-0.0.4.tar.gz

Download URL	zword-0.0.4.tar.gz
Size	3.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`b89e581321e205840149108c5d386e5f31dd3a887bc27612d9eb61a904a8d7e2`
BLAKE2b-256 checksum How to use checksums	`43634813740e7557d5eafe0c606b005342f42f5f16a1ae39fb8ef12bf9c5d975`
Upload date	Jul 21, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8`

Release files / zword-0.0.4-py3-none-any.whl

Download URL	zword-0.0.4-py3-none-any.whl
Size	9.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`d798de61a9f27e95b8d266b3ed98d6bf9bda8f07b281ee3d808a7df703267df2`
BLAKE2b-256 checksum How to use checksums	`1312e8027aa21f0479461e1d3dae8779a3661fdbe60986961a5ceafed15e8beb`
Upload date	Jul 21, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8`

Security for zword 0.0.4

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.