Skip to main content

PyAutoCorpus

A python interface to the excellent AutoCorpus library.

Right now, it only supports the wiki markup textify function, which strips out markup. From my benchmarks, this ends up being ~40x faster than methods to strip markup using other libraries:

mwparserfromhell 0.208 sec/doc
wikitextparser 0.215 sec/doc
pyautocorpus 0.005 sec/doc

where:

  • mwparserfromhell is mwparserfromhell.parse(x).strip_code()
  • wikitextparser is wikitextparser.parse(x).plain_text()
  • pyautocorpus is pyautocorpus.Textifier().textify(x)

Installing

From pypi:

pip install pyautocorpus

From source:

Be sure to clone recursively:

git clone --recursive https://github.com/seanmacavaney/pyautocorpus.git

You will first need the pcre library installed.

python setup.py install

Usage

Example:

import pyautocorpus
textifier = pyautocorpus.Textifier()
textifier.textify("==Wiki Marked up text==\n [[Some Page|link text]] example.")
'Wiki Marked up text\n\n\n link text example.'

Known issues

  • Windows is not yet supported

Credits

AutoCorpus

Contributors to this repository:

  • Sean MacAvaney (University of Glasgow)
  • Thomas Jänich (University of Glasgow)

Metadata

Release files for pyautocorpus 0.1.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyautocorpus 0.1.20
File Size Uploaded Provenance Actions
pyautocorpus-0.1.20.tar.gz 11.4 kB Without provenance Details for pyautocorpus-0.1.20.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for pyautocorpus 0.1.20
File Actions
pyautocorpus-0.1.20-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.15 CPython 3.15 free-threading Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 Details for pyautocorpus-0.1.20-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
pyautocorpus-0.1.20-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 Details for pyautocorpus-0.1.20-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
pyautocorpus-0.1.20-cp37-abi3-win_amd64.whl CPython 3.7 abi3 Windows x86-64 Details for pyautocorpus-0.1.20-cp37-abi3-win_amd64.whl
pyautocorpus-0.1.20-cp37-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.7 abi3 Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 Details for pyautocorpus-0.1.20-cp37-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
pyautocorpus-0.1.20-cp37-abi3-macosx_10_9_universal2.whl CPython 3.7 abi3 macOS 10.9+ universal2 (ARM64, x86-64) Details for pyautocorpus-0.1.20-cp37-abi3-macosx_10_9_universal2.whl

Total release size: 1.2 MB

Release files / pyautocorpus-0.1.20.tar.gz

Download URL pyautocorpus-0.1.20.tar.gz
Size 11.4 kB
Tags Source
SHA-256 checksum
How to use checksums
240861164b15369eb75333049845851e82c2f592d5069fc8b82fd5d450b94ccc
BLAKE2b-256 checksum
How to use checksums
954bde4ee08623dc0355202ce07bb284a8acc599e4f24baf3fd7c19b0c71d6d1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.2

Release files / pyautocorpus-0.1.20-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL pyautocorpus-0.1.20-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 374.0 kB
Tags CPython 3.15 CPython 3.15 free-threading Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
18f90afaa8db0361e2f95782f777e30cbab195dd85cbd5164504a5f90452bcbf
BLAKE2b-256 checksum
How to use checksums
e578ff90d5ff3981f950b5b1aedc540e4e169eef0e68781cebed978f665edce7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.2

Release files / pyautocorpus-0.1.20-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL pyautocorpus-0.1.20-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 373.9 kB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
9f30798929b77a7312a411346dc270c0b74e888ad97403ef24217916064cb9ff
BLAKE2b-256 checksum
How to use checksums
fecdac4f0fc570fa3ae7ed80e13bfe01a9af8e60c8a4a9fed0b33cd6aa472d1c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.2

Release files / pyautocorpus-0.1.20-cp37-abi3-win_amd64.whl

Download URL pyautocorpus-0.1.20-cp37-abi3-win_amd64.whl
Size 6.7 kB
Tags CPython 3.7 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
8b42d0b0c39ea1e73058ebfee6c13e7f97ba49677b7b08831352a02c185fdfc3
BLAKE2b-256 checksum
How to use checksums
d166c17eb2a4ad3be1ffefcc04f4849dac9cded0e6c9a4b9b13fcd5d5826930b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.13

Release files / pyautocorpus-0.1.20-cp37-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL pyautocorpus-0.1.20-cp37-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 369.1 kB
Tags CPython 3.7 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
c57e53cd2bd1f8b1c7c173b94979590a3983e7fe7120d0f58337c86599ebfbb8
BLAKE2b-256 checksum
How to use checksums
eac0cb7d979b463d5df528a79aae13d3d60df4e9a19bb452c28a81494eba518f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.2

Release files / pyautocorpus-0.1.20-cp37-abi3-macosx_10_9_universal2.whl

Download URL pyautocorpus-0.1.20-cp37-abi3-macosx_10_9_universal2.whl
Size 44.4 kB
Tags CPython 3.7 abi3 macOS 10.9+ universal2 (ARM64, x86-64)
SHA-256 checksum
How to use checksums
52b6ef5bd534c25ebfeb4cc7d966d67f47e0a3dbfb5be4428fa9cb2484c89265
BLAKE2b-256 checksum
How to use checksums
6e40e8830bccaeb9db16c81d636b039370747976ac3d5bb580ba25c95ca48fb9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.13

Security for pyautocorpus 0.1.20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page