split-words
CharSplit repacked with poetry, published as pypi split-words--
all credit goes to the original author.
pip install split-words
# replace charsplit with split_words in the following, e.g.
from split_words import Splitter
...
CharSplit - An ngram-based compound splitter for German
Splits a German compound into its body and head, e.g.
Autobahnraststätte -> Autobahn - Raststätte
Implementation of the method decribed in the appendix of the thesis:
Tuggener, Don (2016). Incremental Coreference Resolution for German. University of Zurich, Faculty of Arts.
TL;DR: The method calculates probabilities of ngrams occurring at the beginning, end and in the middle of words and identifies the most likely position for a split.
The method achieves ~95% accuracy for head detection on the Germanet compound test set.
A model is provided, trained on 1 Mio. German nouns from Wikipedia.
Usage
Train a new model:
training.py --input_file --output_file
from command line, where input_file contains one word (noun) per line and output_file is a json file with computed n-gram probabilities.
Compound splitting
In python
>> from charsplit import Splitter
>> splitter = Splitter()
>> splitter.split_compound("Autobahnraststätte")
returns a list of all possible splits, ranked by their score, e.g.
[(0.7945872450631273, 'Autobahn', 'Raststätte'),
(-0.7143290887876655, 'Auto', 'Bahnraststätte'),
(-1.1132332878581173, 'Autobahnrast', 'Stätte'), ...]
By default, Splitter uses the data from the file charsplit/ngram_probs.json. If you retrained the model, you may specify a custom file with
>> splitter = Splitter(ngram_path=<json_data_file_with_ngram_probs>)
Metadata
Release files for split-words 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| split_words-0.1.3.tar.gz | 12.9 MB | Without provenance | Details for split_words-0.1.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| split_words-0.1.3-py3-none-any.whl | Python 3 | none | any | Details for split_words-0.1.3-py3-none-any.whl |
Total release size: 26.3 MB
Release files / split_words-0.1.3.tar.gz
| Download URL | split_words-0.1.3.tar.gz |
|---|---|
| Size | 12.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7dbb69a85a722355c32de8520fe4d083aa9b4020cbe2cb44175758260fdadeb0
|
|
BLAKE2b-256 checksum How to use checksums |
6a28f9c3c5e886b9e2a0088e50c0f87c3080b345be46bc7bf540e18ae5ab0e94
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.8.10 Windows/10
|
Release files / split_words-0.1.3-py3-none-any.whl
| Download URL | split_words-0.1.3-py3-none-any.whl |
|---|---|
| Size | 13.3 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2bd73849a7786a2a1892d87a2cde2ecf78b69bcdc9146a2f38680e74d3908274
|
|
BLAKE2b-256 checksum How to use checksums |
0066c753381236dd46ccecb423292ae16e44768588ac922093868291080448c1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.8.10 Windows/10
|
Security for split-words 0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.