wiktionary-de-parser

A Python library (3.13+) that extracts structured data from German Wiktionary XML dumps: IPA, hyphenation, inflection tables, part-of-speech tags, lemma references, rhymes, and meanings.

Features

Streams compressed XML dumps memory-efficiently.
Yields one structured entry per language and part of speech (a single Wiktionary page often holds several).
Optional multiprocessing mode for full-dump throughput.

Installation

pip install wiktionary-de-parser

The project uses uv for development; any standard pip/PyPI install works for consumers.

Usage

Locating the dump file

from wiktionary_de_parser import WiktionaryDump

# Either point at an existing local file.
dump = WiktionaryDump(
    dump_file_path="path/to/dewiktionary-latest-pages-articles-multistream.xml.bz2"
)

# Or download into a directory on first call.
dump = WiktionaryDump(dump_dir_path="dumps/")
dump.download_dump()

Parsing entries (serial)

from wiktionary_de_parser import WiktionaryParser

parser = WiktionaryParser()

for page in dump.pages():
    if page.redirect_to or not page.wikitext:
        continue
    for entry in parser.entries(page):
        parsed = parser.parse(entry)
        if parsed.page_name == "Abend":
            print(parsed)

Parsing entries (parallel)

For full-dump runs use iter_parsed. XML iteration stays on the main process while parsing is sharded over a worker pool.

for parsed in dump.iter_parsed(workers=15):
    ...  # ParsedEntry instances yielded across all workers

workers defaults to os.cpu_count() - 1. Pass workers=1 to skip multiprocessing entirely (useful with pdb).

Output schema

ParsedEntry(
    page_name="Abend",
    page_id=2742,
    entry_index=0,
    language="Deutsch",
    language_code="de",
    lemma="Abend",
    reference=None,                          # LemmaReference if the page is an inflected/variant form
    pos=[PosTag(pos="Substantiv", subtypes=())],
    inflection={
        "gender": "m",
        "nominative_singular": "Abend",
        "nominative_plural": "Abende",
        "genitive_singular": "Abends",
        "genitive_plural": "Abende",
        "dative_singular": "Abend",
        "dative_plural": "Abenden",
        "accusative_singular": "Abend",
        "accusative_plural": "Abende",
    },
    ipa=["ˈaːbn̩t", "ˈaːbm̩t"],
    hyphenation=["Abend"],
    rhymes=["aːbn̩t"],
    meanings=[Meaning(text="…", tags=["Astronomie"], raw_tags=[])],
)

All result containers are @dataclass(slots=True). The full schema lives in wiktionary_de_parser/models.py.

Inflection keys

Inflection-table parameter names are token-translated to English lowercase + underscore: "Nominativ Singular" → "nominative_singular", "Präsens_er, sie, es" → "present_3sg". Unknown tokens are kept verbatim (lowercased).

Lemma references

If the entry is an inflected form or alternative spelling, lemma holds the canonical target and reference records the type:

# "gehörte" → "gehören"
parsed.lemma == "gehören"
parsed.reference == LemmaReference(target="gehören", type=ReferenceType.INFLECTED)

# "Geografie" → "Geographie"
parsed.reference == LemmaReference(target="Geographie", type=ReferenceType.VARIANT)

Development

uv sync                 # install dependencies
uv run pytest           # run the test suite
uv run ruff format
uv run ruff check

License

MIT © Gregor Weichbrodt

Metadata

Release files for wiktionary-de-parser 0.16.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for wiktionary-de-parser 0.16.0
File	Size	Uploaded
wiktionary_de_parser-0.16.0.tar.gz	39.3 kB	May 29, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for wiktionary-de-parser 0.16.0
File	Interpreter	ABI	Platform	Reset
wiktionary_de_parser-0.16.0-py3-none-any.whl 39.5 kB May 29, 2026	Python 3	none	any	Details

Total release size: 78.8 kB

Release files / wiktionary_de_parser-0.16.0.tar.gz

Download URL	wiktionary_de_parser-0.16.0.tar.gz
Size	39.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`477c27d8381e424b88361110b8036ef7ba49b1777adcd74bcdeabb302c4b5589`
BLAKE2b-256 checksum How to use checksums	`bdba23defd9a23815fe539dbd3e5fc1b994e6ccae5cb7ef19e4230fccfda89f9`
Upload date	May 29, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Release files / wiktionary_de_parser-0.16.0-py3-none-any.whl

Download URL	wiktionary_de_parser-0.16.0-py3-none-any.whl
Size	39.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`eb1abe33e49f57fc2af9dbbc353fc198920bb4d9c1dcaea1fb527fa766d18997`
BLAKE2b-256 checksum How to use checksums	`06b131224449b0ef9cf173a2e9514c660fa4ff9ef52f2960e867f835791156b4`
Upload date	May 29, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`