passageway-connector-wikipedia
A passageway connector that turns part of Wikipedia (or any MediaWiki site) into searchable items: a topic (categories), some titles, or a place. English and German are both tested; any language code works.
Each article is one item. Its id is the page id, its version is the revision id, and its sections follow the article's headings (History › 19th century). Boilerplate sections such as References or Einzelnachweise are left out. A formula is indexed as its TeX between $ signs ($a^{2}+b^{2}=c^{2}$), not as the MathML tokens the plain-text extract carries.
Configure
In passageway.toml, one [[connector]] per language (use the interpreter that has passageway installed, see Install):
[[connector]]
name = "deep-learning-en"
command = ["/path/to/python", "-m", "passageway_connector_wikipedia"]
env = { LANGUAGE = "en", CATEGORIES = "Deep learning", CATEGORY_DEPTH = "0" }
[[connector]]
name = "deep-learning-de"
command = ["/path/to/python", "-m", "passageway_connector_wikipedia"]
env = { LANGUAGE = "de", CATEGORIES = "Kategorie:Deep Learning" }
[[connector]]
name = "munich"
command = ["/path/to/python", "-m", "passageway_connector_wikipedia"]
env = { LANGUAGE = "de", GEO = "48.1372,11.5756,5000" } # articles within 5 km of central Munich
| Setting | Default | Meaning |
|---|---|---|
LANGUAGE |
en |
Wikipedia language code |
CATEGORIES |
Categories, separated by |. A bare name gets the Category: prefix; a name with a prefix (Kategorie:...) is used as given |
|
CATEGORY_DEPTH |
1 |
How many levels of subcategories to follow (0 = only the listed categories) |
TITLES |
Article titles, separated by |; redirects are followed, missing pages are ignored |
|
GEO |
lat,lon,radius_in_meters: articles about places near a point. The radius is limited to 10 to 10000 m by MediaWiki, so this suits a city or neighbourhood, not a region. For a region use CATEGORIES |
|
MAX_PAGES |
500 |
Upper bound on the scope. A bigger scope fails with a message instead of being cut off |
SKIP_SECTIONS |
English and German boilerplate | Headings to leave out, separated by |; none keeps everything. A skipped heading also skips its subsections |
API_URL, PAGE_URL |
derived from LANGUAGE |
Point at another MediaWiki site |
CONTACT |
Added to the User-Agent; Wikimedia asks automated clients to identify themselves | |
PAUSE_SECONDS |
0.1 |
Pause between requests |
At least one of CATEGORIES, TITLES or GEO is required.
Behavior
- Polite: sequential requests with a descriptive User-Agent,
maxlag, and back-off on 429 and 503. - Redirects: redirect pages inside a category are left out;
TITLESfollow redirects to the real article. - Retries: 429, 502, 503 and 504 answers,
maxlagand network errors are retried with a growing wait (1 s, doubling up to 60 s; a longerRetry-Afteris honored up to 300 s) until a request has waited 15 minutes in total; the error then says how many attempts and how long it waited. Every wait of 10 s or more is logged as it starts (HTTP 429 … waiting 16 s (15 s of 900 s waited)), so a sync that waits does not look hung. Waiting out a long rate limit needs a coretimeoutabove its default of 300 s. - No
changedmode: edits appear at the next reconcile (every 30 minutes by default). - Content license: Wikipedia text is CC BY-SA 4.0. The link on each item points to the source article; its history page lists the authors.
Test
uv sync # installs this connector too
uv run pytest connectors/wikipedia # offline, against an in-test MediaWiki server
PASSAGEWAY_LIVE=1 uv run pytest connectors/wikipedia/tests/test_wikipedia_live.py # real Wikipedia
Metadata
Release files for passageway-connector-wikipedia 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| passageway_connector_wikipedia-0.3.0.tar.gz | 14.2 kB | Without provenance | Details for passageway_connector_wikipedia-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| passageway_connector_wikipedia-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for passageway_connector_wikipedia-0.3.0-py3-none-any.whl |
Total release size: 24.0 kB
Release files / passageway_connector_wikipedia-0.3.0.tar.gz
| Download URL | passageway_connector_wikipedia-0.3.0.tar.gz |
|---|---|
| Size | 14.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2561ae1b40538f7fee5370578ae82b1e3d22d806a883f8dbcdb030c1b8761aac
|
|
BLAKE2b-256 checksum How to use checksums |
4616c5318d453b02639417d493a4d0712edfdcab2f9a982e350c22d391e7122e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / passageway_connector_wikipedia-0.3.0-py3-none-any.whl
| Download URL | passageway_connector_wikipedia-0.3.0-py3-none-any.whl |
|---|---|
| Size | 9.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bae1fa1280a2c1687788dba50922357a1817c369d34e9a231aef71ca0a8c0791
|
|
BLAKE2b-256 checksum How to use checksums |
4aa3aab40a7dd1b786b87b9e054235f3858aa1ad69177a80aefbfea13ab59633
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for passageway-connector-wikipedia 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.