Skip to main content

passageway-connector-wikipedia

A passageway connector that turns part of Wikipedia (or any MediaWiki site) into searchable items: a topic (categories), some titles, or a place. English and German are both tested; any language code works.

Each article is one item. Its id is the page id, its version is the revision id, and its sections follow the article's headings (History › 19th century). Boilerplate sections such as References or Einzelnachweise are left out. A formula is indexed as its TeX between $ signs ($a^{2}+b^{2}=c^{2}$), not as the MathML tokens the plain-text extract carries.

Configure

In passageway.toml, one [[connector]] per language (use the interpreter that has passageway installed, see Install):

[[connector]]
name = "deep-learning-en"
command = ["/path/to/python", "-m", "passageway_connector_wikipedia"]
env = { LANGUAGE = "en", CATEGORIES = "Deep learning", CATEGORY_DEPTH = "0" }

[[connector]]
name = "deep-learning-de"
command = ["/path/to/python", "-m", "passageway_connector_wikipedia"]
env = { LANGUAGE = "de", CATEGORIES = "Kategorie:Deep Learning" }

[[connector]]
name = "munich"
command = ["/path/to/python", "-m", "passageway_connector_wikipedia"]
env = { LANGUAGE = "de", GEO = "48.1372,11.5756,5000" }   # articles within 5 km of central Munich
Setting Default Meaning
LANGUAGE en Wikipedia language code
CATEGORIES Categories, separated by |. A bare name gets the Category: prefix; a name with a prefix (Kategorie:...) is used as given
CATEGORY_DEPTH 1 How many levels of subcategories to follow (0 = only the listed categories)
TITLES Article titles, separated by |; redirects are followed, missing pages are ignored
GEO lat,lon,radius_in_meters: articles about places near a point. The radius is limited to 10 to 10000 m by MediaWiki, so this suits a city or neighbourhood, not a region. For a region use CATEGORIES
MAX_PAGES 500 Upper bound on the scope. A bigger scope fails with a message instead of being cut off
SKIP_SECTIONS English and German boilerplate Headings to leave out, separated by |; none keeps everything. A skipped heading also skips its subsections
API_URL, PAGE_URL derived from LANGUAGE Point at another MediaWiki site
CONTACT Added to the User-Agent; Wikimedia asks automated clients to identify themselves
PAUSE_SECONDS 0.1 Pause between requests

At least one of CATEGORIES, TITLES or GEO is required.

Behavior

  • Polite: sequential requests with a descriptive User-Agent, maxlag, and back-off on 429 and 503.
  • Redirects: redirect pages inside a category are left out; TITLES follow redirects to the real article.
  • Retries: 429, 502, 503 and 504 answers, maxlag and network errors are retried with a growing wait (1 s, doubling up to 60 s; a longer Retry-After is honored up to 300 s) until a request has waited 15 minutes in total; the error then says how many attempts and how long it waited. Every wait of 10 s or more is logged as it starts (HTTP 429 … waiting 16 s (15 s of 900 s waited)), so a sync that waits does not look hung. Waiting out a long rate limit needs a core timeout above its default of 300 s.
  • No changed mode: edits appear at the next reconcile (every 30 minutes by default).
  • Content license: Wikipedia text is CC BY-SA 4.0. The link on each item points to the source article; its history page lists the authors.

Test

uv sync                                                              # installs this connector too
uv run pytest connectors/wikipedia                                   # offline, against an in-test MediaWiki server
PASSAGEWAY_LIVE=1 uv run pytest connectors/wikipedia/tests/test_wikipedia_live.py   # real Wikipedia

Metadata

Release files for passageway-connector-wikipedia 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for passageway-connector-wikipedia 0.4.0
File Size Uploaded Provenance Actions
passageway_connector_wikipedia-0.4.0.tar.gz 14.2 kB Without provenance Details for passageway_connector_wikipedia-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for passageway-connector-wikipedia 0.4.0
File Interpreter ABI Platform Actions
passageway_connector_wikipedia-0.4.0-py3-none-any.whl Python 3 none any Details for passageway_connector_wikipedia-0.4.0-py3-none-any.whl

Total release size: 24.0 kB

Release files / passageway_connector_wikipedia-0.4.0.tar.gz

Download URL passageway_connector_wikipedia-0.4.0.tar.gz
Size 14.2 kB
Tags Source
SHA-256 checksum
How to use checksums
e6f4d75e5eab6e8a280f06efe9fd349603f56942ab79e2345ede96f22d009bc7
BLAKE2b-256 checksum
How to use checksums
8f5e4ba29606a569d3481cb4b1a01df4c9199694f459c78bea26b896c3c1616e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / passageway_connector_wikipedia-0.4.0-py3-none-any.whl

Download URL passageway_connector_wikipedia-0.4.0-py3-none-any.whl
Size 9.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a41892e34914f06c42fbdd003fabcd0cc30e3361090af45d9b3f05e96288fdb9
BLAKE2b-256 checksum
How to use checksums
05f52173f87b0291c59f5ff381b355a56b9684f81b23113f9ceb77b3beb44615
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for passageway-connector-wikipedia 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page