mx-opendata-tools

Standard-library clients for Wikidata, Wikipedia, Commons and OpenStreetMap, with the wikitext and geometry helpers that go with them.

Installs as mx-opendata-tools, imports as opendata_tools. The plain name was unavailable on PyPI — it collapses to an existing opendatatools — so the distribution carries a personal prefix while the module keeps the readable name. The same split as pillow → PIL or scikit-learn → sklearn.

This is not a general Wikimedia SDK. It is the set of pieces one open-data project actually needed and got right — a retry policy tuned by things that really went wrong, an infobox parser that survives nested templates, geometry that puts a coordinate on a road rather than near it — extracted because a second project needed the same pieces. Where it is opinionated, the docstring says what the opinion cost to learn.

pip install mx-opendata-tools

The core imports nothing but the standard library, and it will stay that way. Two optional extras exist and nothing else needs them:

pip install 'mx-opendata-tools[ddgs]'       # the account-free web-search backend
pip install 'mx-opendata-tools[langchain]'  # the tools, wrapped as LangChain tools

The two paid search providers need no extra — they are plain HTTP, which the standard library already speaks.

Two ways to call it

For one call, use the functions. There is no setup:

from opendata_tools import wikidata_search, sparql

wikidata_search("Kyiv", language="en")
# [{'qid': 'Q1899', 'label': 'Kyiv', 'description': 'capital and largest city of Ukraine'}, …]

sparql("SELECT ?item WHERE { ?item wdt:P31 wd:Q2095 } LIMIT 5")

For more than a couple, build a client. This is also how you set the User-Agent — which the Wikimedia APIs require to identify your application, and will throttle you for omitting:

from opendata_tools import HttpClient, Wikidata, user_agent

http = HttpClient(
    user_agent=user_agent("street-atlas", "2.1", "https://example.org/street-atlas"),
    timeout=60,
)
wikidata = Wikidata(http=http, languages="uk|en")
wikidata.entities(["Q1899", "Q6436261"])

HttpClient is frozen, so it is safe to share across threads, and http.replace(timeout=600) gives you a variant for one slow call without disturbing the original.

What is in it

Wikidata

from opendata_tools import wikidata_entities, wikidata_search, sparql

wikidata_entities(["Q1899"], languages="en")  # batched at 50 per call, transparently
sparql("SELECT ?s WHERE { wd:Q1899 rdfs:label ?s } LIMIT 1")

sparql POSTs, because a query of any size overruns a URL, and returns raw bindings — flattening them means deciding about datatypes and language tags, and that belongs to whoever wrote the query.

Look a QID up; never recall one. Q80895 reads like a plausible guess for "asphalt" and is in fact guerrilla warfare. Anything deriving an identifier from memory is producing fiction that will validate cleanly.

Wikipedia

from opendata_tools import wikipedia_article, wikipedia_search, resolve_titles

wikipedia_search("Khreshchatyk", lang="en")  # titles + snippets
article = wikipedia_article(
    "Q1076911",
    lang="uk",  # a QID or a title
    infobox_template="Вулиця України",
    sections=("Історія",),
)
article["infobox"], article["lead"], article["sections"]["Історія"], article["links"]

resolve_titles(["Kyiv", "Dnieper"], lang="en")  # titles -> QIDs

Search first, then read: wikipedia_search returns page titles and wikipedia_article takes one. Web search answers the same question far worse — short snippets off a rotating set of engines, so the same query gives different results run to run.

Commons

from opendata_tools import Commons, commons_page_exists

commons_page_exists("File:Kyiv collage.jpg")  # a read; needs nothing

commons = Commons(http=http)  # a write; needs a bot password
commons.login(user, password)
commons.create("Data:Example.map", text, "summary")  # refuses to overwrite

create sends createonly unless you pass overwrite=True, so an existing page comes back as a server-side refusal rather than being replaced by accident.

OpenStreetMap and geometry

from opendata_tools import osm_relation, stitch, features, midpoint, centre, endpoints

relation, ways, nodes = osm_relation("421866")
lines = features(stitch(list(ways.values())), nodes, properties={"stroke": "#f00"})

midpoint(lines)  # (lat, lon) — a point *on* the line
centre(lines)  # (lat, lon) — the bounding-box centre; a display hint only
endpoints(lines)  # the two ends, in way order

Coordinates go in as (lon, lat) (GeoJSON order, and what the OSM functions return) and single points come out as (lat, lon), which is how people write them. Prefer midpoint over centre for anything that has to sit on the feature: across fourteen sample streets the bounding-box centre was a median 8.7 m off the carriageway and at worst 182 m.

Wikitext

from opendata_tools import parse_template, section, wikilinks, strip_comments

parse_template tracks brace and bracket depth, because template values contain nested templates and piped wikilinks whose pipes are not the template's — a plain split("|") shreds them quietly into fragments that look almost right.

Web search

from opendata_tools import web_search

web_search("query", provider="ddgs")  # no account needed
web_search("query", provider="serper", api_key=key)  # plain HTTP
web_search("query", provider="searlo", api_key=key)

All three return {"provider": …, "results": [{"title", "url", "snippet"}]}. These never raise — a failure is {"provider": …, "error": …} — because web search is the one call whose failure is routinely uninteresting, and a caller in the middle of a long run should be able to note it and carry on.

When things fail

Everything raised descends from OpenDataError:

OpenDataError
├─ HttpError            .url .what .status .body
│  ├─ HttpStatusError   a final, non-retryable status
│  ├─ TransientError    timeout / dropped connection / unparseable body
│  └─ RetriesExhausted
├─ ApiError             HTTP 200, and the payload reports a failure
│  ├─ MediaWikiError    .code .info
│  └─ LoginError
├─ NotFound · TagMismatch · GeometryError · ConfigurationError

Two deliberate exceptions to "raise on failure":

• fetch_page returns None for a 404, "" for anything unreadable, and the text otherwise. The two failures mean opposite things about whatever cited the URL: a 404 says the citation points at nothing, a timeout says nothing about the citation at all.
• web_search returns an error key, as above.

Logging

The library configures no logging and attaches a NullHandler. Retries are logged at WARNING and one line per request at DEBUG — never headers or bodies, so an API key or a login POST cannot reach a log file.

A retrying run looks like a hang if you never turn these on. In a CLI:

import logging

logging.basicConfig(level=logging.WARNING, format="%(message)s")

Keys and .env

from opendata_tools import load_env_file, require_env

load_env_file(".env")  # returns the NAMES it set, never the values
require_env("SERPER_API_KEY", hint="Create one at https://serper.dev")

A real environment variable always wins over the file, so a CI secret beats a stale local .env without anyone having to remember. Values are never returned alongside their names, logged, or put in an error message.

Tests

uv run pytest                                              # offline; no network
OPENDATA_TOOLS_INTEGRATION=1 uv run pytest -m integration   # hits the live APIs

Integration tests are gated by a marker and an environment variable, so a bare pytest is green with no network and even an explicit pytest -m integration stays skipped. Reaching the real APIs has to be deliberate. There is no live test that writes to Commons — a test that edits a public wiki is not a test.

Versioning

Semantic versioning from 0.1.0. While the major version is 0, a breaking change bumps the minor — pin mx-opendata-tools>=0.1,<0.2. See CHANGELOG.md.

Licence

MIT.