wikiloc

Extract citations and parse in-source locators from wikitext.

An in-source location is the specific part of a source that supports a statement — a page, a page range, a chapter, a quotation.... Editors record these through dedicated citation-template parameters (page, pages, chapter, quote, …), which we call locators. wikiloc reads the locators (and identifiers) declared by references, using the exact set of parameters each citation template authorises.

• Zero runtime dependencies (Python standard library only) for parsing, optional mwparserfromhell for ref extraction from wikitext.
• Three levels: a single reference, an article's reference list, or whole-article wikitext.
• Only English has been tested, though more languages can be added.

The parser is regression-tested against a golden set of 160 hand-annotated citations (English Wikipedia, 2014 and 2026 snapshots, stratified by template and locator type) in wikiloc/tests/test_golden.py. For each row the test asserts the exact set of locator types, so precision and recall for locator presence and type are 100% on the set; it also asserts the page/pages value (when one is expected) and the compute_located_pages() count. Four rows where the parser's output differs from the human gold — noisy page values (pages 32, a page plus an external link), an unparseable range (4B), and an HTML entity (–) — assert the parser's value instead, each with a note recording the mismatch.

Install

pip install wikiloc            # core (stdlib only)
pip install wikiloc[extract]   # + whole-article extraction (mwparserfromhell)

Quickstart

A single reference:

from wikiloc import parse

parse("{{cite book|title=X|isbn=978-0-13-468599-1|page=42}}")
# {'cite_type': 'cite book', 'locators': {'page': '42'}, 'ids': {'isbn': '978-0-13-468599-1'}}

parse("<ref name=\"a\">{{cite book|title=B|page=87}}</ref>{{rp|page=92}}")
# {'cite_type': 'cite book', 'locators': {'page': '92'}, 'ids': {}}

A list of references from one article (resolved with named-reference inheritance):

parse(["<ref name=a>{{cite book|page=10}}</ref>", "<ref name=a/>{{rp|13}}"])
# second dict: {'cite_type': 'cite book', 'ref_type': 'repeated_rp', 'page': '13', ...}

Whole-article wikitext (extraction + resolution; needs [extract]):

from wikiloc import parse_article

parse_article(article_wikitext)
# [{'ref_type': 'main', 'page': '42', ...}, ...]

Output

parse() on a single reference returns:

Locators keep their literal parameter key (e.g. p, pp); canonicalising aliases onto page/pages is left to the caller. The one exception is numbered {{r}} aliases (p1/1p → page), which have no literal counterpart.

The return shape depends on the input. A single reference string returns the grouped dict above; a list (article-level parse()) returns flat records with the locator keys at the top level ({'cite_type': …, 'page': '42', …}, plus ref_type/ref_name). Use wikiloc.locators_of(record) to read the locator map from either shape:

from wikiloc import parse, locators_of
locators_of(parse("{{cite book|page=42}}"))          # {'page': '42'}
locators_of(parse(["{{cite book|page=42}}"])[0])     # {'page': '42'}

A bundled {{r}} returns a list of grouped dicts, one per citation (see Supported reference forms). Article-level parse(list)/ parse_article() likewise return one flat record per citation, and wikiloc.parse_citations(record) exposes the per-citation view for a single reference record.

Supported reference forms

{{cite …}} / {{citation}} (CS1/CS2), {{rp}}, {{r}}, {{sfn}}, {{sfnp}}, Harvard short cites ({{harvnb}} and variants), <ref>…</ref> tags (optionally followed by an {{rp|…}}), and untemplated (plain-text) references. For untemplated text, only pages/page-ranges introduced by an explicit marker (p., pp., page, pages, pg, pgs) are read, so years, scores and dates are not mistaken for page numbers.

{{r}} accepts Template:R's full parameter set — including at/loc, the quote family, and the numbered/named aliases. A {{r}} can bundle up to nine references in one call ({{r|A|B|C}} or {{r|1=A|p1=10|2=B|pp2=20}}); each bundled citation becomes its own record, so the package keeps one record per citation. Bundling is not expanded when the {{r}} is wrapped inside a <ref> tag.

Short-cite (CITEREF) resolution

At article level, parse() on a list and parse_article() resolve shortened footnotes ({{sfn}}, {{sfnp}}, the Harvard family) and {{r}} keys against full CS1/CS2 citations.

{{sfn}}/{{sfnp}}/Harvard short cites link through their HTML anchor: the explicit |ref= value or the auto-generated CITEREF<last-names><year> id (built from up to four author last names — editors when there is no author — plus the year from |year= or |date=, mirroring Module:Citation/CS1). A matching short cite inherits the full citation's cite_type and identifiers (isbn, doi, …), so enrichment follows the link.

{{r}} keys follow Template:R, which links to a named reference ({{r|RefName}} is equivalent to <ref name="RefName"/>). A key therefore resolves first through named-reference inheritance, then — as a heuristic — against an explicit |ref= value or the auto-generated CITEREF<key> anchor.

parse_article(
    "Text.{{sfn|Smith|2020|p=3}}\n"
    "<ref>{{cite book|last=Smith|year=2020|isbn=978-0-13-468599-1|pages=100-150}}</ref>"
)
# the sfn record inherits cite_type='cite book', isbn=… and pages='100-150'

|ref=none disables the anchor, |ref=harv forces the auto id, any other |ref= value is the literal id, and {{sfnref|…}}/{{harvid|…}} expand to CITEREF ids.

Flags

Opt-in issue detectors help audit citations. Each accepts either a parse() result or a flat locator record:

• wikiloc.detect_locator_issues(parsed_ref) — page/place locators;
• wikiloc.detect_quote_issues(parsed_ref) — quotation locators;
• wikiloc.detect_issues(parsed_ref, include=("page", "quote")) — both, merged.

Page/place flags

• no_locator — the reference declares no locator at all;
• pages_single — a pages/pp value is a single page number (often a total-page count);
• page_range — a page/p value holds a range or comma list (probably belongs in pages);
• page_unparseable / pages_unparseable — the value could not be parsed;
• page_huge — a page/p number exceeds PAGE_HUGE_THRESHOLD (default 99999);
• pages_range_huge — a pages/pp range exceeds PAGES_RANGE_HUGE_THRESHOLD (default 999);
• page_reversed_range / pages_reversed_range — a range runs backwards (hi < lo);
• page_noisy — a parseable page/p value also carries prose (e.g. 200–201 & sketch 19);
• page_outside_pages — page/p falls outside the pages/pp range;
• multiple_page_locators — more than one of page/p, pages/pp and at (CS1 allows exactly one; the module flags the extras as Extra |pages= or |at=). Short cites are exempt: in {{sfn}}/{{sfnp}}/Harvard, at/loc supplements p/pp, so only the page-vs-pages choice is exclusive there.

Quotation flags

detect_quote_issues mirrors the above for the quote family:

• quote_alias / quote_conflicting_aliases — non-canonical spelling (qp, qpp, q, quotation…) or two aliases that disagree;
• quote_unsupported_param — a quote parameter the template does not recognise (e.g. qp on a CS1 cite book, though it is valid on {{rp}});
• quote_pages_single / quote_page_range — value in the wrong quote-page vs quote-pages parameter;
• quote_page_both, quote_location_and_quote_page — mutually exclusive quote locators together;
• quote_page_redundant — quote-page duplicates the cited page span exactly;
• quote_page_outside_cited / quote_pages_outside_cited — the quote locator spills outside the cited page/pages span (CS1 says it should be a subset);
• quote_page_unparseable / quote_pages_unparseable, quote_page_reversed_range / quote_pages_reversed_range, quote_page_huge / quote_pages_range_huge;
• quote_page_without_quote, quote_without_locator — a quote-page with no quote, or a quote with no locating parameter at all;
• quote_markup — wikitext/HTML left inside the quote;
• quote_excessive — quote text longer than QUOTE_EXCESSIVE_THRESHOLD.

A differing quote-page is not flagged on its own — the parameter exists to differ from page.

Flags are not run by default, so parse() output stays a faithful transcription of the source. Opt in with with_flags=True:

parse("{{cite book|title=X|pages=240}}", with_flags=True)
# {'cite_type': 'cite book', 'locators': {'pages': '240'}, 'ids': {}, 'flags': ['pages_single']}

For a list of references, each resolved record gets its own flags key. with_flags=True runs the page/place detector only; quote flags are opted into separately (detect_quote_issues(record) or detect_issues(record)). Thresholds live in wikiloc._grammar (PAGE_HUGE_THRESHOLD, PAGES_RANGE_HUGE_THRESHOLD, re-exported via wikiloc.parser) and QUOTE_EXCESSIVE_THRESHOLD in wikiloc.flags.

Located-page counting

wikiloc.compute_located_pages turns a reference's page/pages locators into a page count. It accepts either a parse() result or a flat locator record.

• a single number counts as 1 page;
• a page/p locator counts as 1 page even if its value is malformed (detect_locator_issues reports those);
• a range a–b counts as b − a + 1 pages, with abbreviated ends expanded (446–52 → pp. 446–452, 7 pages);
• a reversed range counts as 1;
• a comma-separated list of page numbers counts its items;
• section-style ranges such as S1–S5 count their members;
• an unparseable pages/pp value is treated as absent;
• when page and pages coexist, the minimum is used.

Quote/chapter/at/loc and other non-paginated locators are deliberately excluded: the package stays unopinionated about how much they narrow a source.

What the count does not cover

compute_located_pages is an estimate, not an exact page count:

• only page/pages (p/pp) are counted — a page number buried in an at/loc value (e.g. at=cc1278-84) is not;
• a single pages/pp number counts as 1 page even when it is really a total-page count rather than a pin (see the pages_single flag);
• malformed values are absorbed, not surfaced: an unparseable page/p still counts as 1, while one bad item in a pages/pp list drops the whole list (treated as absent);
• a reversed range counts as 1;
• when page and pages coexist, only the minimum is returned;
• Roman numerals are accepted only in the strict 1–4999 grammar (IIII, VX are rejected).

Pair the count with detect_locator_issues when these cases matter.

Quotation helpers

wikiloc.quotes is a pure, offline layer for quote locators (its issue flags live with the other detectors, above).

Canonicalisation. quote_locators(parsed_ref) collapses the many quote aliases onto quote, quote-page, quote-pages and quote-location (French extrait/citation → quote):

from wikiloc import quote_locators
quote_locators({"locators": {"q": "To be", "qp": "12"}})
# {'quote': 'To be', 'quote-page': '12'}

Plain text. quote_plaintext() strips wikitext/HTML from a quote, and quote_search_fragments() splits it on [...]/... omissions into searchable runs.

Source URL deep-links. citation_url(wikitext) extracts a citation's url (or chapter-url …), and improve_quote_url(url, quote) appends the anchor that jumps to the quotation — dq= for Google Books, ?q= (or the legacy #search/) for the Internet Archive — leaving a URL that already searches untouched:

from wikiloc import citation_url, improve_quote_url
url = citation_url("{{cite book|url=https://books.google.com/books?id=abc|title=X}}")
improve_quote_url(url, "MIT for a long time stood virtually alone as a university")
# https://books.google.com/books?id=abc&dq=MIT+for+a+long+time+stood+virtually+alone+as+a+university

Locating a source that is not yet linked is a network concern and stays out of the package (see tools/).

Tools

Standalone helpers built on wikiloc live in tools/; they are not part of the published package (they are in neither the wheel nor the source distribution, and may pull in mwparserfromhell/pywikibot). See tools/README.md and each script docstring and --help for details.

Extending

The locator vocabulary and the flag rules live in wikiloc/constants.py:

• New language — add a LOC_PARAMS[lang] mapping of template name → locators, plus a LOCATOR_ROLES[lang] table mapping each literal key to a canonical role (page, pages, place, quote, …). Only en is populated today; French quote keys are bridged by FOREIGN_QUOTE_TEXT_KEYS until a full fr role table lands.
• New locator — append the parameter alias to the relevant template's list and give it a role in LOCATOR_ROLES.
• New family rule — add a LOCATOR_RULES[family] entry (which roles are mutually exclusive, which should be subsets) for a new citation family.
• New identifier — add an alias → canonical mapping in ID_ALIASES (and, for free-text detection, a pattern in _extract_ids_from_text).

The parser, the flag detectors (wikiloc/flags.py) and the quotation helpers (wikiloc/quotes.py) share the page grammar and record accessors in the private wikiloc/_grammar.py.

AI-Assisted Development

Portions of this codebase were generated or refactored with AI assistance:

• Pi (coding agent)
• DeepSeek V4 Flash (open-weight model)

The project concept, architecture decisions, and final review are the author's own.

Citation

If you use wikiloc in academic work, please cite the accompanying paper:

@inproceedings{morvan2026citation,
  author    = {Morvan, Mario},
  title     = {Citation Location Needed: In-Source Locators and Wikipedia's Growing Verification Cost},
  booktitle = {The 2026 ACM/IEEE Joint Conference on Digital Libraries (JCDL '26)},
  series    = {JCDL '26},
  year      = {2026},
  month     = oct,
  location  = {Frisco, TX, USA},
  publisher = {ACM},
  address   = {New York, NY, USA},
  isbn      = {979-8-4007-2597-5},
  doi       = {10.1145/3805696.3846057},
  note      = {In press}
}

License

MIT License © 2026 Mario Morvan — see LICENSE for details.