Wikidata NER Classifier 0.6.0

Classification into one shared retrieval-oriented hierarchy through two complementary paths:

1. WikidataNERClassifier classifies Wikidata items deterministically from P31/P279 token clues and optional descriptions.
2. OpenRouterNERClassifier infers the type of one target mention from free text, a structured record, or tabular context using an LLM.

Both paths return classes from the packaged hierarchy:

coarse_type -> fine_type -> subtype -> specific_type

Fast input prediction with Cerebras through OpenRouter

For free text and tabular inputs, the Cerebras-hosted LLM makes the prediction from the target and its context. It considers the complete packaged hierarchy; the deterministic Wikidata token-clue classifier is a separate path and does not restrict LLM predictions. Returned coarse types, fine types, subtypes, facets, and specific types are validated locally.

export OPENROUTER_API_KEY='...'

from wikidata_ner import OpenRouterNERClassifier

classifier = OpenRouterNERClassifier(
    model="openai/gpt-oss-120b",
    provider="cerebras",
    allow_fallbacks=False,
    reasoning_effort="low",
)
prediction = classifier.predict_text(
    "Rome Against Rome is a 1964 sword-and-sandal film.",
    mention="Rome Against Rome",
)

assert prediction.fine_type == "FILM"
print(prediction.specific_type)  # SWORD_AND_SANDAL_FILM
print(prediction.usage)

Inference uses two compact structured-output requests:

1. Select one coarse type from every coarse branch in the hierarchy.
2. Select one fine type from every fine type in that coarse branch, together with legal branch-local refinements.

This keeps Cerebras schemas small and inference fast without removing valid classes from consideration.

The mention classifier cannot invent labels: structured outputs enumerate the library's classes, and returned branches, subtypes, and facets are validated again locally.

What changed in 0.6.0

• Added 23 controlled occupation types for real people, including POLITICIAN, ACTOR, MUSICIAN, WRITER, ATHLETE, and SCIENTIST.
• Human occupations are multi-valued: one person can expose several compatible specific_types without forcing an arbitrary single occupation.
• Enabled the same human-specific taxonomy for deterministic Wikidata input and locally validated OpenRouter mention inference.
• Generic humans still resolve to HUMAN when no occupation is supported.

What changed in 0.5.1

• Added order-preserving native predict_batch() for mappings and generators.
• Added an instance-local bounded LRU that reuses coarse/fine branch decisions, including abstentions, across batch calls.
• Kept description and context refinement independent for every item.
• Indexed subtype, facet, and composite retrieval rules by selected fine branch.
• Added cache statistics and explicit cache clearing.

See CHANGELOG.md for release history.

What changed in 0.5.0

• Added mention-focused type inference for free text and structured/tabular data.
• Added a dependency-free OpenRouter client with strict JSON-schema output.
• Added complete coarse-to-fine LLM inference over the packaged hierarchy, with branch-local subtype and facet selection in the second stage.
• Added OpenRouter provider pinning for fast Cerebras inference with fallbacks disabled when deterministic latency is required.
• Added exact mention-span marking, target-focus validation, and safe abstention.
• Added an auditable, bounded context report and table preview for cell inference.
• Retained the deterministic Wikidata token/clue classifier unchanged.

The QID is retained as an identifier and is never used as a lookup key.

Two classification paths

Wikidata items: deterministic token clues

Use this path when P31/P279 labels are already available:

from wikidata_ner import WikidataNERClassifier

classifier = WikidataNERClassifier()
prediction = classifier.predict(
    qid="Q3441181",
    types=[{"id": "Q11424", "name": "film"}],
    description="1964 sword-and-sandal film directed by Giuseppe Vari",
)

The primary branch is selected with the library's deterministic token/clue rules. Descriptions can refine that branch but cannot replace its P31/P279 anchor.

Input mentions: LLM inference through OpenRouter

Use this path when the input is a mention and its type must be inferred from context:

export OPENROUTER_API_KEY='...'

from wikidata_ner import OpenRouterNERClassifier

classifier = OpenRouterNERClassifier(
    model="openai/gpt-oss-120b",
    provider="cerebras",
    allow_fallbacks=False,
    reasoning_effort="low",
)

prediction = classifier.predict_text(
    "Rome Against Rome is a 1964 sword-and-sandal film directed by "
    "Giuseppe Vari; the story is set partly in Rome.",
    mention="Rome Against Rome",
)

print(prediction.fine_type)       # FILM
print(prediction.specific_type)   # SWORD_AND_SANDAL_FILM

Only Rome Against Rome is classified. The later Rome is contextual evidence about a different mention and cannot become the prediction target.

For a structured record:

prediction = classifier.predict_record(
    {
        "label": "Rome Against Rome",
        "types": [{"name": "film"}],
        "description": "1964 sword-and-sandal film",
    }
)

For a table cell:

prediction = classifier.predict_table_cell(
    "acetylsalicylic acid",
    column_header="active ingredient",
    row_context={
        "drug": "Aspirin",
        "molecular_formula": "C9H8O4",
    },
    same_column_values=["ibuprofen", "paracetamol", "naproxen"],
)

print(prediction.table_preview)
print(prediction.context_report["context_usage"])

The cell is always the target. Headers, row attributes, and same-column samples are evidence about the cell, never alternative targets. The returned context_report renders the exact selected table information, explains how each context component was interpreted, and reports whether fields or samples were omitted. The LLM receives that information as structured JSON; the Markdown preview is human-readable and is not duplicated in the prompt.

By default, table context is bounded to 12 non-empty same-row fields and 8 distinct same-column samples. Empty values, duplicate samples, and the target itself are removed from the sample set. Adjust the limits only when the table requires it:

prediction = classifier.predict_table_cell(
    cell,
    column_header="title",
    row_context=relevant_row_fields,
    same_column_values=column_examples,
    max_row_fields=8,
    max_column_samples=5,
)

For best accuracy, pass fields that describe or relate directly to the target cell—such as a type/category, description, unit, identifier, creator, location, or parent relation. Avoid unrelated display metadata and entire unfiltered rows.

For contextual free text, mention= is required. You can disambiguate repeated surface forms with an exact character span:

prediction = classifier.predict_text(
    text,
    mention="Rome",
    mention_span=(start, end),
)

Use preview_prompts(...) to inspect the normalized mention, exact prompts, and JSON schemas without making an API call:

preview = classifier.preview_prompts(
    text,
    mention="Rome Against Rome",
    assumed_coarse_type="CREATIVE_WORK",
    assumed_fine_type="FILM",
)

The default predictor uses two compact hierarchy-aware calls:

1. Select exactly one controlled coarse branch or abstain.
2. Select one fine type and only its legal subtypes and facets inside that branch.

Passing a model slug does not select a hosting provider on OpenRouter. Use provider="cerebras" with allow_fallbacks=False when Cerebras latency is required. Each stage reports the routed provider and wall-clock duration under prediction.usage.

OpenRouter is called at https://openrouter.ai/api/v1/chat/completions with strict JSON-schema output, provider.require_parameters=true, temperature zero, and optional response healing. The package continues to have no runtime dependencies. You may inject a custom client= for testing or infrastructure integration.

MentionPrediction supports the same retrieval conveniences as deterministic predictions:

payload = prediction.to_dict()
fields = prediction.to_retrieval_fields(prefix="ner")
query_filter = prediction.elasticsearch_filter()

The selected OpenRouter model must support structured outputs. Pin a model slug in production and store the returned model, prompt version, taxonomy version, usage, evidence, and confidence with each result.

Deterministic evidence policy

The default evidence policy is now:

1. types[].name selects coarse_type and fine_type.
2. ancestor_types[].name, when supplied, provides lower-weight class ancestry.
3. Direct type labels and description refine only the selected branch.
4. context_string is ignored by the classifier by default because it often contains related people, organizations, countries, genres, and formats.
5. Description evidence cannot change a FILM branch into a location, company, person, or another unrelated branch.
6. Unsupported specificity is not invented.

The packaged configuration contains:

• 187 fine-type rules;
• 295 structural subtype rules;
• 56 controlled facet rules;
• 7 branch-local composite-type templates.

Installation

python -m pip install wikidata-ner-classifier

Python 3.10 or newer is required. The library has no runtime dependencies.

Deterministic basic use

from wikidata_ner import WikidataNERClassifier

classifier = WikidataNERClassifier()

prediction = classifier.predict(
    qid="Q3441181",
    types=[{"id": "Q11424", "name": "film"}],
    description="1964 sword-and-sandal film directed by Giuseppe Vari",
)

print(prediction.to_dict())

Relevant output:

{
  "coarse_type": "CREATIVE_WORK",
  "fine_type": "FILM",
  "subtype": null,
  "specific_type": "SWORD_AND_SANDAL_FILM",
  "specific_types": [
    "SWORD_AND_SANDAL_FILM"
  ],
  "facets": {
    "genre": [
      "SWORD_AND_SANDAL"
    ]
  },
  "refinement_sources": [
    "description"
  ]
}

The description adds specificity only inside the already established FILM branch.

Native batch prediction

Use predict_batch() when classifying many items. It accepts any iterable of item mappings, returns normal Prediction objects in input order, and retains each QID:

items = [
    {
        "qid": "Q3441181",
        "types": [{"id": "Q11424", "name": "film"}],
        "ancestor_types": [],
        "label": "Rome Against Rome",
        "description": "1964 sword-and-sandal film",
        "context_string": None,
    },
]

predictions = classifier.predict_batch(items, cache_size=100_000)

Batch prediction normalizes the direct and ancestor type labels, groups identical ordered signatures, and performs the full coarse/fine rule scan once per missing signature. The bounded cache is a true least-recently-used cache and is local to the classifier instance, so custom rules and configuration never share entries with another classifier. Successful decisions and abstentions are both cached.

Only the primary coarse/fine decision is reused. Subtypes, facets, composite specific types, evidence, and refinement sources are calculated independently for every item, so descriptions and context strings remain item-specific. Results are exactly equivalent to calling predict() on every item.

By default, the cache key is the ordered normalized direct and ancestor label signature. If use_description=True, the normalized description is also part of the key. If use_entity_label=True, the normalized entity label is also part of the key. Description/context settings used only for branch-local refinement do not widen the primary key.

Inspect or reset the cache with:

info = classifier.branch_cache_info()
print(info.hits, info.misses, info.maxsize, info.currsize)

classifier.clear_branch_cache()

Passing cache_size=0 disables reuse across calls while still deduplicating repeated signatures inside the current batch. Changing cache_size on a later call immediately evicts the least recently used entries until the cache fits. The implementation is synchronous, dependency-free, and deterministic; callers can place independent classifier instances in an external process pool.

Alpaca or Elasticsearch entities

Both source objects and complete Elasticsearch hits are accepted:

prediction = classifier.predict_entity(hit_or_source)

{
  "qid": "Q3441181",
  "types": [{"name": "film"}],
  "description": "1964 sword-and-sandal film directed by Giuseppe Vari"
}

{
  "_id": "Q3441181",
  "_source": {
    "qid": "Q3441181",
    "types": [{"name": "film"}],
    "description": "1964 sword-and-sandal film directed by Giuseppe Vari"
  }
}

A complete Elasticsearch response can be processed with:

predictions = classifier.predict_elasticsearch_response(response)

Why both subtype and specific type exist

A subtype describes a structural kind. A facet describes an independent characteristic. A specific type is a retrieval-oriented composition.

prediction = classifier.predict(
    "Q1",
    [
        {"name": "film"},
        {"name": "feature film"},
        {"name": "comedy film"},
    ],
)

This can produce:

{
  "fine_type": "FILM",
  "subtype": "FEATURE_FILM",
  "specific_type": "FEATURE_FILM",
  "specific_types": [
    "FEATURE_FILM",
    "COMEDY_FILM"
  ],
  "facets": {
    "genre": [
      "COMEDY"
    ]
  }
}

FEATURE_FILM and COMEDY_FILM are compatible. They may be combined by the retriever rather than forced into a single mutually exclusive label.

Human occupations use the same compatibility model:

person = classifier.predict(
    "Q7259",
    [{"name": "human"}],
    description="British politician and writer",
)

assert person.fine_type == "HUMAN"
assert person.specific_type == "POLITICIAN"
assert person.specific_types == ("POLITICIAN", "WRITER")
assert person.facets["occupation"] == ("POLITICIAN", "WRITER")

The controlled human occupation types are ACADEMIC, ACTIVIST, ACTOR, ARCHITECT, ARTIST, ATHLETE, BUSINESSPERSON, EDUCATOR, ENGINEER, EXPLORER, FILMMAKER, INVENTOR, JOURNALIST, LEGAL_PROFESSIONAL, MEDICAL_PROFESSIONAL, MILITARY_PERSONNEL, MUSICIAN, POLITICIAN, PUBLIC_OFFICIAL, RELIGIOUS_FIGURE, ROYALTY, SCIENTIST, and WRITER.

Example refinements from the Alpaca query

Type labels	Description	Fine type	Subtype	Most specific retrieval type
film	1951 film directed by Luigi Zampa	FILM	none	FILM
film	1964 sword-and-sandal film ...	FILM	none	SWORD_AND_SANDAL_FILM
album	album by Holger Czukay	MUSICAL_WORK_SONG_OR_ALBUM	MUSIC_ALBUM	MUSIC_ALBUM
literary work	Alternative history, military science fiction story	BOOK_OR_WRITTEN_WORK	FICTION_STORY	MILITARY_SCIENCE_FICTION_LITERARY_WORK
pencil drawing	1953 work of art ...	VISUAL_ARTWORK_PHOTOGRAPH_OR_COMIC	PENCIL_DRAWING	PENCIL_DRAWING
human	British politician and writer	HUMAN	none	POLITICIAN, WRITER

A generic description cannot justify an invented subtype. A generic film remains FILM when neither its type labels nor description contain a safe refinement.

Retrieval indexing helpers

Store the prediction alongside each entity using stable keyword fields:

fields = prediction.to_retrieval_fields(prefix="ner")

Example fields:

{
  "ner_coarse_type": "CREATIVE_WORK",
  "ner_fine_type": "FILM",
  "ner_subtype": null,
  "ner_specific_type": "SWORD_AND_SANDAL_FILM",
  "ner_specific_types": [
    "SWORD_AND_SANDAL_FILM"
  ],
  "ner_facets": {
    "genre": [
      "SWORD_AND_SANDAL"
    ]
  }
}

A deterministic Elasticsearch filter can be generated with:

query_filter = prediction.elasticsearch_filter(
    field="ner_specific_types",
    require_all=True,
)

For several compatible specific types, require_all=True emits one term filter per type. Use require_all=False to emit a terms disjunction.

Index-time and query-time predictions should use the same library and rule-file version.

Description and context controls

Description refinement is enabled by default:

classifier = WikidataNERClassifier(
    use_description_for_refinement=True,
    use_context_string_for_refinement=False,
)

Disable it when only class labels should be considered:

classifier = WikidataNERClassifier(
    use_description_for_refinement=False,
)

Noisy context refinement is available only as an explicit opt-in:

classifier = WikidataNERClassifier(
    use_context_string_for_refinement=True,
)

The separate use_description=True option allows description text to add low-weight support to the primary coarse/fine scorer. It is disabled by default. Descriptions therefore do not rescue a missing or unknown type anchor unless the caller explicitly changes that policy.

Live Alpaca notebook

Open examples/alpaca_live_test.ipynb.

The notebook:

1. issues the supplied Alpaca Elasticsearch request;
2. extracts QID, type labels, and description;
3. ignores context_string during classification;
4. displays predicted_subtype, predicted_specific_type, all compatible specific_types, facets, and confidence values;
5. demonstrates a retrieval filter generated from the prediction.

Set the bearer token before starting Jupyter:

export ALPACA_TOKEN='your-token'

The notebook also supports a hidden token prompt when the environment variable is not set.

CLI

wikidata-ner response.json > predictions.json
cat response.json | wikidata-ner

Relevant flags:

--no-description-refinement
--context-refinement
--description-for-primary
--entity-label

Validation

The source package includes unit tests for:

• direct type-label classification;
• description-only branch refinement;
• context exclusion by default;
• explicit context opt-in;
• subtype and facet compatibility;
• generic fallback behavior;
• QID independence;
• Elasticsearch hit input;
• retrieval-field generation;
• generated Elasticsearch filters.

Mention-focused OpenRouter notebook

examples/openrouter_mention_focused_ner.ipynb classifies one explicit target mention at a time from free text, structured records, or table cells. Context is used only as evidence for that target. Contextual free text requires mention= or an exact span; table helpers make the selected cell the target. The installable library now exposes the same workflow through OpenRouterNERClassifier; the notebook remains useful as an expanded prompt inspection and evaluation example.