wikidata-ner-classifier 0.3.1

Deterministic Wikidata NER prediction with description-based branch-local refinement

Wikidata NER Classifier 0.3.1

Deterministic, explainable prediction of retrieval-oriented entity types from a Wikidata QID, direct type-class labels, optional ancestor labels, and an entity description.

The QID is retained as an identifier and is never used as a lookup key.

What changed in 0.3.1

The classifier now exposes four progressively narrower retrieval levels:

coarse_type -> fine_type -> subtype -> specific_type

It also returns every compatible narrow label in specific_types and keeps orthogonal properties such as genre in facets.

The default evidence policy is now:

1. types[].name selects coarse_type and fine_type.
2. ancestor_types[].name, when supplied, provides lower-weight class ancestry.
3. Direct type labels and description refine only the selected branch.
4. context_string is ignored by the classifier by default because it often contains related people, organizations, countries, genres, and formats.
5. Description evidence cannot change a FILM branch into a location, company, person, or another unrelated branch.
6. Unsupported specificity is not invented.

The packaged configuration contains:

• 187 fine-type rules;
• 295 structural subtype rules;
• 33 controlled facet rules;
• 6 branch-local composite-type templates.

Installation

python -m pip install wikidata-ner-classifier

Python 3.10 or newer is required. The library has no runtime dependencies.

Basic use

from wikidata_ner import WikidataNERClassifier

classifier = WikidataNERClassifier()

prediction = classifier.predict(
    qid="Q3441181",
    types=[{"id": "Q11424", "name": "film"}],
    description="1964 sword-and-sandal film directed by Giuseppe Vari",
)

print(prediction.to_dict())

Relevant output:

{
  "coarse_type": "CREATIVE_WORK",
  "fine_type": "FILM",
  "subtype": null,
  "specific_type": "SWORD_AND_SANDAL_FILM",
  "specific_types": [
    "SWORD_AND_SANDAL_FILM"
  ],
  "facets": {
    "genre": [
      "SWORD_AND_SANDAL"
    ]
  },
  "refinement_sources": [
    "description"
  ]
}

The description adds specificity only inside the already established FILM branch.

Alpaca or Elasticsearch entities

Both source objects and complete Elasticsearch hits are accepted:

prediction = classifier.predict_entity(hit_or_source)

{
  "qid": "Q3441181",
  "types": [{"name": "film"}],
  "description": "1964 sword-and-sandal film directed by Giuseppe Vari"
}

{
  "_id": "Q3441181",
  "_source": {
    "qid": "Q3441181",
    "types": [{"name": "film"}],
    "description": "1964 sword-and-sandal film directed by Giuseppe Vari"
  }
}

A complete Elasticsearch response can be processed with:

predictions = classifier.predict_elasticsearch_response(response)

Why both subtype and specific type exist

A subtype describes a structural kind. A facet describes an independent characteristic. A specific type is a retrieval-oriented composition.

prediction = classifier.predict(
    "Q1",
    [
        {"name": "film"},
        {"name": "feature film"},
        {"name": "comedy film"},
    ],
)

This can produce:

{
  "fine_type": "FILM",
  "subtype": "FEATURE_FILM",
  "specific_type": "FEATURE_FILM",
  "specific_types": [
    "FEATURE_FILM",
    "COMEDY_FILM"
  ],
  "facets": {
    "genre": [
      "COMEDY"
    ]
  }
}

FEATURE_FILM and COMEDY_FILM are compatible. They may be combined by the retriever rather than forced into a single mutually exclusive label.

Example refinements from the Alpaca query

Type labels	Description	Fine type	Subtype	Most specific retrieval type
film	1951 film directed by Luigi Zampa	FILM	none	FILM
film	1964 sword-and-sandal film ...	FILM	none	SWORD_AND_SANDAL_FILM
album	album by Holger Czukay	MUSICAL_WORK_SONG_OR_ALBUM	MUSIC_ALBUM	MUSIC_ALBUM
literary work	Alternative history, military science fiction story	BOOK_OR_WRITTEN_WORK	FICTION_STORY	MILITARY_SCIENCE_FICTION_LITERARY_WORK
pencil drawing	1953 work of art ...	VISUAL_ARTWORK_PHOTOGRAPH_OR_COMIC	PENCIL_DRAWING	PENCIL_DRAWING

A generic description cannot justify an invented subtype. A generic film remains FILM when neither its type labels nor description contain a safe refinement.

Retrieval indexing helpers

Store the prediction alongside each entity using stable keyword fields:

fields = prediction.to_retrieval_fields(prefix="ner")

Example fields:

{
  "ner_coarse_type": "CREATIVE_WORK",
  "ner_fine_type": "FILM",
  "ner_subtype": null,
  "ner_specific_type": "SWORD_AND_SANDAL_FILM",
  "ner_specific_types": [
    "SWORD_AND_SANDAL_FILM"
  ],
  "ner_facets": {
    "genre": [
      "SWORD_AND_SANDAL"
    ]
  }
}

A deterministic Elasticsearch filter can be generated with:

query_filter = prediction.elasticsearch_filter(
    field="ner_specific_types",
    require_all=True,
)

For several compatible specific types, require_all=True emits one term filter per type. Use require_all=False to emit a terms disjunction.

Index-time and query-time predictions should use the same library and rule-file version.

Description and context controls

Description refinement is enabled by default:

classifier = WikidataNERClassifier(
    use_description_for_refinement=True,
    use_context_string_for_refinement=False,
)

Disable it when only class labels should be considered:

classifier = WikidataNERClassifier(
    use_description_for_refinement=False,
)

Noisy context refinement is available only as an explicit opt-in:

classifier = WikidataNERClassifier(
    use_context_string_for_refinement=True,
)

The separate use_description=True option allows description text to add low-weight support to the primary coarse/fine scorer. It is disabled by default. Descriptions therefore do not rescue a missing or unknown type anchor unless the caller explicitly changes that policy.

Live Alpaca notebook

Open examples/alpaca_live_test.ipynb.

The notebook:

1. issues the supplied Alpaca Elasticsearch request;
2. extracts QID, type labels, and description;
3. ignores context_string during classification;
4. displays predicted_subtype, predicted_specific_type, all compatible specific_types, facets, and confidence values;
5. demonstrates a retrieval filter generated from the prediction.

Set the bearer token before starting Jupyter:

export ALPACA_TOKEN='your-token'

The notebook also supports a hidden token prompt when the environment variable is not set.

CLI

wikidata-ner response.json > predictions.json
cat response.json | wikidata-ner

Relevant flags:

--no-description-refinement
--context-refinement
--description-for-primary
--entity-label

Validation

The source package includes unit tests for:

• direct type-label classification;
• description-only branch refinement;
• context exclusion by default;
• explicit context opt-in;
• subtype and facet compatibility;
• generic fallback behavior;
• QID independence;
• Elasticsearch hit input;
• retrieval-field generation;
• generated Elasticsearch filters.

Mention-focused OpenRouter notebook

examples/openrouter_mention_focused_ner.ipynb classifies one explicit target mention at a time from free text, structured records, or table cells. Context is used only as evidence for that target. Contextual free text requires mention= or an exact span; table helpers make the selected cell the target.