pyannotators-entityfishing

Annotator based on entity-fishing for named entity recognition and disambiguation against Wikidata.

Installation

pip install pyannotators-entityfishing

For noun-form filtering (optional):

pip install pyannotators-entityfishing[spacy]
python -m spacy download en_core_web_sm  # or other language models

Usage

from pymultirole_plugins.v1.schema import Document
from pyannotators_entityfishing.entityfishing import EntityFishingAnnotator, EntityFishingParameters

annotator = EntityFishingAnnotator()
parameters = EntityFishingParameters(
    default_label="ENTITY",
    minSelectorScore=0.3,
)

docs = annotator.annotate(
    [Document(text="Albert Einstein was born in Ulm.", metadata={"language": "en"})],
    parameters,
)

for ann in docs[0].annotations:
    print(f"{ann.start}:{ann.end} {ann.labelName} {ann.terms[0].identifier}")

Linking candidates (processor entityfishing_candidates)

The package also registers a processor, entityfishing_candidates (group pyprocessors.plugins), which does not decide the link: it writes, on the mentions of an upstream NER, the Wikidata concepts each one may refer to, for a decision step (Jev, pyprocessors_jev) to choose among them. It implements step 2 of ADR-0001, and does not call /disambiguate, whose selector drops the right candidate, or answers nothing, when its prior is low.

For each group of mentions of the same entity (properties.entity_group, written by pyprocessors_coreference, else the same label and surface):

1. term lookup (/kb/term) on the group's canonical_form, falling back on its surfaces: the senses, most frequent first, without those under min_prior, up to top_k;
2. the concepts of those senses (/kb/concept, by Wikipedia page id, in the document's language);
3. the mapped_labels expression of the mention's label keeps only the concepts of the right type;
4. every mention of the group gets the candidates in terms: identifier = QID, lexicon = wikidata, preferredForm, score = the prior (prob_c), and in properties the wikidataId, prob_c, pageid, the first sentence of the definition and the requested Wikidata facts.

The mentions themselves (span, label) are never changed; a mention with no candidate keeps empty terms; terms of other lexicons (AFP...) are kept. It is a processor and not an annotator because Sherpa sends an annotator the text and metadata only, never the annotations of the NER before it.

from pymultirole_plugins.v1.schema import Annotation, Document

from pyannotators_entityfishing.candidates import EntityFishingCandidatesParameters, EntityFishingCandidatesProcessor

text = "Macron à Washington."
document = Document(
    text=text,
    metadata={"language": "fr"},
    annotations=[
        Annotation(start=0, end=6, labelName="Person", text="Macron"),
        Annotation(start=9, end=19, labelName="Location", text="Washington"),
    ],
)
[document] = EntityFishingCandidatesProcessor().process([document], EntityFishingCandidatesParameters(top_k=3))
for ann in document.annotations:
    print(ann.text, [(t.identifier, t.preferredForm, t.score) for t in ann.terms])

Development

The build is driven by Task and uv, with the shared stages coming from the python-archetype submodule.

Getting started

The stages live in a Git submodule, so clone with --recurse-submodules:

git clone --recurse-submodules git@bitbucket.org:kairntech/pyannotators_entityfishing.git
cd pyannotators_entityfishing
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task

Already cloned without it? The submodule directory is empty, and task fails on:

task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"

which means exactly that, and nothing worse:

git submodule update --init

Task is the only manual prerequisite. An archetype cannot bootstrap itself: uv and the Python interpreter install themselves on demand (every task that runs uv depends on an internal install-python task), but the thing that runs them does not. Make sure ~/.local/bin is on your PATH — that is where task and uv both land.

Running the pipeline

task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs

task with no argument is safe by construction: it runs every stage but the last, and that bound is computed from the STAGES list rather than written down. The last stage is the only one with an effect outside your machine.

STAGES, declared once in Taskfile.yml, is the single definition of the pipeline order — so what you run locally is what Jenkins runs.

Individual stages

uv.lock is not versioned here, so py:sync always resolves from scratch (--upgrade): a stale lock lying around on a machine would otherwise make you test and audit versions the CI never sees.

SBOM & vulnerability check

task py:sbom and task py:check-vulnerabilities wrap the underlying tools. To run them by hand:

uv sync --extra test --extra sbom
uv run cyclonedx-py environment -o sbom.cdx.json --output-format json
uv run pip-audit --skip-editable --format json --output audit-report.json

Ask for all the extras, not just sbom: uv sync synchronises rather than adds, so whatever the requested extras do not pull in gets removed. --extra sbom on its own uninstalls ruff, pytest-cov, coverage and dirty-equals, leaving an environment that can no longer lint or measure coverage.

--skip-editable skips the project itself, which no advisory database can know about. The tasks do the same, and pip-audit still exits non-zero when it finds a real vulnerability — which is the contract the CI relies on. Avoid --strict: it turns "dependency not found on PyPI" into a fatal error, so it breaks as soon as the local version is not a published one.