pyprocessors-reconciliation 0.6.21

Sherpa reconciliation processor

pyprocessors_reconciliation

Reconciliation annotations coming from different annotators.

Installation

pip install pyprocessors-reconciliation

Overview

ReconciliationProcessor is a pymultirole processor plugin that reconciles overlapping annotations produced by multiple annotators (NER models, knowledge-base linkers, white/kill lists) into a single coherent set.

The processor is registered under the pyprocessors.plugins entry point as reconciliation.

Parameters

Parameter	Type	Default	Description
type	ReconciliationType	linker	Reconciliation strategy (currently only linker)
kill_label	str | None	None	Label whose annotations suppress matching model annotations
white_label	str | None	None	Label treated as authoritative (terms stripped so it acts like a model annotation)
whitelisted_lexicons	list[str] | None	None	Lexicons whose annotations are duplicated as term-free model candidates
person_label	str | None	None	Label used to identify person annotations for last-name resolution
remove_suspicious	bool	True	Drop model annotations that contain no capitalised word (numbers, percentages, etc.)
resolve_lastnames	bool	False	Resolve isolated last names / first names using full names seen earlier in the document

How it works

1. Sentence filtering — sentence-labelled annotations are removed before processing.
2. Whitelist marking (mark_whitelisted) — annotations matching white_label have their terms cleared so they behave like model candidates; annotations from whitelisted_lexicons get a term-free duplicate added alongside the original.
3. Grouping (group_annotations) — annotations are grouped by their first term's lexicon (empty string = model / no-lexicon). Same-span annotations in the same group have their term lists merged and deduplicated.
4. Linker consolidation (consolidate_linker):
   • Suspicious model annotations (no capitalised word) are optionally dropped.
   • Kill-list annotations suppress matching model annotations.
   • KB annotations at the same span enrich the matching model annotation with their terms.
   • Overlapping or mismatched-label KB matches are logged as warnings and skipped.
5. Last-name resolution — when resolve_lastnames=True, isolated person names (single token) are resolved to the full-name annotation seen earliest in the document.

Developing

Prerequisites

uv is required as the package manager.

pip install uv

Clone the repository:

git clone https://github.com/oterrier/pyprocessors_reconciliation
cd pyprocessors_reconciliation

Install in development mode

uv sync --extra test

Running the test suite

uv run pytest

Linting and formatting

uv run ruff check .
uv run ruff format .

Building the documentation

uv run --extra docs sphinx-build docs docs/_build

The built documentation is available at docs/_build/index.html.

Building and publishing

uv build
uv publish

SBOM & vulnerability check

Install the SBOM dependencies:

uv sync --extra sbom

Generate a CycloneDX SBOM from the current environment:

uv run cyclonedx-py environment -o sbom.cdx.json --output-format json

Audit dependencies for known vulnerabilities:

uv run pip-audit --format json --output audit-report.json

To fail on any known vulnerability (useful in CI):

uv run pip-audit --strict