hsds-entity-resolution

Reusable Dagster component for incremental entity resolution on HSDS (Human Services Data Specification) records.

Install:

pip install hsds-entity-resolution

What this package does

Community 211 directories and social-services platforms often ingest the same organizations and services from multiple data partners. Over time those records diverge — different spellings, phone formats, partial addresses — making it hard to know which rows describe the same real-world entity.

hsds-entity-resolution provides a single Dagster component, EntityResolutionComponent, that runs an incremental seven-stage entity-resolution pipeline on your HSDS data:

The component is storage-agnostic: it accepts and emits Polars DataFrames, so you wire it into whatever persistence layer your Dagster project uses.

Prerequisites

• Python 3.10–3.13
• A Dagster project managed with the dg CLI
• Four upstream Dagster assets that supply denormalized HSDS entity data (see Input assets)

Quickstart

1. Discover the component

dg list components --package hsds_entity_resolution

2. Add to your component YAML

type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
attributes:
  team_id: my_team
  scope_id: regional
  entity_type: organization

That's enough to wire the component into your Dagster asset graph. The four required upstream assets default to the keys organization_entities, service_entities, previous_entity_index, and previous_pair_state_index — override them with the *_asset_key attributes if your project uses different names.

Input assets

The component depends on four upstream assets. Each must yield a Polars DataFrame (or be coercible to one).

organization_entities and service_entities

Denormalized HSDS entity records. Required columns:

Optional passthrough columns (display_name, display_description, alternate_name, short_description, application_process, fees_description, eligibility_description, resource_writer_name, assured_date, assurer_email, original_id, organization_original_id, organization_name, organization_id, services_rollup) are forwarded unchanged to the normalized entity cache and persistence artifacts.

previous_entity_index

The normalized_organization or normalized_service output from the previous run of this component. Pass an empty pl.DataFrame() on first run.

previous_pair_state_index

The pair_state_index output from the previous run of this component. Pass an empty pl.DataFrame() on first run. The component uses this to carry forward stable pair scores and detect pair identity continuity across incremental runs.

Component attributes

Output assets

The component emits fifteen assets. Using output_asset_prefix: ["my_prefix"] namespaces all keys under my_prefix/.

Tuning scoring constants

The default thresholds are calibrated for HSDS data. Override any constant via constants_overrides without subclassing:

type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
attributes:
  team_id: my_team
  scope_id: regional
  entity_type: organization
  constants_overrides:
    scoring:
      duplicate_threshold: 0.85
      maybe_threshold: 0.70
      nlp:
        fuzzy_threshold: 0.90

Key thresholds:

Multiple instances

Deploy one component instance per entity type and scope:

# organizations
type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
attributes:
  team_id: my_team
  scope_id: regional
  entity_type: organization
  output_asset_prefix: ["org_er"]

---

# services
type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
attributes:
  team_id: my_team
  scope_id: regional
  entity_type: service
  output_asset_prefix: ["svc_er"]

Pair Judge contract

The engine fixes what a judge reads and what it answers; it ships no questions or model.

• hsds_entity_resolution.judge.PairState: both records as HSDS text (name, alternate name, description, short description, eligibility, fees, application process, taxonomy names, sites, phones as digits, websites, organization name and description), a code-computed site comparison word (same address, same city, different, unknown), the prior score as a bucket in words, and one free-text source_profile slot per side. The engine fills the HSDS fields and leaves the profile slots empty unless you pass source_profiles={source_schema: text}.
• hsds_entity_resolution.judge.PairAnswers: same_site, same_offering, physically_delivered (probabilities; the last two are null for organization pairs), and organization_relation as a distribution over same, parent_and_chapter, affiliated, unrelated, cannot_tell, plus model_id and question_set_version. The judge never answers "duplicate"; compose that in your own code.
• hsds_entity_resolution.judge.PairJudge: a protocol with judge(states) -> answers and a state_token_budget. Size states for the smallest model you may switch to (SMALLEST_MODEL_STATE_BUDGET_TOKENS, 32k).
• ReferenceJudge answers 0.5 everywhere and a uniform relation, so the pipeline runs end to end without a model.

from hsds_entity_resolution.core import run_incremental
from hsds_entity_resolution.judge import ReferenceJudge

result = run_incremental(..., judge=ReferenceJudge(), source_profiles={"source_a": "..."})
result.judge_answers  # one row per scored pair, JUDGE_ANSWERS_SCHEMA

judge_scored_pairs(...) runs the same stage on its own, for hosts that run the judge as a separate step.

Candidate admission

One generic rule decides which pairs are scored (2.1.0):

• Informative Keys. Two records that share a value of name, phone, email, website or address (street line required) are a candidate, whatever their embedding similarity, provided the field is informative in both records' source schemas. A field is informative where its values mostly identify a record; a value repeated across hundreds of rows (a category label used as a name) is not.
• Embedding floor. Records at or above blocking.similarity_threshold are a candidate, up to blocking.max_candidates_per_entity per anchor.
• Structural Exclusions. A caller-supplied callable can veto any pair. Excluded pairs are returned as excluded_pairs with the reason, never dropped silently.

Pass both to run_incremental (or run_incremental_until_candidates):

from hsds_entity_resolution.core.admission import build_informative_key_table

informative_keys = build_informative_key_table(
    ratios={"SOURCE_A": {"name": 0.12, "phone": 0.74}},  # distinct-value ratios you measure
    cutoff=0.3,
    overrides={"SOURCE_B": {"name": True}},
)

def exclude(entity_a, entity_b):
    return "different_by_construction" if ... else None

result = run_incremental(..., informative_keys=informative_keys, structural_exclusion=exclude)
result.excluded_pairs  # pair_key, entity ids, schemas, similarity, exclusion_reason

Without them every key field is informative and nothing is excluded. Scoring does not depend on how a pair was admitted: pair_rules still pick scoring overrides from the entity type and the two source schemas. source_policy.admission_rules and blocking.overlap_prefilter_channels were removed; passing them fails validation with a migration message.

Normalisers

hsds_entity_resolution.normalize holds pure, idempotent functions on plain strings — normalize_phone (digits plus extension), normalize_email, normalize_url, normalize_address_component, normalize_postal_code — usable outside the engine.

Migrating from hsds-record-matcher 1.x

• Install name. The distribution is now hsds-entity-resolution, matching the import name hsds_entity_resolution. Uninstall hsds-record-matcher first.
• ML scoring section removed (1.2.0). scoring.ml, scoring.calibration, scoring.ml_section_weight and pair-rule ml_section_weight / ml_gate_threshold now fail validation. deterministic_section_weight + nlp_section_weight must equal 1.0; to keep your scores, divide each by their old sum. Scored pairs no longer carry ml_section_score, legacy_confidence_score, shadow_confidence_score, shadow_log_odds or calibration_version, and run_incremental no longer takes taxonomy_embeddings.
• Judge stage (2.0.0). run_incremental accepts judge and source_profiles; without a judge every output is unchanged.
• Logging. Core modules log through logging.getLogger("hsds_entity_resolution...") instead of Dagster's logger. Add hsds_entity_resolution to your Dagster python_logs.managed_python_loggers to see them in the Dagster UI.

Links

• Source: github.com/211-Connect/hsds-entity-resolution
• Issues: github.com/211-Connect/hsds-entity-resolution/issues
• PyPI: pypi.org/project/hsds-entity-resolution