Portiere

Local-first clinical data mapping with human-in-the-loop review — PHI never leaves your machine.

Documentation · Quick Start · Examples · Issues

What is Portiere?

Mapping clinical data to standard models like OMOP CDM, FHIR R4, HL7 v2, and OpenEHR is one of the most time-consuming and error-prone tasks in health informatics. It typically requires domain experts to manually map hundreds of source fields and thousands of clinical codes — a process that can take weeks or months.

Portiere puts that work on rails: a 5-stage pipeline (profiling, schema mapping, concept mapping, ETL generation, validation) that runs entirely on your own infrastructure and routes every mapping to the right tier — auto-accept, needs-review, or manual — so your terminologists spend their time only where judgment is required. AI (clinical embeddings, reranking, optional BYO-LLM verification) assists the process; it never replaces the reviewer, and it never requires a cloud.

Since v0.4.0 the data-sovereignty claim is executable, not aspirational:

portiere doctor --assert-no-egress   # exit 0 ⇒ this configuration cannot send data anywhere

PortiereConfig(offline=True) structurally rejects any remote-provider configuration (COMPLIANCE.md · threat model), and value-level PHI scrubbing cleans examples and samples before they reach any artifact or prompt.

flowchart LR
    A["Source Data"] --> B["Ingest & Profile (Stage 1)"]
    B --> C["Schema Mapping (Stage 2)"]
    C --> D["Concept Mapping (Stage 3)"]
    D --> E["ETL Generation (Stage 4)"]
    E --> F["Validation (Stage 5)"]

Portiere combines clinical-domain embeddings (SapBERT as default model), lexical search (BM25s), cross-encoder reranking, and optional LLM verification to achieve high-accuracy mappings with confidence routing — automatically accepting high-confidence results while flagging uncertain ones for human review.

Key Features

• Multi-Standard Support — OMOP CDM v5.4 (19 entities, incl. core vocabulary tables), FHIR R4 (18 resources), HL7 v2.5.1, OpenEHR 1.0.4 (extensible via YAML). Full coverage details below.
• Real Plausibility Validation — Kahn-style cross-table and ValueSet checks via a hybrid YAML DSL + Python rules. Plausibility guide.
• Reproducibility Manifest — every pipeline run emits a versioned manifest.lock.json capturing model identity, vocab fingerprints, source-data hash, threshold snapshot, and per-stage entries. portiere replay reconstructs the project from a manifest. Reproducibility guide.
• Bundled Quickstart — portiere quickstart runs the full OMOP pipeline against ~20 synthetic patients in <60 s, fully offline (no SapBERT download, no Athena required).
• AI-Powered Mapping — SapBERT embeddings (default) + cross-encoder reranking + optional LLM verification.
• 9 Knowledge Backends — BM25s, FAISS, Elasticsearch, ChromaDB, PGVector, MongoDB, Qdrant, Milvus, Hybrid (RRF fusion).
• BYO-LLM — Bring your own LLM: OpenAI, Anthropic Claude, AWS Bedrock, Ollama (local).
• Pluggable Engines — Polars (default), PySpark / Databricks, Pandas, DuckDB.
• Standalone ETL Artifacts — Generated ETL scripts run without the SDK.
• Agent & analytics integrations (v0.5.0) — MCP server (portiere mcp), LangChain tools (get_langchain_tools()), and dbt project generation (portiere dbt). All offline. MCP · LangChain · dbt.
• Confidence Routing — Auto-accept, needs-review, and manual tiers with human-in-the-loop. Streamlit-based Mapping Review UI ships in v0.3.1: pip install "portiere-health[review]" then portiere review <project-dir>.
• Cross-Standard Mapping — Transform between standards (OMOP ↔ FHIR, HL7v2 → FHIR, OMOP → OpenEHR).
• Local-First — All processing runs on your machine; no cloud dependency.

Quickstart in 60 seconds

pip install portiere-health[polars,quality]
portiere quickstart

Runs the full pipeline (ingest → schema-map → concept-map → ETL → validate) against bundled synthetic data. Produces a real OMOP mapping plus a reproducibility manifest. No network calls, no extra setup.

Quick Start

Install

pip install portiere-health

# With a compute engine (pick one)
pip install "portiere-health[polars]"    # Lightweight (recommended)
pip install "portiere-health[spark]"     # Large-scale / Databricks
pip install "portiere-health[pandas]"    # Prototyping

Map Clinical Data to OMOP CDM

import portiere
from portiere.engines import PolarsEngine

# Initialize a project
project = portiere.init(
    name="Hospital OMOP Migration",
    engine=PolarsEngine(),
    target_model="omop_cdm_v5.4",
    vocabularies=["SNOMED", "LOINC", "RxNorm", "ICD10CM"],
)

# Add and profile a data source
source = project.add_source("patients.csv")
profile = project.profile(source)

# AI-powered schema mapping (source columns → OMOP tables)
schema_map = project.map_schema(source)

# AI-powered concept mapping (clinical codes → standard concepts)
concept_map = project.map_concepts(codes=["E11.9", "I10", "R73.03"])

# Review mappings
schema_map.summary()
concept_map.summary()

# Generate and run ETL
result = project.run_etl(source, schema_map, concept_map)

Cross-Standard Mapping (OMOP → FHIR)

project = portiere.init(
    name="FHIR Export",
    engine=PolarsEngine(),
    task="cross_map",
    source_standard="omop_cdm_v5.4",
    target_model="fhir_r4",
)

Installation

Core Package

pip install portiere-health

Optional Extras

Install only what you need:

Requirements: Python 3.10+

How It Works

Portiere implements a 5-stage AI pipeline for clinical data transformation:

Stage 1: Ingest & Profile

Connects to your data source (CSV, Parquet, databases) and extracts schema metadata — column names, types, cardinality, detected code columns, and PHI indicators.

Export an aggregated data-profile report (HTML + CSV) across one or more sources (v0.3.2):

portiere profile-report data/*.parquet -o outputs/ --format parquet --system HIS
# → outputs/data_profile.html, data_profile.csv, data_profile_summary.csv

The report shows per-source completeness bars and, per column, presence / missing / distinct / top value (share) / value-length / example. For the Excel deliverables teams actually exchange — a White-Rabbit-parity scan workbook, the per-table schema-mapping working document, and the OMOP source_to_concept_map workbook — see portiere workbook (docs/workbooks.md, xlsx extra) (v0.4.0). Empty and whitespace-only strings count as missing by default (--no-empty-as-missing to count only true nulls). Also available from Python via portiere.quality.build_source_profile + export_profile_report.

Stage 2: Schema Mapping

Maps source columns to target standard entities using a fusion of:

• Pattern matching — Regex patterns defined in YAML standard files
• Embedding similarity — SapBERT clinical embeddings for semantic matching
• Cross-encoder reranking — Precision reranking of top candidates

Stage 3: Concept Mapping

Maps clinical codes (ICD-10, CPT, local codes) to standard vocabularies (SNOMED CT, LOINC, RxNorm) through:

1. Direct code lookup — Exact match in knowledge base
2. Knowledge layer search — BM25s lexical / FAISS vector / Hybrid search
3. Cross-encoder reranking — Rerank top-k candidates for precision
4. LLM verification — Optional AI verification for medium-confidence mappings
5. Confidence routing — Auto-accept (>0.95), needs-review (0.70–0.95), manual (<0.70)

Stage 4: ETL Generation

Generates standalone ETL scripts (Spark, Polars, or Pandas) and lookup tables (CSV) that run without the Portiere SDK — no vendor lock-in.

Stage 5: Validation

Post-ETL data quality checks using Great Expectations, with standards-aware conformance for all supported models (OMOP, FHIR, HL7, OpenEHR, custom YAML):

• Completeness — Non-null percentages for required fields
• Conformance — Type and constraint compliance derived from YAML field metadata
• Plausibility — Domain-specific clinical rules

Supported Standards

Standards are defined as YAML files and are fully extensible — you can define custom hospital CDMs or registry schemas.

Cross-Standard Mapping

Built-in crossmaps for transforming between standards:

Custom Standards

Portiere is not limited to built-in standards. You can define any clinical data model — a hospital CDM, a disease registry schema, a research database, a legacy warehouse — as a YAML file and use it identically to built-in standards.

Define a Custom Standard (YAML)

Create a .yaml file with the following structure:

name: "hospital_cdm_v1"
version: "1.0"
standard_type: "relational"
organization: "General Hospital Research"
description: "Internal clinical data model for General Hospital"

entities:
  patients:
    description: "Core patient demographics"
    fields:
      patient_id:
        type: integer
        required: true
        description: "Unique patient identifier"
        ddl: "INTEGER PRIMARY KEY"
      date_of_birth:
        type: date
        description: "Patient date of birth"
        ddl: "DATE NOT NULL"
      sex:
        type: string
        description: "Biological sex (M/F/U)"
        ddl: "VARCHAR(1)"

    # Fast pattern matching: source column name → target field
    source_patterns:
      patient_id: "patient_id"
      subject_id: "patient_id"
      dob: "date_of_birth"
      birth_date: "date_of_birth"
      gender: "sex"
      sex: "sex"

    # Embedding descriptions: optimized text for AI semantic matching
    # Write what a clinician would search for, not just the field name
    embedding_descriptions:
      patient_id: "unique patient identifier number"
      date_of_birth: "patient birth date birthday date of birth"
      sex: "biological sex gender male female M F"

  encounters:
    description: "Hospital visits and admissions"
    fields:
      encounter_id:
        type: integer
        required: true
        description: "Unique encounter identifier"
        ddl: "INTEGER PRIMARY KEY"
      admit_date:
        type: datetime
        description: "Admission date and time"
        ddl: "TIMESTAMP NOT NULL"
      encounter_type:
        type: string
        description: "Type of encounter (inpatient, outpatient, ED)"
        ddl: "VARCHAR(20)"

    source_patterns:
      encounter_id: "encounter_id"
      visit_id: "encounter_id"
      hadm_id: "encounter_id"
      admit_date: "admit_date"
      admittime: "admit_date"
      visit_type: "encounter_type"

    embedding_descriptions:
      encounter_id: "hospital encounter visit admission identifier"
      admit_date: "admission date time when patient was admitted"
      encounter_type: "visit type inpatient outpatient emergency department"

Use Your Custom Standard

import portiere
from portiere.engines import PolarsEngine

# Reference via "custom:" prefix — works anywhere target_model is accepted
project = portiere.init(
    name="Hospital Migration",
    engine=PolarsEngine(),
    target_model="custom:/path/to/hospital_cdm_v1.yaml",
)

source = project.add_source("patients.csv")
schema_map = project.map_schema(source)
concept_map = project.map_concepts(codes=["E11.9", "I10"])
result = project.run_etl(source, schema_map, concept_map)

Or load directly for inspection:

from portiere.standards import YAMLTargetModel

model = YAMLTargetModel("/path/to/hospital_cdm_v1.yaml")
print(model.get_schema())      # entity → [fields]
print(model.get_source_patterns())  # source column hints

You can also ship your custom standard as a built-in by placing the YAML in src/portiere/standards/ — it will then be loadable by name:

model = YAMLTargetModel.from_name("hospital_cdm_v1")

Column Naming Guide

Portiere's schema mapper uses two strategies in sequence: exact pattern matching (fast, zero-cost) then embedding similarity (AI-powered). Understanding both helps you get higher auto-accept rates.

Strategy 1 — Source Patterns (rule-based, highest priority)

Each entity in a standard YAML defines source_patterns — a dictionary mapping source column names to target fields. Matches here are always accepted, regardless of confidence score.

Built-in OMOP patterns include common aliases:

To maximize pattern EHR in your own standard, add all known aliases to source_patterns in your YAML:

source_patterns:
  patient_id: "person_id"     # exact name
  pid: "person_id"            # short alias
  subject_id: "person_id"     # research alias
  pt_id: "person_id"          # abbreviated
  medical_record_number: "person_id"  # verbose

Strategy 2 — Embedding Similarity (semantic, AI-powered)

When no pattern matches, the mapper encodes both the source column name and the embedding_descriptions into vectors using SapBERT, then finds the closest target field by cosine similarity.

What to write in embedding_descriptions:

Write natural-language phrases a clinician would use to describe what that column contains — not just a rephrasing of the field name.

# ❌ Too literal — just re-states the name
embedding_descriptions:
  admit_date: "admission date"
  dx_code: "diagnosis code"

# ✅ Rich synonyms and clinical context — maximizes semantic recall
embedding_descriptions:
  admit_date: "hospital admission date time when patient was admitted inpatient start"
  dx_code: "ICD diagnosis code ICD-10-CM ICD-9 disease condition clinical code"

Naming your source columns well also helps. The source column name itself is encoded alongside the description. Prefer descriptive names over cryptic abbreviations:

Confidence Tiers

After matching, every column receives a confidence score:

Tune these thresholds to match your project's risk tolerance:

from portiere import PortiereConfig, ThresholdsConfig
from portiere.config import SchemaMappingThresholds

config = PortiereConfig(
    thresholds=ThresholdsConfig(
        schema_mapping=SchemaMappingThresholds(
            auto_accept=0.90,   # lower → more auto-accepts
            needs_review=0.60,  # lower → fewer manual items
        )
    )
)

Full Workflow with Review

schema_map = project.map_schema(source)

# Inspect what needs review
for item in schema_map.needs_review():
    print(f"{item.source_column} → {item.target_table}.{item.target_column} "
          f"(confidence={item.confidence:.2f})")
    for c in item.candidates[:3]:
        print(f"  candidate: {c['target_table']}.{c['target_column']} ({c['confidence']:.2f})")

# Approve, override, or reject
schema_map.approve("patient_name")
schema_map.override("pt_zip", target_table="location", target_column="zip")
schema_map.reject("internal_audit_flag")

# Approve all remaining items
schema_map.approve_all()
schema_map.finalize()

Knowledge Layer Backends

Hybrid Search Example

from portiere import PortiereConfig, KnowledgeLayerConfig

config = PortiereConfig(
    knowledge_layer=KnowledgeLayerConfig(
        backend="hybrid",
        hybrid_backends=["bm25s", "faiss"],
        hybrid_fusion="rrf",  # Reciprocal Rank Fusion
    )
)

LLM Providers

Portiere supports Bring-Your-Own-LLM for concept verification:

from portiere import PortiereConfig, LLMConfig

config = PortiereConfig(
    llm=LLMConfig(
        provider="openai",
        model="gpt-4o-mini",
        api_key="sk-...",
    )
)

Configuration

Portiere auto-discovers configuration from multiple sources (in priority order):

1. Python Objects

from portiere import PortiereConfig, EmbeddingConfig, KnowledgeLayerConfig

config = PortiereConfig(
    target_model="omop_cdm_v5.4",
    embedding=EmbeddingConfig(
        provider="huggingface",
        model="cambridgeltl/SapBERT-from-PubMedBERT-fulltext",
    ),
    knowledge_layer=KnowledgeLayerConfig(backend="bm25s"),
)

2. YAML File (portiere.yaml)

target_model: omop_cdm_v5.4
storage: local

embedding:
  provider: huggingface
  model: cambridgeltl/SapBERT-from-PubMedBERT-fulltext

knowledge_layer:
  backend: bm25s

llm:
  provider: openai
  model: gpt-4o-mini

thresholds:
  auto_accept: 0.95
  needs_review: 0.70

3. Environment Variables

export PORTIERE_TARGET_MODEL=omop_cdm_v5.4
export PORTIERE_LLM__PROVIDER=openai
export PORTIERE_LLM__API_KEY=sk-...
export PORTIERE_KNOWLEDGE_LAYER__BACKEND=faiss

Building the Knowledge Layer

Before concept mapping, build a searchable index from standard vocabularies (e.g., OHDSI Athena):

from portiere import build_knowledge_layer, PortiereConfig

config = PortiereConfig()

stats = build_knowledge_layer(
    vocabulary_dir="./data/athena/",
    config=config,
    vocabularies=["SNOMED", "LOINC", "RxNorm", "ICD10CM"],
)

print(f"Indexed {stats['total_concepts']:,} concepts")

Documentation

Project Structure

portiere/
├── src/portiere/
│   ├── __init__.py          # Public API: init(), PortiereProject, configs
│   ├── config.py            # Configuration with auto-discovery
│   ├── project.py           # Unified project interface
│   ├── exceptions.py        # Error hierarchy
│   ├── stages/              # 5-stage pipeline implementation
│   ├── engines/             # Compute engines (Polars, Spark, Pandas, DuckDB)
│   ├── knowledge/           # Knowledge layer backends (9 backends)
│   ├── embedding/           # Embedding providers & gateway
│   ├── llm/                 # LLM providers & gateway
│   ├── local/               # Local AI components (schema mapper, concept mapper)
│   ├── artifacts/           # ETL code generation (Jinja2 templates)
│   ├── runner/              # ETL execution engine
│   ├── quality/             # Data quality validation (Great Expectations)
│   ├── standards/           # Clinical standard YAML definitions & crossmaps
│   ├── storage/             # Storage backends (local filesystem)
│   └── models/              # Pydantic data models
├── tests/                   # 36 test modules, 689 tests
├── docs/
│   ├── documentations/      # 22 guides and references
│   └── notebooks_examples/  # 19 Jupyter notebook examples
├── pyproject.toml           # Package configuration (hatchling)
└── LICENSE                  # Apache 2.0

Standards Coverage

v0.2.0 ships an opinionated subset of each standard — the required + commonly-used optional fields and entities, not every column from the spec. Long-tail fields are accepted contributions.

For a programmatic listing: python -c "from portiere.standards import YAMLTargetModel; print(sorted(YAMLTargetModel.from_name('omop_cdm_v5.4').get_schema()))".

Limitations

Portiere is in active development. Current limitations (will be addressed in upcoming releases):

• Standards coverage is partial. OMOP CDM v5.4: 19 of ~37 tables — but the covered set includes the core clinical-event tables that carry most rows in a typical conversion (person, visit_occurrence, condition_occurrence, drug_exposure, measurement, observation, procedure_occurrence). FHIR R4: 18 of 145 resources. Check the support matrix against your protocol; PRs to extend coverage are welcome. See how Portiere compares.
• FHIR profile coverage is US Core + mCODE only. v0.3.0 validates against US Core 6.1.0 (10 resource types); v0.3.1 adds mCODE STU3 2.0.0 (5 oncology profiles). IPS, mCODE-extended (treatments, additional staging) planned for v0.5.0.
• Mapping Review UI covers schema mappings. v0.3.1 ships the Streamlit-based UI for schema-mapping review (approve / override / reject); concept-mapping review planned for v0.4.x.
• PHI scrubbing is opt-in and structural-first. v0.4.0 ships value-level detection/scrubbing (docs/phi-scrubbing.md): a built-in regex backend (email/phone/MRN/national-ID/date) plus optional Presidio NER ([phi] extra) for person names. Off by default in v0.4.0 (scrub_phi=True to enable); planned default-on in v1.0. Unlabeled free-text names require the NER extra.
• SNOMED CT and CPT-4 not bundled. Both have licensing constraints. portiere quickstart operates on bundled ICD-10-CM/LOINC/RxNorm only; for SNOMED, see vocabulary setup.
• Replay reproduces stages best-effort. portiere replay --auto-replay re-runs deterministic stages (ingest, validate) and records LLM-bound stages (schema, concept, ETL) as UNAVAILABLE. Full BYO-LLM rehydration planned for v0.3.x. Within-tolerance outputs may still differ ±1% due to LLM sampling. See reproducibility guide.
• Benchmark coverage. v0.3.1 publishes the ICD-10-CM → SNOMED 4-row ablation (BM25 / FAISS / hybrid / USAGI baseline). LOINC / RxNorm pairs planned for v0.3.x.

Roadmap

• v0.3.2 (shipped): aggregated data-profile report export (portiere profile-report, HTML + CSV) with additive engine profile enrichment.
• v0.3.x: Full BYO-LLM rehydration for replay --auto-replay LLM-bound stages; additional benchmark pairs (LOINC, RxNorm); active-learning loop on review-UI override decisions.
• v0.4.0 (this release): executable no-egress guarantee (offline mode + portiere doctor); value-level PHI scrubber; compliance docs (COMPLIANCE.md + threat model); reranker-ablation harness + score-blending fix; Docker; published docs site.
• v0.5.0 (shipped): MCP / LangChain / dbt integration surface; domain-reranker decision harness (scripts/reranker_spike.py).
• v0.5.x: concept-mapping review UI page; second benchmark vocabulary pair (RxNorm→ATC); auto-tier precision metric.
• v0.5.0+: PCORnet / Sentinel / i2b2 / CDISC SDTM CDMs; clinical NLP path (scispaCy / GLiNER-clinical); OHDSI DataQualityDashboard parity.

Each release tracks via GitHub Milestones; please open issues or PRs against the relevant milestone. See specs/ for the design docs behind each release.

Benchmarks

ICD-10-CM → SNOMED concept mapping, n=1,000 held-out, Athena 2026-04-30, against the OHDSI CONCEPT_RELATIONSHIP gold standard — including the head-to-head against OHDSI's own USAGI tool (v0.4.0, measured after the score-blending fix):

Every Portiere configuration beats the USAGI baseline (best top-1 0.303 vs 0.262, +16% relative). At default thresholds, 51.8% of codes route to auto-accept — half the workload never needs a reviewer. Honest history: v0.3.x published rows where "the AI lost to plain BM25"; v0.4.0 found and fixed the score-scale defect behind that result (the full ablation, including where each configuration is weak, is in the benchmark doc).

Reproduce any row: portiere benchmark athena-icd-snomed --backend bm25s|faiss|hybrid --athena-dir <path>. Stratified sampling (opt-in): add --stratify-by domain to sample proportionally across Athena domains. Methodology: docs/benchmarks/athena-icd-snomed.md.

FHIR Interoperability

v0.3.0 + v0.3.1 ship:

• US Core 6.1.0 profile validation — Project.validate(fhir_profile="us-core-6.1.0") or portiere validate --fhir-profile us-core-6.1.0 --input resources.json. 10 resource types (Patient, Practitioner, Organization, Encounter, Condition, Observation, MedicationRequest, AllergyIntolerance, Procedure, DocumentReference).
• mCODE STU3 2.0.0 profile validation (v0.3.1) — --fhir-profile mcode-2.0.0. 5 core oncology profiles (CancerPatient, PrimaryCancerCondition, CancerDiseaseStatus, CancerStage, TNMStageGroup). Profile is keyed on meta.profile URL substring (not resourceType).
• Bundle + NDJSON export — portiere export --format bundle --out out.json or --format ndjson --out out_dir/. Optional --fhir-profile us-core-6.1.0|mcode-2.0.0 validates before writing.

Install the FHIR extra: pip install "portiere-health[fhir]". See: docs/fhir-profile-validation.md, docs/fhir-bundle-export.md.

Mapping Review UI (v0.3.1)

Streamlit-based human-in-the-loop review of AI-generated schema mappings:

pip install "portiere-health[review]"
portiere review <project-dir>        # opens http://127.0.0.1:8501

Reviewer actions: approve (accept AI suggestion), reject (mark unmappable), override (replace with table.column). Decisions persist to <project_dir>/schema_mappings/schema_mapping_reviewed.json next to the original — originals never modified. Local-only by default (no auth); --host 0.0.0.0 opt-in for LAN demos.

See: docs/mapping-review-ui.md. Concept-mapping review planned for v0.4.x.

Reproducibility (v0.3.1)

portiere replay --auto-replay manifest.lock.json

Re-runs each recorded stage and compares outputs to the manifest within tolerance bands. Stages whose dependencies are unavailable (LLM, knowledge layer) record as UNAVAILABLE and do not fail the report; full BYO-LLM rehydration is a v0.3.x roadmap item. See docs/reproducibility.md.

Star History

Contributing

We welcome contributions! Here's how to get started:

# Clone the repository
git clone https://github.com/Cuspal/portiere.git
cd portiere

# Create a virtual environment
python -m venv .venv
source .venv/bin/activate

# Install in development mode
pip install -e ".[dev,docs,polars,quality]"

# Run tests
pytest

# Run linter
ruff check src/ tests/

# Run type checker
mypy src/portiere/

Please read our contributing guidelines before submitting a pull request.

License

Portiere is licensed under the Apache License 2.0.

Copyright 2026 Cuspal Co. Ltd.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Citation

If you use Portiere in your research, please cite:

@software{portiere2026,
  title     = {Portiere: AI-Powered Clinical Data Mapping SDK},
  author    = {Cuspal Co.,Ltd.},
  year      = {2026},
  url       = {https://github.com/Cuspal/portiere},
  license   = {Apache-2.0},
}