Portiere
Local-first clinical data mapping with human-in-the-loop review — PHI never leaves your machine.
Documentation · Quick Start · Examples · Issues
What is Portiere?
Mapping clinical data to standard models like OMOP CDM, FHIR R4, HL7 v2, and OpenEHR is one of the most time-consuming and error-prone tasks in health informatics. It typically requires domain experts to manually map hundreds of source fields and thousands of clinical codes — a process that can take weeks or months.
Portiere puts that work on rails: a 5-stage pipeline (profiling, schema mapping, concept mapping, ETL generation, validation) that runs entirely on your own infrastructure and routes every mapping to the right tier — auto-accept, needs-review, or manual — so your terminologists spend their time only where judgment is required. AI (clinical embeddings, reranking, optional BYO-LLM verification) assists the process; it never replaces the reviewer, and it never requires a cloud.
Since v0.4.0 the data-sovereignty claim is executable, not aspirational:
portiere doctor --assert-no-egress # exit 0 ⇒ this configuration cannot send data anywhere
PortiereConfig(offline=True) structurally rejects any remote-provider configuration (COMPLIANCE.md · threat model), and value-level PHI scrubbing cleans examples and samples before they reach any artifact or prompt.
flowchart LR
A["Source Data"] --> B["Ingest & Profile (Stage 1)"]
B --> C["Schema Mapping (Stage 2)"]
C --> D["Concept Mapping (Stage 3)"]
D --> E["ETL Generation (Stage 4)"]
E --> F["Validation (Stage 5)"]
Portiere combines clinical-domain embeddings (SapBERT as default model), lexical search (BM25s), cross-encoder reranking, and optional LLM verification to achieve high-accuracy mappings with confidence routing — automatically accepting high-confidence results while flagging uncertain ones for human review.
Key Features
- Source-scoped review (v0.6.0) — Each source keeps its own schema and concept mappings. The SDK and review UI share saved decisions, and revision checks reject stale edits. Review and migration guide.
- Reusable concept review files (v0.6.0) — JSON and CSV exports retain source/revision context. Explicit pending or rejected decisions stay out of executable lookups and approved exports.
- Multi-Standard Support — OMOP CDM v5.4 (19 entities, incl. core vocabulary tables), FHIR R4 (18 resources), HL7 v2.5.1, OpenEHR 1.0.4 (extensible via YAML). Full coverage details below.
- Real Plausibility Validation — Kahn-style cross-table and ValueSet checks via a hybrid YAML DSL + Python rules. Plausibility guide.
- Reproducibility Manifest — every pipeline run emits a versioned
manifest.lock.jsoncapturing model identity, vocab fingerprints, source-data hash, threshold snapshot, and per-stage entries.portiere replayreconstructs the project from a manifest. Reproducibility guide. - Bundled Quickstart —
portiere quickstartprofiles three synthetic patients, generates mapping proposals, and validates reviewed demographic ETL, fully offline (no model downloads or Athena setup). - AI-Powered Mapping — SapBERT embeddings (default) + cross-encoder reranking + optional LLM verification.
- 9 Knowledge Backends — BM25s, FAISS, Elasticsearch, ChromaDB, PGVector, MongoDB, Qdrant, Milvus, Hybrid (RRF fusion).
- BYO-LLM — Bring your own LLM: OpenAI, Anthropic Claude, AWS Bedrock, Ollama (local).
- Pluggable Engines — Polars (default), PySpark / Databricks, Pandas, DuckDB.
- Standalone ETL Artifacts — Generated ETL scripts run without the SDK.
- Agent & analytics integrations (v0.5.0) — MCP server (
portiere mcp), LangChain tools (get_langchain_tools()), and dbt project generation (portiere dbt). All offline. MCP · LangChain · dbt. - Confidence Routing — Auto-accept, needs-review, and manual tiers with human-in-the-loop. Streamlit-based Mapping Review UI ships in v0.3.1:
pip install "portiere-health[review]"thenportiere review <project-dir>. - Cross-Standard Mapping — Transform between standards (OMOP ↔ FHIR, HL7v2 → FHIR, OMOP → OpenEHR).
- Local-First — All processing runs on your machine; no cloud dependency.
Offline quickstart
python -m pip install "portiere-health[polars,quality]"
portiere quickstart
Runs ingest → schema-map → concept-map → ETL → validate against bundled synthetic data. Produces a three-row person.csv using bundled demographic review decisions, diagnosis concept proposals for separate review, and a reproducibility manifest. It prints PASS/FAIL/SKIP for each stage and exits successfully only when all required stages pass. After installation, this demo requires no network access or model downloads.
Use portiere quickstart --output-dir ./demo-output to choose where artifacts go. Each run writes to a new ETL directory. The vocabulary is simplified for demonstration; it does not provide a clinical ICD-to-SNOMED conversion.
Quick Start
Install
pip install portiere-health
# With a compute engine (pick one)
pip install "portiere-health[polars]" # Lightweight (recommended)
pip install "portiere-health[spark]" # Large-scale / Databricks
pip install "portiere-health[pandas]" # Prototyping
Map Clinical Data to OMOP CDM
For your own data, first configure a knowledge layer and any optional models in portiere.yaml following the vocabulary setup guide. Inspect and save reviewed mapping decisions before running ETL; printing a summary does not approve them.
import portiere
from portiere.engines import PolarsEngine
from portiere.config import PortiereConfig
# Initialize a project
project = portiere.init(
name="Hospital OMOP Migration",
engine=PolarsEngine(),
target_model="omop_cdm_v5.4",
vocabularies=["SNOMED", "LOINC", "RxNorm", "ICD10CM"],
config=PortiereConfig.from_yaml("portiere.yaml"),
)
# Add and profile a data source
source = project.add_source("patients.csv")
profile = project.profile(source)
# AI-powered schema mapping (source columns → OMOP tables)
schema_map = project.map_schema(source)
# AI-powered concept mapping (clinical codes → standard concepts)
concept_map = project.map_concepts(source, code_columns=["diagnosis_code"])
# Review mappings
print(schema_map.summary())
print(concept_map.summary())
# After reviewing and saving the mappings, generate and run ETL
result = project.run_etl(
source,
output_dir="./output",
schema_mapping=schema_map,
concept_mapping=concept_map,
)
print(result.summary())
Version 0.6.0 keeps each source's mappings separate and checks revisions when saving reviews. For source selection, CSV/JSON review exchange, and upgrading from 0.5.x, see Review and reuse mappings. JSON exports now use a versioned envelope; saving concepts no longer refreshes an implicit source_to_concept_map.csv. Export that projection explicitly after review.
This release establishes mapping ownership and review consistency. Explicit concept destination compilation, frozen mapping bundles and a unified run report remain planned; inspect generated concept columns against your target standard before applying them to clinical data.
Cross-Standard Mapping (OMOP → FHIR)
project = portiere.init(
name="FHIR Export",
engine=PolarsEngine(),
task="cross_map",
source_standard="omop_cdm_v5.4",
target_model="fhir_r4",
)
Installation
Core Package
pip install portiere-health
Optional Extras
Install only what you need:
| Category | Extra | Command |
|---|---|---|
| Engines | Polars | pip install "portiere-health[polars]" |
| PySpark | pip install "portiere-health[spark]" |
|
| Pandas | pip install "portiere-health[pandas]" |
|
| LLM Providers | OpenAI | pip install "portiere-health[openai]" |
| Anthropic | pip install "portiere-health[anthropic]" |
|
| AWS Bedrock | pip install "portiere-health[bedrock]" |
|
| Ollama | pip install "portiere-health[ollama]" |
|
| Knowledge Backends | FAISS | pip install "portiere-health[faiss]" |
| Elasticsearch | pip install "portiere-health[elasticsearch]" |
|
| ChromaDB | pip install "portiere-health[chromadb]" |
|
| PGVector | pip install "portiere-health[pgvector]" |
|
| MongoDB | pip install "portiere-health[mongodb]" |
|
| Qdrant | pip install "portiere-health[qdrant]" |
|
| Milvus | pip install "portiere-health[milvus]" |
|
| Quality | Great Expectations | pip install "portiere-health[quality]" |
| FHIR | US Core validation + Bundle/NDJSON export | pip install "portiere-health[fhir]" |
| Everything | All extras | pip install "portiere-health[all]" |
Requirements: Python 3.10+
How It Works
Portiere implements a 5-stage AI pipeline for clinical data transformation:
Stage 1: Ingest & Profile
Connects to your data source (CSV, Parquet, databases) and extracts schema metadata — column names, types, cardinality, detected code columns, and PHI indicators.
Export an aggregated data-profile report (HTML + CSV) across one or more sources (v0.3.2):
portiere profile-report data/*.parquet -o outputs/ --format parquet --system HIS
# → outputs/data_profile.html, data_profile.csv, data_profile_summary.csv
The report shows per-source completeness bars and, per column, presence / missing / distinct / top value (share) / value-length / example. For the Excel deliverables teams actually exchange — a White-Rabbit-parity scan workbook, the per-table schema-mapping working document, and the OMOP source_to_concept_map workbook — see portiere workbook (docs/workbooks.md, xlsx extra) (v0.4.0). Empty and whitespace-only strings count as missing by default (--no-empty-as-missing to count only true nulls). Also available from Python via portiere.quality.build_source_profile + export_profile_report.
Stage 2: Schema Mapping
Maps source columns to target standard entities using a fusion of:
- Pattern matching — Regex patterns defined in YAML standard files
- Embedding similarity — SapBERT clinical embeddings for semantic matching
- Cross-encoder reranking — Precision reranking of top candidates
Stage 3: Concept Mapping
Maps clinical codes (ICD-10, CPT, local codes) to standard vocabularies (SNOMED CT, LOINC, RxNorm) through:
- Direct code lookup — Exact match in knowledge base
- Knowledge layer search — BM25s lexical / FAISS vector / Hybrid search
- Cross-encoder reranking — Rerank top-k candidates for precision
- LLM verification — Optional AI verification for medium-confidence mappings
- Confidence routing — Auto-accept (>0.95), needs-review (0.70–0.95), manual (<0.70)
Stage 4: ETL Generation
Generates standalone ETL scripts (Spark, Polars, or Pandas) and lookup tables (CSV) that run without the Portiere SDK — no vendor lock-in.
Stage 5: Validation
Post-ETL data quality checks using Great Expectations, with standards-aware conformance for all supported models (OMOP, FHIR, HL7, OpenEHR, custom YAML):
- Completeness — Non-null percentages for required fields
- Conformance — Type and constraint compliance derived from YAML field metadata
- Plausibility — Domain-specific clinical rules
Supported Standards
| Standard | Version | Use Case |
|---|---|---|
| OMOP CDM | v5.4 | Observational research, population health |
| FHIR R4 | R4 | Interoperability, health information exchange |
| HL7 v2 | 2.5.1 | Legacy hospital system integration |
| OpenEHR | 1.0.4 | European clinical data, archetype-based EHRs |
Standards are defined as YAML files and are fully extensible — you can define custom hospital CDMs or registry schemas.
Cross-Standard Mapping
Built-in crossmaps for transforming between standards:
| Source | Target | File |
|---|---|---|
| FHIR R4 | OMOP CDM | fhir_r4_to_omop.yaml |
| OMOP CDM | FHIR R4 | omop_to_fhir_r4.yaml |
| HL7 v2 | FHIR R4 | hl7v2_to_fhir_r4.yaml |
| OMOP CDM | OpenEHR | omop_to_openehr.yaml |
| FHIR R4 | OpenEHR | fhir_r4_to_openehr.yaml |
Custom Standards
Portiere is not limited to built-in standards. You can define any clinical data model — a hospital CDM, a disease registry schema, a research database, a legacy warehouse — as a YAML file and use it identically to built-in standards.
Define a Custom Standard (YAML)
Create a .yaml file with the following structure:
name: "hospital_cdm_v1"
version: "1.0"
standard_type: "relational"
organization: "General Hospital Research"
description: "Internal clinical data model for General Hospital"
entities:
patients:
description: "Core patient demographics"
fields:
patient_id:
type: integer
required: true
description: "Unique patient identifier"
ddl: "INTEGER PRIMARY KEY"
date_of_birth:
type: date
description: "Patient date of birth"
ddl: "DATE NOT NULL"
sex:
type: string
description: "Biological sex (M/F/U)"
ddl: "VARCHAR(1)"
# Fast pattern matching: source column name → target field
source_patterns:
patient_id: "patient_id"
subject_id: "patient_id"
dob: "date_of_birth"
birth_date: "date_of_birth"
gender: "sex"
sex: "sex"
# Embedding descriptions: optimized text for AI semantic matching
# Write what a clinician would search for, not just the field name
embedding_descriptions:
patient_id: "unique patient identifier number"
date_of_birth: "patient birth date birthday date of birth"
sex: "biological sex gender male female M F"
encounters:
description: "Hospital visits and admissions"
fields:
encounter_id:
type: integer
required: true
description: "Unique encounter identifier"
ddl: "INTEGER PRIMARY KEY"
admit_date:
type: datetime
description: "Admission date and time"
ddl: "TIMESTAMP NOT NULL"
encounter_type:
type: string
description: "Type of encounter (inpatient, outpatient, ED)"
ddl: "VARCHAR(20)"
source_patterns:
encounter_id: "encounter_id"
visit_id: "encounter_id"
hadm_id: "encounter_id"
admit_date: "admit_date"
admittime: "admit_date"
visit_type: "encounter_type"
embedding_descriptions:
encounter_id: "hospital encounter visit admission identifier"
admit_date: "admission date time when patient was admitted"
encounter_type: "visit type inpatient outpatient emergency department"
Use Your Custom Standard
import portiere
from portiere.engines import PolarsEngine
# Reference via "custom:" prefix — works anywhere target_model is accepted
project = portiere.init(
name="Hospital Migration",
engine=PolarsEngine(),
target_model="custom:/path/to/hospital_cdm_v1.yaml",
)
source = project.add_source("patients.csv")
schema_map = project.map_schema(source)
concept_map = project.map_concepts(codes=["E11.9", "I10"])
result = project.run_etl(
source,
output_dir="./output",
schema_mapping=schema_map,
concept_mapping=concept_map,
)
Or load directly for inspection:
from portiere.standards import YAMLTargetModel
model = YAMLTargetModel("/path/to/hospital_cdm_v1.yaml")
print(model.get_schema()) # entity → [fields]
print(model.get_source_patterns()) # source column hints
You can also ship your custom standard as a built-in by placing the YAML in src/portiere/standards/ — it will then be loadable by name:
model = YAMLTargetModel.from_name("hospital_cdm_v1")
Column Naming Guide
Portiere's schema mapper uses two strategies in sequence: exact pattern matching (fast, zero-cost) then embedding similarity (AI-powered). Understanding both helps you get higher auto-accept rates.
Strategy 1 — Source Patterns (rule-based, highest priority)
Each entity in a standard YAML defines source_patterns — a dictionary mapping source column names to target fields. Matches here are always accepted, regardless of confidence score.
Built-in OMOP patterns include common aliases:
| Your column name | Maps to |
|---|---|
patient_id, subject_id, mrn |
person.person_id |
dob, birth_date, date_of_birth |
person.birth_datetime |
gender, sex |
person.gender_concept_id |
icd_code, diagnosis_code, dx_code |
condition_occurrence.condition_source_value |
admit_date, admittime |
visit_occurrence.visit_start_date |
drug_code, ndc, medication_code |
drug_exposure.drug_source_value |
To maximize pattern EHR in your own standard, add all known aliases to source_patterns in your YAML:
source_patterns:
patient_id: "person_id" # exact name
pid: "person_id" # short alias
subject_id: "person_id" # research alias
pt_id: "person_id" # abbreviated
medical_record_number: "person_id" # verbose
Strategy 2 — Embedding Similarity (semantic, AI-powered)
When no pattern matches, the mapper encodes both the source column name and the embedding_descriptions into vectors using SapBERT, then finds the closest target field by cosine similarity.
What to write in embedding_descriptions:
Write natural-language phrases a clinician would use to describe what that column contains — not just a rephrasing of the field name.
# ❌ Too literal — just re-states the name
embedding_descriptions:
admit_date: "admission date"
dx_code: "diagnosis code"
# ✅ Rich synonyms and clinical context — maximizes semantic recall
embedding_descriptions:
admit_date: "hospital admission date time when patient was admitted inpatient start"
dx_code: "ICD diagnosis code ICD-10-CM ICD-9 disease condition clinical code"
Naming your source columns well also helps. The source column name itself is encoded alongside the description. Prefer descriptive names over cryptic abbreviations:
| Less matchable | More matchable |
|---|---|
col_32 |
diagnosis_code |
dt1 |
admission_date |
flg_act |
is_active |
cd_race |
race_code |
proc_nm |
procedure_name |
Confidence Tiers
After matching, every column receives a confidence score:
| Score | Tier | Action |
|---|---|---|
| ≥ 0.95 | Auto-accepted | Written to output immediately |
| 0.70 – 0.95 | Needs review | Flagged for human inspection |
| < 0.70 | Manual | Requires explicit override |
Tune these thresholds to match your project's risk tolerance:
from portiere import PortiereConfig, ThresholdsConfig
from portiere.config import SchemaMappingThresholds
config = PortiereConfig(
thresholds=ThresholdsConfig(
schema_mapping=SchemaMappingThresholds(
auto_accept=0.90, # lower → more auto-accepts
needs_review=0.60, # lower → fewer manual items
)
)
)
Full Workflow with Review
schema_map = project.map_schema(source)
# Inspect what needs review
for item in schema_map.needs_review():
print(f"{item.source_column} → {item.target_table}.{item.target_column} "
f"(confidence={item.confidence:.2f})")
for c in item.candidates[:3]:
print(f" candidate: {c['target_table']}.{c['target_column']} ({c['confidence']:.2f})")
# Approve, override, or reject
schema_map.approve("patient_name")
schema_map.override("pt_zip", target_table="location", target_column="zip")
schema_map.reject("internal_audit_flag")
# Approve all remaining items
schema_map.approve_all()
schema_map.finalize()
Knowledge Layer Backends
| Backend | Type | Dependencies | Best For |
|---|---|---|---|
| BM25s | Lexical | None (built-in) | Quick start, no infra needed |
| FAISS | Vector | faiss-cpu, sentence-transformers |
High-accuracy local search |
| Elasticsearch | Hybrid | elasticsearch |
Production deployments |
| ChromaDB | Vector | chromadb |
Lightweight vector store |
| PGVector | Vector | psycopg, pgvector |
PostgreSQL environments |
| MongoDB | Vector | pymongo |
Atlas Vector Search users |
| Qdrant | Vector | qdrant-client |
Dedicated vector DB |
| Milvus | Vector | pymilvus |
Large-scale vector search |
| Hybrid | Fusion | Varies | Combine backends with RRF |
Hybrid Search Example
from portiere import PortiereConfig, KnowledgeLayerConfig
config = PortiereConfig(
knowledge_layer=KnowledgeLayerConfig(
backend="hybrid",
hybrid_backends=["bm25s", "faiss"],
hybrid_fusion="rrf", # Reciprocal Rank Fusion
)
)
LLM Providers
Portiere supports Bring-Your-Own-LLM for concept verification:
| Provider | Extra | Model Examples |
|---|---|---|
| OpenAI | portiere[openai] |
GPT-4o, GPT-4o-mini |
| Anthropic | portiere[anthropic] |
Claude Sonnet, Claude Haiku |
| AWS Bedrock | portiere[bedrock] |
Claude, Titan, Llama |
| Ollama | portiere[ollama] |
Llama 3, Mistral, Gemma (local) |
from portiere import PortiereConfig, LLMConfig
config = PortiereConfig(
llm=LLMConfig(
provider="openai",
model="gpt-4o-mini",
api_key="sk-...",
)
)
Configuration
Portiere auto-discovers configuration from multiple sources (in priority order):
1. Python Objects
from portiere import PortiereConfig, EmbeddingConfig, KnowledgeLayerConfig
config = PortiereConfig(
target_model="omop_cdm_v5.4",
embedding=EmbeddingConfig(
provider="huggingface",
model="cambridgeltl/SapBERT-from-PubMedBERT-fulltext",
),
knowledge_layer=KnowledgeLayerConfig(backend="bm25s"),
)
2. YAML File (portiere.yaml)
target_model: omop_cdm_v5.4
storage: local
embedding:
provider: huggingface
model: cambridgeltl/SapBERT-from-PubMedBERT-fulltext
knowledge_layer:
backend: bm25s
llm:
provider: openai
model: gpt-4o-mini
thresholds:
auto_accept: 0.95
needs_review: 0.70
3. Environment Variables
export PORTIERE_TARGET_MODEL=omop_cdm_v5.4
export PORTIERE_LLM__PROVIDER=openai
export PORTIERE_LLM__API_KEY=sk-...
export PORTIERE_KNOWLEDGE_LAYER__BACKEND=faiss
Building the Knowledge Layer
Before concept mapping, build a searchable index from standard vocabularies (e.g., OHDSI Athena):
from portiere import build_knowledge_layer, PortiereConfig
config = PortiereConfig()
stats = build_knowledge_layer(
vocabulary_dir="./data/athena/",
config=config,
vocabularies=["SNOMED", "LOINC", "RxNorm", "ICD10CM"],
)
print(f"Indexed {stats['total_concepts']:,} concepts")
Documentation
| Resource | Description |
|---|---|
| Quick Start Guide | Get started in 5 minutes |
| API Reference | Full SDK API documentation |
| Configuration Guide | YAML, Python, and env var config |
| Knowledge Layer Guide | All 9 backends explained |
| LLM Integration | BYO-LLM setup |
| Pipeline Architecture | 5-stage pipeline deep dive |
| Multi-Standard Support | Standards and custom schemas |
| Cross-Standard Mapping | OMOP ↔ FHIR, HL7v2 → FHIR |
| Example Notebooks | 19 Jupyter notebooks with walkthroughs |
Project Structure
portiere/
├── src/portiere/
│ ├── __init__.py # Public API: init(), PortiereProject, configs
│ ├── config.py # Configuration with auto-discovery
│ ├── project.py # Unified project interface
│ ├── exceptions.py # Error hierarchy
│ ├── stages/ # 5-stage pipeline implementation
│ ├── engines/ # Compute engines (Polars, Spark, Pandas, DuckDB)
│ ├── knowledge/ # Knowledge layer backends (9 backends)
│ ├── embedding/ # Embedding providers & gateway
│ ├── llm/ # LLM providers & gateway
│ ├── local/ # Local AI components (schema mapper, concept mapper)
│ ├── artifacts/ # ETL code generation (Jinja2 templates)
│ ├── runner/ # ETL execution engine
│ ├── quality/ # Data quality validation (Great Expectations)
│ ├── standards/ # Clinical standard YAML definitions & crossmaps
│ ├── storage/ # Storage backends (local filesystem)
│ └── models/ # Pydantic data models
├── tests/ # 36 test modules, 689 tests
├── docs/
│ ├── documentations/ # 22 guides and references
│ └── notebooks_examples/ # 19 Jupyter notebook examples
├── pyproject.toml # Package configuration (hatchling)
└── LICENSE # Apache 2.0
Standards Coverage
v0.2.0 ships an opinionated subset of each standard — the required + commonly-used optional fields and entities, not every column from the spec. Long-tail fields are accepted contributions.
| Standard | Version | Entities / Resources | Notes |
|---|---|---|---|
| OMOP CDM | v5.4 | 19 (of ~37) | 14 clinical-data tables + 5 vocabulary tables (concept, vocabulary, concept_relationship, domain, concept_class) needed for FK validation |
| FHIR R4 | R4 (4.0.1) | 18 (of 145) | Patient, Practitioner, Organization, Location, Encounter, Condition, Observation, Specimen, Immunization, ServiceRequest, MedicationRequest, MedicationAdministration, MedicationDispense, Procedure, AllergyIntolerance, DiagnosticReport, Bundle, DocumentReference |
| HL7 v2 | 2.5.1 | Selected message types | Legacy hospital integration |
| OpenEHR | 1.0.4 | Selected archetypes | European EHRs |
For a programmatic listing: python -c "from portiere.standards import YAMLTargetModel; print(sorted(YAMLTargetModel.from_name('omop_cdm_v5.4').get_schema()))".
Limitations
Portiere is in active development. Current limitations (will be addressed in upcoming releases):
- Standards coverage is partial. OMOP CDM v5.4: 19 of ~37 tables — but the covered set includes the core clinical-event tables that carry most rows in a typical conversion (
person,visit_occurrence,condition_occurrence,drug_exposure,measurement,observation,procedure_occurrence). FHIR R4: 18 of 145 resources. Check the support matrix against your protocol; PRs to extend coverage are welcome. See how Portiere compares. - FHIR profile coverage is US Core + mCODE only. v0.3.0 validates against US Core 6.1.0 (10 resource types); v0.3.1 adds mCODE STU3 2.0.0 (5 oncology profiles). IPS and mCODE-extended coverage (treatments, additional staging) remain planned.
- Review interchange has limits. The UI supports schema and concept review. Concept JSON preserves candidates and provenance; CSV is a row-level projection. Frozen, portable schema/concept bundles remain planned. See the review guide.
- PHI scrubbing is opt-in and structural-first. v0.4.0 ships value-level detection/scrubbing (docs/phi-scrubbing.md): a built-in regex backend (email/phone/MRN/national-ID/date) plus optional Presidio NER (
[phi]extra) for person names. Off by default in v0.4.0 (scrub_phi=Trueto enable); planned default-on in v1.0. Unlabeled free-text names require the NER extra. - SNOMED CT and CPT-4 not bundled. Both have licensing constraints.
portiere quickstartoperates on bundled ICD-10-CM/LOINC/RxNorm only; for SNOMED, see vocabulary setup. - Replay reproduces stages best-effort.
portiere replay --auto-replayre-runs deterministic stages (ingest, validate) and records LLM-bound stages (schema, concept, ETL) asUNAVAILABLE. Full BYO-LLM rehydration and deterministic replay from frozen mappings remain planned. See reproducibility guide. - Benchmark coverage. The measured benchmark below covers ICD-10-CM → SNOMED. LOINC and RxNorm benchmark pairs remain planned.
Roadmap
- v0.3.2 (shipped): aggregated data-profile report export (
portiere profile-report, HTML + CSV) with engine profile enrichment. - v0.4.0 (shipped): offline configuration checks; value-level PHI scrubbing; compliance documentation; reranker evaluation and score blending; Docker and the docs site.
- v0.5.0 (shipped): MCP, LangChain and dbt integrations; domain-reranker decision harness.
- v0.6.0 (this release): source-scoped mappings; shared SDK/UI review decisions; revision-aware concept file exchange; complete code extraction; offline quickstart and packaging fixes.
- Next: explicit concept destination compilation, immutable mapping bundles, a unified run report, strict CI comparison and bounded resource use.
- Later: additional vocabulary benchmarks and CDMs, expanded FHIR profiles, clinical NLP, review-driven learning and wider deployment verification.
Each release tracks via GitHub Milestones; please open issues or PRs against the relevant milestone. See specs/ for the design docs behind each release.
Benchmarks
ICD-10-CM → SNOMED concept mapping, n=1,000 held-out, Athena 2026-04-30, against the OHDSI CONCEPT_RELATIONSHIP gold standard — including the head-to-head against OHDSI's own USAGI tool (v0.4.0, measured after the score-blending fix):
| Backend | top-1 | top-5 | top-10 | MRR |
|---|---|---|---|---|
| BM25 + cross-encoder reranker | 0.303 | 0.533 | 0.588 | 0.402 |
| Hybrid (BM25+FAISS via RRF) + reranker | 0.286 | 0.505 | 0.587 | 0.377 |
| SapBERT + FAISS + reranker | 0.285 | 0.462 | 0.546 | 0.363 |
| USAGI 1.4.3 (OHDSI baseline; top-1-only export) | 0.262 | — | — | — |
Every Portiere configuration beats the USAGI baseline (best top-1 0.303 vs 0.262, +16% relative). At default thresholds, 51.8% of codes route to auto-accept — half the workload never needs a reviewer. Honest history: v0.3.x published rows where "the AI lost to plain BM25"; v0.4.0 found and fixed the score-scale defect behind that result (the full ablation, including where each configuration is weak, is in the benchmark doc).
Reproduce any row: portiere benchmark athena-icd-snomed --backend bm25s|faiss|hybrid --athena-dir <path>.
Stratified sampling (opt-in): add --stratify-by domain to sample proportionally across Athena domains.
Methodology: docs/benchmarks/athena-icd-snomed.md.
FHIR Interoperability
v0.3.0 + v0.3.1 ship:
- US Core 6.1.0 profile validation —
Project.validate(fhir_profile="us-core-6.1.0")orportiere validate --fhir-profile us-core-6.1.0 --input resources.json. 10 resource types (Patient, Practitioner, Organization, Encounter, Condition, Observation, MedicationRequest, AllergyIntolerance, Procedure, DocumentReference). - mCODE STU3 2.0.0 profile validation (v0.3.1) —
--fhir-profile mcode-2.0.0. 5 core oncology profiles (CancerPatient, PrimaryCancerCondition, CancerDiseaseStatus, CancerStage, TNMStageGroup). Profile is keyed onmeta.profileURL substring (notresourceType). - Bundle + NDJSON export —
portiere export --format bundle --out out.jsonor--format ndjson --out out_dir/. Optional--fhir-profile us-core-6.1.0|mcode-2.0.0validates before writing.
Install the FHIR extra: pip install "portiere-health[fhir]".
See: docs/fhir-profile-validation.md, docs/fhir-bundle-export.md.
Mapping Review UI (v0.3.1)
Streamlit-based human-in-the-loop review of AI-generated schema mappings:
pip install "portiere-health[review]"
portiere review <project-dir> # opens http://127.0.0.1:8501
Reviewer actions: approve (accept AI suggestion), reject (mark unmappable), override (replace with table.column). Decisions persist to <project_dir>/schema_mappings/schema_mapping_reviewed.json next to the original — originals never modified. Local-only by default (no auth); --host 0.0.0.0 opt-in for LAN demos.
See: docs/mapping-review-ui.md. Concept-mapping review planned for v0.4.x.
Reproducibility (v0.3.1)
portiere replay --auto-replay manifest.lock.json
Re-runs each recorded stage and compares outputs to the manifest within tolerance bands. Stages whose dependencies are unavailable (LLM, knowledge layer) record as UNAVAILABLE and do not fail the report; full BYO-LLM rehydration is a v0.3.x roadmap item. See docs/reproducibility.md.
Star History
Contributing
We welcome contributions! Here's how to get started:
# Clone the repository
git clone https://github.com/Cuspal/portiere.git
cd portiere
# Create a virtual environment
python -m venv .venv
source .venv/bin/activate
# Install pinned contributor dependencies and the local source
python -m pip install -r requirements/dev.txt
python -m pip install --no-deps --no-build-isolation -e .
# Run tests
pytest
# Run linter
ruff check src/ tests/
# Run type checker
mypy src/portiere/
See CONTRIBUTING.md for the pinned contributor environment, Windows setup, optional model tests and release checks.
License
Portiere is licensed under the Apache License 2.0.
Copyright 2026 Cuspal Co. Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Citation
If you use Portiere in your research, please cite:
@software{portiere2026,
title = {Portiere: AI-Powered Clinical Data Mapping SDK},
author = {Cuspal Co.,Ltd.},
year = {2026},
url = {https://github.com/Cuspal/portiere},
license = {Apache-2.0},
}
Metadata
Release files for portiere-health 0.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| portiere_health-0.6.0.tar.gz | 562.3 kB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| portiere_health-0.6.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.2 MB
Release files / portiere_health-0.6.0.tar.gz
| Download URL | portiere_health-0.6.0.tar.gz |
|---|---|
| Size | 562.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ff72245e618bfce0c8f5acffd3ca8b6791e39cffa2ca4f32a0f2ad888a00c57b
|
|
BLAKE2b-256 checksum How to use checksums |
2ead9423df6dfef44717072ad1db727da8ffdaf650e0a7317011cad4c0e853b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 7, 2026.
Transparency logRelease files / portiere_health-0.6.0-py3-none-any.whl
| Download URL | portiere_health-0.6.0-py3-none-any.whl |
|---|---|
| Size | 653.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
25fa9145936eb19ae4fe9c0264bf3d440cbe566ae36943a6b4dfad96b5f2c92e
|
|
BLAKE2b-256 checksum How to use checksums |
41bb60bb9b062f6997f837daf6ac5ea8b4a869d67d0d3c76ee460a2829a26826
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 7, 2026.
Transparency logSecurity for portiere-health 0.6.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for portiere_health-0.6.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 7, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.