corpusscope 0.1.1

Profile a clinical EMR database into a portable, schema-validated corpus profile: exact tiktoken token counts (full-record vs clinical-content), per-stream scope metrics, one worked patient record, and a schema-validated YAML/JSON deliverable.

corpusscope

Profile a clinical EMR database into a portable, schema-validated corpus profile.

Point corpusscope at your SQL database. It reads through the whole thing, counts tokens exactly with tiktoken, measures the scope of every clinical data stream, captures one representative patient in full, and hands you two files:

• profile.yaml — the profile in a human-readable form you can open and read.
• profile.json — the same content, machine-readable and validated against the corpus schema bundled with the tool.

That pair is the whole deliverable. It describes what your corpus contains and how big it is — enough to size, plan, and reason about the dataset. corpusscope reads and measures only; it leaves the database untouched.

---

Table of contents

• What problem it solves
• What it produces
• How it works — the four steps
• Install
• Using corpusscope on the command line
  • The three commands
  • Step-by-step walkthrough
  • Command & option reference
  • Exit codes
• Using corpusscope from Python
• The mapping file — full reference
• The token model (full vs clinical)
• Quality gates
• Read-only
• Requirements & supported databases
• License

---

What problem it solves

Every clinical EMR database is laid out differently — different table names, different column names, labs stored one way here and another way there, some data streams present and some absent. Before you can plan anything with such a dataset, you need an honest, precise description of it: how many patients and encounters, how many years it spans, which clinical streams exist, how the diagnoses are coded, and — crucially — how large it is in tokens, the unit that actually governs the cost and feasibility of any downstream language-model work.

corpusscope produces exactly that description, in a single standard shape, from any SQL schema. You teach it your schema once (a small, hand-editable mapping file); after that it does the counting, the measuring, and the self-checking on its own.

---

What it produces

The output has three parts. All three appear in both profile.yaml and profile.json.

1. corpus — identity block

The dataset's identity: name, provider, country, source_system, source_database, and a contact. You fill these in at the top of the mapping file; corpusscope copies them through verbatim. They describe whose data this is and where it came from — the tool cannot infer them.

2. Scope — the whole dataset measured

Twelve scope sections, A1–A12, each computed as SQL aggregates across the entire database (not a sample):

#	Section	What it measures
A1	scale	Patients, encounters, source rows, linked tables — plus the exact token counts (see token model).
A2	stream_inventory	For each of the 17 canonical clinical streams: whether it's present, and how many source rows it holds. Streams you don't hold are recorded as absent.
A3	record_depth	Fields populated per encounter, visits per patient (mean & median), and the split of documentation across consultation / treatment / investigations.
A4	longitudinal	First and last encounter date, number of years covered, and encounters (and new patients) per year.
A5	geography	Distinct facilities, regions covered, and the per-region share of activity.
A6	demographics_scope	Gender split, mean age, age-parse rate, and the age-band distribution.
A7	diagnoses_scope	Coding system, coded-record count, distinct codes, ICD-10 shape-match rate, paired free-text rate, a breakdown by ICD-10 chapter, and the top conditions.
A8	laboratory_scope	Distinct analytes, result and order counts, how often units and reference ranges travel with a result, and the top analytes.
A9	vitals_scope	Triage row count and per-vital coverage (temperature, blood pressure, pulse, weight, height, BMI, …).
A10	examination_scope	Regions in the exam grid, total region cells, and the normal / abnormal / not-examined split.
A11	medications_scope	Prescription lines, distinct items, and how completely frequency / route / duration are recorded.
A12	specialties_scope	Number of distinct clinical specialties.

Sections for streams your dataset doesn't hold come back empty or present: false, so a dataset that lacks (say) radiology or vitals profiles just as cleanly as one that has them.

3. One worked patient

A single real patient assembled into the full nested record shape — demographics, then encounters, each with its notes, diagnoses, labs, prescriptions, and so on. This shows the shape of a record end-to-end, so the scope numbers above have a concrete example to stand next to. corpusscope picks the first patient that has a real encounter, so the example is representative rather than a stub.

---

How it works — the four steps

   ┌─────────────┐   1 autodetect    ┌──────────────┐   2 review / edit
   │  your SQL   │ ────────────────▶ │ mapping.yaml │ ◀───────────────  you
   │  database   │                   └──────┬───────┘
   └──────┬──────┘                          │
          │            3 profile            │
          └───────────────┬─────────────────┘
                          ▼
                 ┌──────────────────┐   4 QA + schema-validate
                 │   corpusscope    │ ─────────────────────────▶  profile.yaml
                 │  (exact tokens + │                             profile.json
                 │  scope + patient)│                             (ready to send)
                 └──────────────────┘

1. autodetect — corpusscope reflects your live schema and writes a proposed mapping: its best guess at which physical table and columns feed each canonical stream.
2. review — you open that one file and confirm it. Fix any column it guessed wrong, and mark streams you don't have as absent. This is the only manual step, and it's hand-editable YAML.
3. profile — corpusscope reads the whole database. It makes two exact passes: a token pass (one patient at a time, so even a billion-token corpus never loads more than one record into memory) and a pass of SQL aggregates for the scope sections.
4. QA + validate — every run checks the finished profile against the bundled schema and a set of quality gates, then writes the two files. A run that fails a gate stops with a non-zero exit instead of writing.

---

Install

pip install corpusscope

Then confirm it's on your PATH:

corpusscope --version

Everything it needs installs with it — the database drivers (SQLAlchemy + pyodbc), the exact tokeniser (tiktoken), the YAML writer, and the schema validator. There is nothing else to install and no companion tool to run. Python 3.9+.

To install from a source checkout instead of the published package:

pip install .            # or:  pip install -e ".[dev]"   (editable + test deps)

---

Using corpusscope on the command line

Installing the package puts one command on your PATH: corpusscope. It has three sub-commands, run in order. You only ever touch two things: a connection URL (which database to read) and a mapping file (how your schema maps to the canonical streams).

The three commands

Command	What it does	You run it
corpusscope autodetect	Inspects your live schema and writes a proposed mapping.yaml.	Once, to get a starting point.
corpusscope profile	Reads the whole database and writes profile.yaml + profile.json.	Every time you want a profile.
corpusscope validate	Checks an existing profile.json against the schema.	Any time, e.g. before sending.

Run corpusscope --help (or corpusscope <command> --help) to see the same options listed below, and corpusscope --version to print the version.

Step-by-step walkthrough

The connection URL is a standard SQLAlchemy database URL. Two common shapes:

# Microsoft Fabric / Azure SQL analytics endpoint:
export DB_URL="mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"

# A local SQLite file (handy for a trial):
export DB_URL="sqlite:///./mydata.db"

Step 1 — propose a mapping. Point autodetect at your database. It reflects the schema and guesses which table/columns feed each stream:

corpusscope autodetect --source "$DB_URL" --out mapping.yaml

Proposed mapping -> mapping.yaml
REVIEW IT before profiling: confirm each stream's table/columns and which streams are present:false.

If your patient / encounter key columns aren't named patient_id / encounter_id, tell autodetect: --patient-id pat_no --encounter-id visit_no. If your tables live under a named schema, add --schema dbo.

Step 2 — review the mapping (the one manual step). Open mapping.yaml and confirm it against your real schema: fix any column it guessed wrong, fill in the corpus: identity block at the top, and mark any stream you don't have as present: false. See The mapping file for every option and two worked examples.

Step 3 — profile the database. This does the full read — the exact token pass plus the scope aggregates — then runs QA and writes the two files:

corpusscope profile --source "$DB_URL" --mapping mapping.yaml \
    --out-yaml profile.yaml --out-json profile.json

Profiling (exact token pass + scope aggregates)…

QA: 0 error(s), 0 warning(s)

  patients   : 48,213
  tokens     : 412,556,190 full  |  210,004,731 clinical (50.9%)  [tiktoken o200k_base]
  wrote -> profile.yaml
  wrote -> profile.json

If QA finds an error, the run stops and writes nothing:

QA: 1 error(s), 0 warning(s)
  ERROR   [tokens] clinical_content_tokens (…) > total_tokens (…)

QA FAILED — no profile written.

So a run either writes a clean, schema-valid profile or stops without writing. (Both --out-* flags are optional; omit them to do a dry run that prints QA and the headline numbers but writes nothing.)

Step 4 — hand off (and optionally re-check). The two files are the deliverable. You can re-validate the JSON on its own at any time — no database needed:

corpusscope validate --json profile.json

valid against the corpus schema.

Command & option reference

corpusscope autodetect — propose a mapping from a live database.

Option	Required	Meaning
--source <url>	yes	SQLAlchemy connection URL of the source database.
--out <path>	yes	Where to write the proposed mapping YAML.
--schema <name>	no	DB schema/namespace (e.g. dbo) if your tables live under one.
--patient-id <col>	no	Patient key column name (default patient_id).
--encounter-id <col>	no	Encounter key column name (default encounter_id).

corpusscope profile — build the profile from a mapped database.

Option	Required	Meaning
--source <url>	yes	SQLAlchemy connection URL of the source database.
--mapping <path>	yes	Your reviewed mapping YAML.
--out-yaml <path>	no	Write the human-readable YAML here.
--out-json <path>	no	Write the schema-valid JSON here.
--schema <name>	no	DB schema/namespace (e.g. dbo).

corpusscope validate — check a profile JSON against the schema.

Option	Required	Meaning
--json <path>	yes	Profile JSON to validate.

Exit codes

Every command returns 0 on success and a non-zero code on failure, so it drops straight into a script or CI pipeline:

Command	0 (success)	non-zero (failure)
autodetect	mapping written	connection/reflection error
profile	QA passed; files written	any QA error — nothing written
validate	JSON is valid	JSON is invalid (errors printed)

Using corpusscope from Python

Everything the CLI does is available as a library — the same four moves in code:

import corpusscope as cs

# 1. connect (read-only) and load your reviewed mapping
db = cs.Db(cs.connect("<sqlalchemy-url>"))
mapping = cs.Mapping.from_yaml("mapping.yaml")

# 2. build the profile — exact token pass + scope + one worked patient
profile = cs.build_profile(db, mapping)

# 3. run the QA gates and stop on any error (same gate the CLI enforces)
issues = cs.run_qa(profile)
assert not [i for i in issues if i.level == "error"], issues

# 4. write the deliverable
cs.write_yaml(profile, "profile.yaml")
cs.write_json(profile, "profile.json")

build_profile returns the profile as a plain Python dict, so you can inspect any number before writing it — e.g. profile["scale"]["total_tokens"] or profile["scale"]["clinical_content_pct"].

---

The mapping file — full reference

Your tables and columns won't match the canonical names corpusscope reports in, so a small mapping file bridges the two. autodetect writes a first draft; you review it. This is the tool's single point of configuration, and it's plain, auditable YAML.

A mapping has four top-level parts:

corpus:                       # identity — copied verbatim into the profile
  name: Example Clinical Corpus
  provider: Example Health
  country: Colombia
  source_system: Example EMR
  source_database: SQL Server 2019
  contact: { name: Jane Doe, email: [e-mail], role: Data lead }

schema:                       # DB schema/namespace, e.g. dbo — leave blank if none

keys:                         # the columns that link rows to a patient / encounter
  patient_id: patient_id
  encounter_id: encounter_id

streams:                      # one entry per canonical stream (see below)
  ...

Per-stream fields

Each stream tells corpusscope where its data physically lives:

streams:
  demographics:
    table: tbl_patient
    columns: { age_years: age_years, gender: sex, home_region: home_region }

  encounters:
    table: tbl_encounter
    date_column: encounter_start                 # drives the longitudinal metrics
    columns: { facility_id: care_center_code, specialty_id: specialty_code, visit_type: care_setting }

  diagnoses:
    table: tbl_encounter                         # two streams may share one table
    columns: { icd10_code: admission_diagnosis_code, diagnosis_name: admission_diagnosis }

  lab_results:                                   # analytes stored as columns (wide)
    table: tbl_lab
    layout: wide
    analyte_columns: [hemoglobin, hba1c, creatinine, total_cholesterol, hdl, ldl]

  prescriptions:
    table: tbl_medication
    columns: { generic_name: medication_name, dose: dose, route: admin_route }

  # a stream you don't hold:
  immunizations: { present: false }

Every knob a stream can carry:

Field	Meaning
table	The physical table this stream reads from. Two streams may point at the same table (e.g. encounters and diagnoses); the tool de-duplicates so a shared table's storage is never counted twice.
present: false	You don't hold this stream. Recorded as absent in the profile.
columns	Map of canonical_field: physical_column. Only the fields you have.
patient_id_column / encounter_id_column	Override the link columns from keys when this table names them differently (e.g. notes that link by admission_id).
date_column	The date/datetime column for time-based metrics (used for encounters longitudinal coverage).
layout	For lab_results: long (one row per analyte) or wide (one column per analyte). Both are supported.
analyte_columns	For a wide lab layout: the list of analyte columns to count.
where	An optional raw SQL filter applied uniformly to every metric over this stream (e.g. is_annulled = 0 to exclude voided records).
clinical_extra	Extra free-text columns whose values are clinical content but have no canonical field (e.g. result_interpretation, medical_indications). Counted into the clinical-content tokens.
value_maps	Per-field value coding. Most important for gender, where single-letter codes conflict across datasets — m is male in one, mujer/female in another. Declaring the coding makes the buckets correct.

The value_maps gender example, showing why it matters:

demographics:
  table: pacientes
  columns: { gender: sex }
  value_maps:
    gender:                 # in THIS dataset m = mujer (female), h = hombre (male)
      female: [m, mujer, f]
      male:   [h, hombre]
      other:  [i, unknown]

The 17 canonical streams

demographics, encounters, triage_vitals, history_notes, physical_exam, region_findings, impression_notes, diagnoses, lab_requests, lab_results, radiology, prescriptions, pharmacy_requests, procedures, immunizations, allergies, referrals.

Map the streams you have; mark the rest present: false.

Worked examples

The examples/ directory holds two complete, ready-to-edit mappings that show the same tool adapting to very different schema shapes:

• examples/mapping.meridian.yaml — a raw operational tbl_* schema, with labs stored wide (one column per analyte).
• examples/mapping.anonimizado.yaml — an anonymized delivery schema (Spanish table names, banded ages, year-only dates, labs stored long), showing value_maps, clinical_extra, and a where filter in use.

Start from autodetect, then use whichever example is closest as a reference.

---

The token model (full vs clinical)

Tokens are the headline number. Every patient record is measured on two content axes and by two encoders.

Two content axes:

	what it counts
full-record	Every stored field, serialized — values and labels, ids, flags, timestamps, JSON structure. This is the storage / ingestion cost of the record.
clinical-content	Only the mapped clinical values — diagnoses, results, medications, narrative, vitals, findings. This is the medical signal a model would actually learn from.

The split between them (e.g. "51% clinical content") tells you how much of the raw size is real signal versus structural overhead. Which fields count as clinical content is defined declaratively and auditably — one list per record section — not buried in the counter.

Two encoders: both axes are counted with o200k_base (the primary, reported number) and cl100k_base (an independent second count) — a built-in cross-check on the total.

Per-patient distribution: alongside the totals you get min / max, the p50 / p90 / p99 percentiles, and a 12-bin histogram (<1k, 1k-3k, … 5M+) of tokens per patient — so you can see not just the total but how it's spread.

The counting is streaming: corpusscope tokenises one patient at a time and keeps only running totals, so an exact count over an arbitrarily large corpus never loads more than a single record into memory.

---

Quality gates

Every run checks the finished profile before writing it, against these gates:

• the JSON validates against the bundled corpus schema;
• clinical tokens ≤ full tokens, and structure = full − clinical exactly;
• the distribution bins sum to the patient count; percentiles are monotonic (p50 ≤ p90 ≤ p99);
• gender, age-band, exam-outcome, and stream-split shares each sum to ~100%;
• no negative counts anywhere;
• the worked patient is present and non-empty.

An error stops the run (non-zero exit, nothing written); a warning is reported for review but doesn't block. The package also ships an end-to-end test that profiles a synthetic database and checks the numbers against known answers.

---

Read-only

corpusscope reads and measures only. Every SQL statement it runs is a SELECT — no inserts, updates, or deletes — and its entire output is the two profile files.

---

Requirements & supported databases

• Python 3.9+
• Connects via SQLAlchemy, so it works against any database SQLAlchemy + pyodbc can reach — Microsoft Fabric / Azure SQL analytics endpoints, SQL Server, PostgreSQL, and SQLite (handy for a local trial). The scope queries are written to be dialect-aware across these engines.
• All Python dependencies (SQLAlchemy, pyodbc, tiktoken, PyYAML, jsonschema) install automatically with the package.

---

License

Proprietary — © 2026 Meridian Intelligence. All rights reserved. Not open source. This software is provided to the counterparty under a limited, non-transferable license for use solely within the scope of the parties' engagement/agreement, and may not be used, copied, distributed, modified, or exploited beyond that Purpose. See LICENSE for the full terms.