iatro-clintrace 0.0.1.dev2026071214

ClinTrace clinical-course compression, grounding, and longitudinal decision modeling

Iatro ClinTrace

EN | 中文

ClinTrace is a longitudinal clinical-course model for HCC notes. Its job is to turn free-text clinical notes into a reusable course representation that can support treatment-pathway reasoning, evidence-grounded decision completion, and downstream model integration.

Research use only. ClinTrace is not a medical device and must not be used for clinical diagnosis, treatment selection, triage, or patient management.

The authoritative design record is docs/clintrace_total_design.md. This README is the open-source entrypoint: it describes the package boundary, public command surface, and controlled-asset contract.

Table of Contents

• Scientific Contract
• Components
• Release Boundary
• Installation
• Current Command Surface
• Training Workflow
  • 1. Build Source IAC Assets
  • 2. Build or Import Compressed Text Assets
  • 3. Build and Check Paired IAC Assets
  • 4. Train, Select, and Export the Compressor
  • 5. Build Frozen Note Features
  • 6. Build Decision Supervision
  • 7. Build Direct-Token and Decoder Planes
  • 8. Build History-State and Decision-Contrast Assets
  • 9. Train the ClinTrace Decision Expert
  • 10. Evaluate and Freeze Release Candidates
• Configuration
• Package Layout
• Open-Source Hygiene
• Optional LLM Client Utilities

Scientific Contract

ClinTrace learns from clinical notes, not from manually supplied decision-chain labels at inference time.

input
  frozen compressor note representations
  observable note title/type text
  natural longitudinal order

supervision and audit
  LLM-extracted E/P/X/O chains
  timeline-derived labels
  history-state summaries

output
  course expert records
  grounded evidence spans
  ranked open-world decision candidates

E/P/X/O chains are supervision and evaluation assets. They are not deployment inputs. Timeline-derived fields such as document role, label type, pathway labels, and decision-bearing flags are also not model inputs in the formal expert, because they are not guaranteed to exist when a user sends a new note.

ClinTrace also treats document inclusion and admission ordering as an upstream asset contract. Users decide which notes belong in the training/release asset before building IAC packs. The framework does not apply private note-type rules to filter longitudinal history. Within the same admission, users must provide a stable clinical order through admission/episode metadata, timestamps, and stable document keys; ClinTrace consumes that order rather than correcting ambiguous hospital-system timestamps.

The central modeling target is:

history state + observed clinical evidence -> plausible P/X/O decision candidates

This is intentionally open-world. Missing labels in the record are treated as unobserved, not as clinical negatives. The model is evaluated primarily by whether documented decisions are ranked highly and whether generated evidence and decision spans are clinically inspectable.

Components

iatro.clintrace.compressor
  Single-note compression into fixed latent note tokens.

iatro.clintrace.routing
  Compactness router used before tokenization to choose compression vs direct
  token representation.

iatro.clintrace.grounding
  Full-note EXPO grounding: evidence, plan, executed action, and outcome span
  supervision from extracted decision chains.

iatro.clintrace.completion
  Observed-evidence decision completion within a note.

iatro.clintrace.longitudinal
  History-state-conditioned decision completion and decision-contrast diagnostics.

iatro.clintrace.timeline
  Patient-level chronological note indexing.

iatro.clintrace.inspect
  Local IAC asset browsers for development and audit.

Release Boundary

This GitHub repository contains source code, configuration, prompt contracts, tests, and documentation only. Clinical data, extracted chains, generated caches, model weights, evaluation predictions, API credentials, and training outputs are not part of the GitHub repository or the default PyPI package.

Model and auxiliary binary assets are produced by the controlled full training/validation pipeline for a separate gated Hugging Face release. They must not be committed to GitHub, including through Git LFS.

Tracked GitHub content:

src/                             Python package
configs/                         reproducible training/evaluation configs
prompts/                         versioned LLM prompt contracts
docs/                            design and method notes
tests/                           unit tests
README.md, LICENSE, pyproject.toml

Expected local asset classes:

data/00_source/                  raw clinical_text IAC packs
data/01_compressed/              compressed note tables keyed by doc_id
data/02_paired/                  source/compressed paired IAC packs
data/03_timeline_index/          longitudinal note indexes
data/04_clintrace_features/      frozen compressor feature caches
data/05_decision_chains/         extracted decision-chain JSONL
data/06_decision_supervision/    compiled EXPO supervision assets
data/07_grounding_features/
                                 tokenizer/direct-note feature caches
data/08_patient_state_history/   history-state text summaries
runs/                            ignored training/evaluation run outputs
artifacts/compressor/            selected local compressor artifacts
artifacts/clintrace/             selected local ClinTrace planes and auxiliaries
artifacts/hf/clintrace/          gated Hugging Face release tree
dist/                            ignored release/build output directory

Legacy local caches may still use older non-contiguous directory numbers. The public workflow uses the contiguous layout above.

04_clintrace_features is the frozen compressor feature substrate. It is reused when EXPO supervision is refreshed. 07_grounding_features is a downstream cache built from 04_clintrace_features plus 06_decision_supervision; rebuild it whenever 05_decision_chains or 06_decision_supervision changes. 08_patient_state_history is a separate longitudinal history-state asset and is not regenerated just because EXPO chains are refreshed.

data/, artifacts/, runs/, result/ legacy outputs, and dist/ are ignored by git. Public examples should use de-identified toy data or externally releasable fixtures only.

Gated Hugging Face artifacts are generated only as part of the complete controlled training and validation chain. Do not create or commit ad hoc local asset bundles from intermediate results.

Installation

Default installation is intentionally lightweight. It supports IAC packaging, inspection, and clintrace llmc data-preparation clients without installing the model training stack.

pip install iatro-clintrace

Install extras only for the workflows that need them:

.[inference]  local model loading / future Hugging Face artifact inference
.[train]      training, cache generation, and evaluation
.[server]     optional vLLM-compatible local serving
.[serve]      alias for .[server]
.[dev]        tests, build, and packaging checks

Development checkout:

pip install -e .
pip install -e ".[train,server,dev]"

Current Command Surface

The public CLI currently exposes three stable classes of work:

clintrace build      local data-asset construction
clintrace inspect    local IAC inspection
clintrace llmc       OpenAI-compatible LLM client wrappers for data preparation

clintrace train and clintrace evaluate are retained to reproduce the research training chain. They are not the primary interface for downstream users once release artifacts are published.

End-user model inference is intentionally not exposed yet. The inference entry point will be added after the gated Hugging Face artifact layout is frozen and validated. Until then, the PyPI package should be treated as the reproducible data-preparation and training codebase, not as a deployable clinical decision tool.

Training Workflow

The section below documents the full reproduction path used to rebuild release artifacts. It is not the normal runtime path for future users. All commands read and write ignored local assets under data/, runs/, and artifacts/; those outputs are never committed to GitHub.

1. Build Source IAC Assets

Start from a de-identified document table in JSONL, CSV, or TSV. Required columns are:

patient_id      stable de-identified patient identifier
doc_id          stable de-identified document identifier, unique within dataset
text            clinical note text

Optional columns are:

doc_type        observable note title/type
doc_timestamp   sortable clinical timestamp string
episode_index   integer visit/admission index, -1 if unknown
encounter       optional encounter identifier
doc_category    optional source category, defaults to clinical_document
source          optional source-system label

Before packing, finalize the source asset scope. Remove documents that should not be part of longitudinal history or supervision, and normalize admission or encounter boundaries. For same-admission records, provide timestamps and stable document identifiers that encode the intended clinical order. The source packer performs deterministic sorting; it does not infer clinical sequence from damaged timestamps or apply built-in note-type filtering.

Pack the table into a clinical_text IAC:

clintrace build source-iac \
  --input data/source_documents/demo_train.jsonl \
  --output data/00_source/demo_train.iac \
  --institution demo_train \
  --overwrite

The source pack builder sorts documents within each patient by episode_index, doc_timestamp, and doc_id. This is a deterministic serialization rule over user-supplied metadata, not a clinical reordering algorithm. It skips rows with missing required fields, duplicate doc_id, or empty text. It reports:

input_rows=<input table rows>
patients=<packed patient count>
docs=<packed document count>
duplicate_doc_ids=<skipped duplicate document ids>
empty_text=<skipped empty documents>
missing_required=<missing required-field counts>
text_raw_mb=<UTF-8 text payload size>
output=<source IAC path> size_mb=<written pack size>

Verify the source pack:

clintrace build source-iac \
  --input data/source_documents/demo_train.jsonl \
  --output data/00_source/demo_train.iac \
  --institution demo_train \
  --verify

Verification must report:

verify_docs == verify_expected_docs
verify_unique_doc_ids == verify_expected_docs
verify_mismatches == 0

The resulting asset is:

data/00_source/{site}.iac
  clinical_text pack containing original de-identified notes.

2. Build or Import Compressed Text Assets

Create or import compressed note text before compressor training. This asset is a normal JSONL, CSV, or TSV table, not an IAC pack. Required columns are:

doc_id             same de-identified document identifier as source IAC
compressed_text    compressed clinical note text

Accepted aliases for compressed_text are text and output. Optional columns are doc_type, doc_timestamp, and episode_index; when omitted, the source IAC metadata is used.

data/01_compressed/{site}.jsonl
  compressed-note table keyed by doc_id.

The compression method is outside this public contract. It may be produced by a human-reviewed pipeline, a local model, or an external system; ClinTrace only requires aligned doc_id values and clinically readable compressed text. If a dataset requires de-identification or identifier alignment, complete that data preparation before building ClinTrace pairs. The open-source package does not define or distribute private identifier-mapping protocols. For the optional LLM client wrapper that uses the active packaged profile prompt and writes a paired-iac-ready JSONL, see Optional LLM Client Utilities.

3. Build and Check Paired IAC Assets

Once the source IAC and compressed table share de-identified document identifiers, build the source/compressed training pairs:

data/02_paired/{site}.iac
  source/compressed clinical_text_pair pack used by the compressor and router.

Build a paired source/compressed pack:

clintrace build paired-iac \
  --source-iac data/00_source/demo_train.iac \
  --compressed-input data/01_compressed/demo_train.jsonl \
  --output data/02_paired/demo_train.iac \
  --institution demo_train \
  --overwrite

paired-iac is the only command that writes compressed text into an IAC asset. There is no standalone compressed IAC in the public workflow. If a dataset needs de-identification or identifier mapping to make source and compressed IDs align, complete that preparation before paired-iac. The paired IAC itself is source text paired with compressed text; it is not a de-identified/non-de-identified pair.

Record the following IAC-level acceptance metrics before model training:

patients=<paired patient count>
paired_docs=<paired document count>
source_docs=<source document count>
compressed_rows=<compressed input table rows>
compressed_docs=<unique usable compressed document count>
duplicate_doc_ids=<skipped duplicate compressed document ids>
missing_compressed=<source docs without compressed counterpart>
empty_source=<skipped empty source docs>
empty_compressed=<skipped empty compressed docs>
missing_required=<missing required-field counts>
source_raw_mb=<UTF-8 source text payload size>
compressed_raw_mb=<UTF-8 compressed text payload size>
output=<paired IAC path> size_mb=<written pack size>

Verify the paired pack:

clintrace build paired-iac \
  --source-iac data/00_source/demo_train.iac \
  --compressed-input data/01_compressed/demo_train.jsonl \
  --output data/02_paired/demo_train.iac \
  --institution demo_train \
  --verify

Verification must report:

verify_docs == verify_expected_docs
verify_unique_doc_ids == verify_expected_docs
verify_mismatches == 0

Then inspect both the raw pack and paired pack before starting a long run:

clintrace inspect iac data/00_source/demo_train.iac --head 5
clintrace inspect pairs -i demo_train --head 5

The IAC acceptance gate is: all expected in-scope documents are paired exactly once, source/compressed text round-trips without mismatch, admission boundaries and within-admission order are already resolved by the user-provided metadata, and sampled records are clinically readable after de-identification.

4. Train, Select, and Export the Compressor

The compressor maps a clinical note to fixed note tokens. Train it from the paired source/compressed IAC:

clintrace train compressor --config configs/compressor/train.yaml

Training writes checkpoints and metrics under runs/ or the configured compressor output directory. Select the checkpoint by the predeclared validation rule, then export the release-sized compressor artifact into the ignored artifacts/compressor/ tree:

clintrace build compressor-artifact \
  --checkpoint runs/compressor/checkpoints/course_encoder_step5500.pt \
  --output artifacts/compressor/clintrace_compressor_qwen35_2b_n32_d768_step5500.pt

Run a single-note sanity check on the exported artifact:

clintrace compress \
  --checkpoint artifacts/compressor/clintrace_compressor_qwen35_2b_n32_d768_step5500.pt \
  --input tmp/source_note.txt \
  --output tmp/compressed_note.txt

If a validated compressor artifact already exists, start from this sanity check and continue with feature generation.

5. Build Frozen Note Features

Create longitudinal indexes and frozen compressor note-token caches. These caches are the model input substrate for downstream decision training.

The timeline index records document order and audit metadata. It does not decide which documents enter longitudinal history. Downstream history is consumed from fixed assets such as history_document_keys or admission-state rows.

clintrace build timeline
clintrace build compressor-features --checkpoint artifacts/compressor/clintrace_compressor_qwen35_2b_n32_d768_step5500.pt

6. Build Decision Supervision

Compile the extracted EXPO chains into supervision assets. The chain labels and spans are used only as training/evaluation targets, not as inference inputs. History context in the compiled supervision is written explicitly as history_document_keys; training consumes those fixed keys instead of reconstructing history with note-type filters.

clintrace build grounding-supervision \
  --chains data/05_decision_chains/results.jsonl \
  --timeline-dir data/03_timeline_index \
  --features-dir data/04_clintrace_features \
  --output-dir data/06_decision_supervision \
  --train-institutions demo_train \
  --external-institutions demo_external \
  --overwrite

The institution names above are placeholders. Use the local de-identified train and external-site identifiers from your own data/03_timeline_index/*.jsonl manifests. If no institution split is supplied, the first discovered institution is treated as train and the rest as external.

Within each train institution, patients are assigned deterministically to 75% train, 15% validation and 10% test. The assignment keeps patients intact while balancing document-level EXPO bundle signatures, primary supervision roles, chain-count bins and document roles; inspect the generated report.json before training.

7. Build Direct-Token and Decoder Planes

Build the direct-token feature plane used for compact or already-compressed text, then cache per-note grounding features.

clintrace build decoder-plane
clintrace build grounding-features \
  --documents data/06_decision_supervision/document_samples.jsonl \
  --chains data/05_decision_chains/results.jsonl \
  --checkpoint artifacts/compressor/clintrace_compressor_qwen35_2b_n32_d768_step5500.pt \
  --output-dir data/07_grounding_features \
  --overwrite

grounding-features does not read configs/decision/default.yaml; pass its document table, EXPO chain table, compressor checkpoint, and output directory explicitly.

8. Build History-State and Decision-Contrast Assets

History-state summaries provide longitudinal patient context for admission-level training. Decision-contrast pairs are conservative EXPO bundle comparisons used to reduce clinically implausible near-neighbor confusion.

clintrace llmc history --help
clintrace build decision-contrast \
  --samples data/06_decision_supervision/samples.jsonl \
  --output-dir data/06_decision_supervision/expo_distance_analysis

9. Train the ClinTrace Decision Expert

The decision expert is trained in three ordered components. The component names describe the capability being learned, not separate deployable experts.

clintrace train grounding \
  --config configs/decision/default.yaml \
  --overwrite

clintrace train completion \
  --config configs/decision/default.yaml \
  --initialization grounding_init \
  --overwrite

clintrace train longitudinal \
  --config configs/decision/default.yaml \
  --initialization completion_init \
  --eval-splits validation test external \
  --overwrite

All three training entry points accept the same runtime override flags: --output-dir, --device, --batch-size, --token-budget, --num-workers, --max-epochs, and --overwrite. Completion additionally accepts --grounding-checkpoint; longitudinal accepts both --grounding-checkpoint and --completion-checkpoint.

grounding learns full-note EXPO span grounding. completion learns observed-evidence to P/X/O decision completion. longitudinal adds H1 history state conditioning while keeping EXPO grounding and completion active.

Formal training defaults to data.feature_loading: preload for the >=32 GB training target. Use data.feature_loading: mmap only as the lower-memory alternative: workers retain first-touch shards as read-only memory mappings, while each batch coalesces direct/history shard reads and prefetches subsequent complete batches without LRU tensor caches or dense-bank copies.

grounding writes a content-light training sample order plan to data.sample_order_plan. The plan contains sample identifiers only, not tensors or clinical text. completion and longitudinal reuse that sample order when available, then repack batches under their own token-budget rules so the three phases share the same auditable order without forcing identical batch shapes.

When only EXPO chains are refreshed after the compressor note features have already been fixed, keep data/00_source through data/04_clintrace_features and the existing data/08_patient_state_history, then rebuild:

data/05_decision_chains
data/06_decision_supervision
data/07_grounding_features
runs/clintrace_grounding*
runs/clintrace_completion*
runs/clintrace_longitudinal*

10. Evaluate and Freeze Release Candidates

Evaluate every selected checkpoint on validation, internal test, and external splits. The gated release candidate is chosen from the complete validation chain, not from an ad hoc artifact bundle.

clintrace evaluate grounding --split validation
clintrace evaluate completion --checkpoint runs/clintrace_completion_grounding_init/clintrace_decision_expert.pt --split test
clintrace evaluate longitudinal --checkpoint runs/clintrace_longitudinal_completion_init/clintrace_longitudinal.pt --split test
clintrace evaluate decision-contrast --help

After the complete validation chain is frozen, release assets are staged under artifacts/hf/clintrace/. That directory is the local Hugging Face gated-release root and remains ignored by git.

Configuration

Configs are grouped by component:

configs/compressor/              compressor training and local smoke configs
configs/decision/default.yaml     grounding/completion/longitudinal defaults

Package Layout

src/iatro/clintrace/
  compressor/
  routing/
  grounding/
  completion/
  longitudinal/
  timeline/
  inspect/

Top-level scripts/ is intentionally empty for the public package. Stable functionality belongs in the package and is exposed through clintrace.

Open-Source Hygiene

Do not commit:

clinical data
LLM extraction outputs containing protected text
model checkpoints or generated feature caches
runs folders
local credentials or API keys
Hugging Face gated release assets
throwaway intermediate files

Institution-specific names are avoided in public code and documentation. Local asset aliases should use neutral identifiers such as demo_train and demo_external.

Optional LLM Client Utilities

clintrace llmc commands are wrappers around OpenAI-compatible chat endpoints. They are client-side data-preparation utilities, not ClinTrace build steps. The three frozen Chinese prompts, the zh_cn profile, and the EXPO label vocabulary are bundled inside the installed package under iatro.clintrace.assets; --profile, --prompt, and --label-vocab are optional overrides for controlled reruns. The default profile is zh_cn; non-Chinese deployments should add a separate profile asset rather than editing core code.

clintrace llmc compress    profile compressor prompt -> compressed-note JSONL
clintrace llmc expo        profile EXPO prompt + label_vocab.json -> EXPO chain JSONL
clintrace llmc history     profile history-state prompt -> H1 history-state JSONL

llmc compress reads a source IAC, applies the versioned compression prompt, and appends compressed-note rows to a JSONL file that can be consumed directly by clintrace build paired-iac --compressed-input.

The output JSONL contains at least:

doc_id
patient_id
institution
doc_type
doc_timestamp
episode_index
compressed_text
usage

Default execution targets a local OpenAI-compatible server so clinical text stays on the host.

export OPENAI_API_BASE=http://127.0.0.1:8000/v1
export OPENAI_API_KEY=local
export OPENAI_MODEL=qwen36

clintrace llmc compress \
  --institution demo_train \
  --source-iac data/00_source/demo_train.iac \
  --out data/01_compressed/demo_train.jsonl \
  --model qwen36 \
  --workers 16 \
  --thinking disabled

Useful preparation checks:

clintrace llmc compress \
  --institution demo_train \
  --source-iac data/00_source/demo_train.iac \
  --out data/01_compressed/demo_train.jsonl \
  --dry-run

clintrace build paired-iac \
  --source-iac data/00_source/demo_train.iac \
  --compressed-input data/01_compressed/demo_train.jsonl \
  --output data/02_paired/demo_train.iac \
  --institution demo_train \
  --overwrite

Rows are resumable by doc_id: rerunning the wrapper skips documents that already have a non-empty compressed_text in the output file. Non-local API endpoints require --allow-external-endpoint; only use that with an approved clinical-data processing endpoint.

EXPO and history-state preparation use the same endpoint policy:

clintrace llmc expo \
  --input data/01_compressed/demo_train.jsonl \
  --out data/05_decision_chains/results.jsonl \
  --model qwen36 \
  --workers 16

clintrace llmc history \
  --institution demo_train \
  --compressed-input data/01_compressed/demo_train.jsonl \
  --timeline-dir data/03_timeline_index \
  --output data/08_patient_state_history/demo_train.jsonl \
  --model qwen36 \
  --workers 16

llmc expo defaults to a high output ceiling and thinking enabled so the default extraction contract favors quality over speed. On local servers with adequate context length, override only when intentionally running a cost/speed probe:

clintrace llmc expo \
  --input data/01_compressed/demo_train.jsonl \
  --out data/05_decision_chains/results.jsonl \
  --model qwen36 \
  --workers 196 \
  --max-tokens 32768 \
  --thinking enabled