raxtik-etl ⚡

High-performance canonical pharmacovigilance (PV) data engine and packaging toolkit powered by DuckDB and Polars.

raxtik-etl provides the canonical schema definitions, deduplication algorithms, case resolution logic, and self-describing dataset packaging for safety surveillance datasets. It is completely decoupled from source-specific ingestion logic via an extensible Source Adapter Protocol.

🏗️ Architecture

                    SOURCE CONNECTORS (Plugins)
                               │
               ┌───────────────┼───────────────┐
               ▼               ▼               ▼
      raxtik-source-faers  future-source   future-source
      (FDA AERS/FAERS)    (EudraVigilance)   (VigiBase)
               │
               │ implements SourceAdapter
               ▼
   ┌────────────────────────────────────────────────────────┐
   │                       raxtik-etl                       │
   │  1. Canonical Schemas (demo, drug, reac, outc, etc.)   │
   │  2. Deterministic Case Resolution & Deduplication      │
   │  3. Standardized Transformations & Unit Normalization  │
   │  4. Quality Control & Quality Summary Reporting        │
   │  5. Self-describing Packaging (DuckDB & Parquet)       │
   └───────────────────────────┬────────────────────────────┘
                               │
                               ▼
   ┌────────────────────────────────────────────────────────┐
   │                   Packaged Artifacts                   │
   │  - data/packages/faers_{release}.duckdb (Indexed SQL)  │
   │  - data/packages/faers_{release}.parquet/ (ZSTD Shards)│
   └───────────────────────────┬────────────────────────────┘
                               │
                               ▼
   ┌────────────────────────────────────────────────────────┐
   │                 Downstream Ecosystem                   │
   │  - raxtik-parser   (Synthetic narratives, clinical ML) │
   │  - raxtik-api      (Case search, signal detection)     │
   │  - raxtik-studio   (Interactive safety workbench)      │
   └────────────────────────────────────────────────────────┘

Decoupled Source Plugins

raxtik-etl contains zero source-specific proprietary parsing, column aliases, or archive file logic. Connectors such as raxtik-source-faers implement the SourceAdapter protocol and register via Python entry points (raxtik_etl.sources).

🏛️ Canonical Data Model

Every source connector projects raw records into 7 canonical fact tables:

Authoritative Case View (cases)

In addition to raw submissions, raxtik-etl compiles a deduplicated, authoritative view:

• cases: Evaluates CASE / CASEID across multiple submission updates (ISR / PRIMARYID), selecting the latest authoritative report version (FDA_DT, CASEVERSION) and flagging expedited 15-day cases and serious adverse outcomes (has_serious_outcome, is_death).

🚀 Quickstart

1. Installation

Requires Python >= 3.11 (Python 3.13 recommended) and uv.

# Set local Python version
pyenv local 3.13.5

# Create virtual environment and install raxtik-etl
uv venv --python 3.13.5 .venv
source .venv/bin/activate
uv pip install -e ".[dev]"

# Install the FDA FAERS source connector plugin
uv pip install -e ../raxtik-source-faers

2. Build a Quarterly Package via CLI

Generate a standalone DuckDB database and ZSTD-compressed Parquet directory from a raw quarterly extract:

raxtik-etl build-package \
    --source faers \
    --release 2004Q1 \
    --zip-path data/raw/archives_cache/2004q1_ascii.zip \
    --output-dir data/packages

Output:

• data/packages/faers_2004Q1.duckdb (Indexed analytical database)
• data/packages/faers_2004Q1.parquet/ (ZSTD-compressed columnar tables)

3. Inspect Package Metadata & Quality Metrics

raxtik-etl inspect-package --release 2004Q1 --package-dir data/packages

Displays package version, signature ID, fact record counts, and canonical data quality indicators.

💻 Python SDK Usage

Downstream applications can query packages using the high-performance FAERS client with zero-copy Polars DataFrames:

from raxtik_etl import FAERS

# Connect to the 2004Q1 compiled package
with FAERS(release="2004Q1", package_dir="data/packages") as faers:
    # 1. Print self-describing metadata & data quality report
    faers.print_summary()

    # 2. Retrieve authoritative resolved cases
    cases_df = faers.cases(limit=1000)
    print(
        cases_df.select(
            [
                "case_id",
                "primary_id",
                "canonical_case_version",
                "standard_age_years",
                "standard_sex_label",
                "has_serious_outcome",
            ]
        )
    )

    # 3. Query primary suspect drugs and reactions
    drugs_df = faers.drugs(primary_suspect_only=True)
    reactions_df = faers.reactions()

🔌 Implementing a Custom Source Adapter

Adding a new pharmacovigilance source (e.g., EudraVigilance, PMDA, or clinical trial safety feeds) requires zero changes to raxtik-etl:

1. Implement raxtik_etl.sources.protocol.SourceAdapter:

from raxtik_etl.sources.protocol import SourceAdapter, SourceArtifact, SourceLoadResult
import duckdb


class MyCustomSourceAdapter:
    source_name: str = "my_pv_source"

    def extract_and_load(
        self, con: duckdb.DuckDBPyConnection, artifact: SourceArtifact
    ) -> SourceLoadResult:
        # 1. Extract raw files from artifact.path
        # 2. Project raw records into canonical staging tables:
        #    demo, drug, reac, outc, indi, ther, rpsr
        # 3. Return SourceLoadResult with row counts and provenance metadata
        ...

2. Register the entry point in your package's pyproject.toml:

[project.entry-points."raxtik_etl.sources"]
my_pv_source = "my_package.adapter:MyCustomSourceAdapter"

3. raxtik-etl will automatically discover and load the adapter:

raxtik-etl build-package --source my_pv_source --release 2024Q1 --zip-path /path/to/archive.zip

🧪 Testing & Code Quality

Run tests across the canonical engine:

# Run pytest test suite (25 unit and audit tests)
pytest tests/ -v

# Run linting and formatting
ruff check .
ruff format .

📚 Documentation & Guides

• 2004Q1 Dataset Integration & Usage Guide: Downstream synthetic generation recipes and data consumption.
• FDA FAERS Official Data Dictionary: Field references across S1, S2, and S3 eras.
• Historical Schema Evolution Audit: Longitudinal schema analysis from 2004Q1 to 2026Q2.

📄 License

Licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).