ncdb-tools 0.4.0

Tools for processing and analyzing National Cancer Database (NCDB) data

ncdb-tools

A Python package for working with National Cancer Database (NCDB) data. Convert NCDB fixed-width data files into optimized parquet datasets, apply standard cancer-specific transformations, and query data efficiently using Polars.

Features

• Data Ingestion -- Convert NCDB fixed-width .dat files to parquet format using SAS label definitions
• Automatic Transformations -- Age handling, site groupings, histology groupings, schema normalization
• Efficient Querying -- Fluent API to filter by primary site, histology, diagnosis year, and more
• Data Dictionary -- Auto-generate data dictionaries in CSV, JSON, and HTML with variable descriptions
• Memory Efficient -- Automatic memory detection, lazy evaluation, streaming support
• Type Safe -- Full type hints, input validation with NCDBValidationError

Installation

pip install ncdb-tools

Data Access

This package does not include NCDB data. You must obtain access through official channels:

• NCDB Participant Sites: Contact your institution's cancer registry
• Research Access: Apply through the American College of Surgeons
• Data Use Agreements: Required for all NCDB data usage

Quick Start

Build a Dataset

import ncdb_tools

result = ncdb_tools.build_parquet_dataset(
    data_dir="/path/to/ncdb/dat/files",
    generate_dictionary=True,
)
print(f"Dataset: {result['parquet_dir']}")

Query Data

import ncdb_tools

# Fluent filtering API
df = (ncdb_tools.load_data("/path/to/parquet/dataset")
      .filter_by_primary_site(["C509", "C500"])   # Breast
      .filter_by_histology([8500, 8140])           # Ductal, Adenocarcinoma
      .filter_by_year([2018, 2019, 2020])
      .collect())

# Explore before collecting
query = ncdb_tools.load_data("/path/to/parquet/dataset")
print(query)            # NCDBQuery(rows=2500000, cols=338, path=...)
print(query.count())    # 2500000
print(query.columns)    # ['PUF_CASE_ID', 'AGE', ...]

sample = query.sample(n=1000)

Query API

Filter Methods

All filter methods return self for chaining:

query = (ncdb_tools.load_data(path)
         .filter_by_primary_site(["C509"])
         .filter_by_histology([8500])
         .filter_by_year([2020, 2021])
         .drop_missing_vital_status())

Method	Description
filter_by_primary_site(sites)	Filter by ICD-O-3 primary site codes
filter_by_histology(codes)	Filter by histology codes
filter_by_year(years)	Filter by diagnosis year
drop_missing_vital_status()	Remove cases with missing vital status
filter_active_variables()	Keep only variables with data in most recent year
select_demographics()	Select common demographic variables
select_outcomes()	Select common outcome variables

Utility Methods

query.columns        # list of column names
query.count()        # row count without collecting
query.sample(n=100)  # random sample as DataFrame
query.describe()     # {"total_rows": ..., "columns": ..., "parquet_path": ...}
query.collect()      # execute and return DataFrame
query.lazy_frame     # access the underlying Polars LazyFrame

Polars Integration

Any Polars LazyFrame method can be called directly on the query object:

import polars as pl

result = (ncdb_tools.load_data(path)
          .filter_by_primary_site(["C509"])
          .select(["PUF_CASE_ID", "AGE", "PRIMARY_SITE", "YEAR_OF_DIAGNOSIS"])
          .filter(pl.col("AGE") > 50)
          .group_by("PRIMARY_SITE")
          .agg(pl.count())
          .collect())

Standard Transformations

build_parquet_dataset() automatically applies:

1. Data Type Conversion -- Numeric columns detected and converted; clinical codes preserved as strings
2. Age Processing -- Original AGE preserved; AGE_AS_INT (numeric, 90 for "90+") and AGE_IS_90_PLUS flag added
3. Site Groupings -- SITE_GROUP derived from primary site codes (Breast, Lung, Colon, etc.)
4. Histology Groupings -- HISTOLOGY_GROUP for major cancer types (Adenocarcinoma, Squamous, Melanoma, etc.)
5. Schema Normalization -- Consistent data types across all year files

Cancer-Specific Filtering

Primary Site

# ICD-O-3 topography codes
breast = query.filter_by_primary_site(["C509", "C500", "C501"])
lung   = query.filter_by_primary_site(["C340", "C341", "C349"])
colon  = query.filter_by_primary_site(["C180", "C182", "C187"])

Histology

# ICD-O-3 morphology codes
adenocarcinoma = query.filter_by_histology([8140, 8141, 8142])
squamous       = query.filter_by_histology([8070, 8071, 8072])
melanoma       = query.filter_by_histology([8720, 8721, 8722])

Treatment Analysis

import polars as pl

treatment = (ncdb_tools.load_data(path)
    .filter_by_primary_site(["C509"])
    .lazy_frame
    .group_by("RX_SUMM_SURG_PRIM_SITE")
    .agg([
        pl.count().alias("n"),
        pl.col("DX_DEIDENTIFY_DAYS").mean().alias("mean_days"),
    ])
    .sort("n", descending=True)
    .collect())

Data Dictionary

Generated dictionaries include column names, data types, variable descriptions (from SAS labels), missing data counts, and summary statistics.

from ncdb_tools import DataDictionaryGenerator

gen = DataDictionaryGenerator()
gen.generate_from_parquet(
    parquet_dir="/path/to/dataset",
    output_file="data_dictionary.html",
    format="html",
)

Available formats: CSV, JSON, HTML (with color coding and interactive features).

Configuration

# Environment variables
export NCDB_DATA_DIR="/path/to/ncdb/data"
export NCDB_OUTPUT_DIR="/path/to/output"
export NCDB_MEMORY_LIMIT="8GB"

Or use a .env file (install with pip install ncdb-tools[config]).

Memory & Performance

# Automatic memory detection
mem = ncdb_tools.get_memory_info()
print(f"Available: {mem['available']}")

# Explore before committing to full collection
info = query.describe()
print(f"{info['total_rows']:,} rows, {info['columns']} columns")

# Sample for exploration
sample = query.sample(n=10000)

Requirements

• Python >= 3.10
• NCDB participant user files (PUF) in fixed-width .dat format
• SAS labels file (optional, for variable descriptions in data dictionary)

License

MIT

Disclaimer

This package is not affiliated with or endorsed by the American College of Surgeons or the National Cancer Database. Users must obtain data through official channels and comply with all applicable data use agreements.