batch-analytics 0.3.1

PySpark batch analytics: Extract, Transform, Stage, and analytical modules (linear regression, correlation, PCA, t-test).

Batch Analytics

PySpark-based analytics pipeline for ClickHouse data: Extract → Transform → Stage → Analytics. Designed to run as the main application inside a Spark driver container (invoked by analytics_runners via SparkApplication CRD).

Bundle contents

Only the files required for the batch analytics job runner:

analytics/
├── pyproject.toml
├── requirements.txt          # core + scipy + boto3 + clickhouse-connect (single-file install)
├── requirements-batch.txt  # includes requirements.txt
├── README.md
└── src/
    └── batch_analytics/
        ├── __init__.py
        ├── __main__.py        # python -m batch_analytics
        ├── job_runner.py      # Entry point
        ├── config.py
        ├── extract.py
        ├── transform.py
        ├── log.py
        ├── README.md
        └── analytics/
            ├── __init__.py
            ├── linear_regression.py
            ├── correlation.py
            ├── pca_clustering.py
            └── t_test.py

Install

pip install -e .
# or install every runtime dependency used anywhere in the package, then editable:
pip install -r requirements.txt && pip install -e .
# PyPI wheel pulls numpy automatically (pyspark.ml); extras: ttest, s3, clickhouse, output, full
pip install "batch-analytics[full]"

Run

# Via module
python -m batch_analytics

# Via CLI (after pip install -e .)
batch-analytics

# Full pipeline
batch-analytics

# Analytics only (from staged ClickHouse table)
batch-analytics --from-stage --modules lr corr pca ttest

Configuration

See src/batch_analytics/README.md for environment variables and usage.

Docker image

For Spark on Kubernetes, build an image that includes this package and exposes job_runner.py at the path used by mainApplicationFile (e.g. local:///opt/analytics/job_runner.py).