hpcs 0.7.0

CPU-optimized statistical computing library with SIMD vectorization and OpenMP parallelization for massive datasets (10M+ records)

HPCSeries Core

High-Performance Statistical Computing for Large-Scale Data Analysis

---

Overview

HPCSeries Core is a CPU-optimized statistical computing library for massive datasets (10M+ records). Provides 2-100x speedup over NumPy/Pandas through SIMD vectorization (AVX2/AVX-512/NEON), OpenMP parallelization, and cache-optimized algorithms.

Built with Fortran, C, and C++ for maximum performance, with zero-copy Python bindings via Cython.

Key Features

• SIMD-Accelerated Operations: sum, mean, std, min, max, median, MAD, quantile
• Fast Rolling Windows: 50-100x faster than Pandas for rolling operations
• Anomaly Detection: Statistical and robust outlier detection
• Axis/Masked Operations: Efficient 2D array and missing data handling
• Auto-Tuning: One-time calibration for optimal hardware performance
• Architecture-Aware: Automatic optimization for x86 (Intel/AMD) and ARM (Graviton)

Performance Highlights

Operation	Array Size	NumPy/Pandas	HPCSeries	Speedup
sum	1M	0.45 ms	0.12 ms	3.8x
rolling_mean	100K (w=50)	45 ms	0.8 ms	56x
rolling_median	100K (w=50)	850 ms	7.2 ms	118x

Target use cases: 10M-1B records, time-series analysis, sensor data, financial analytics.

---

Installation

Quick Install

pip install hpcs

Verify:

import hpcs
print(hpcs.__version__)  # 0.7.0

Build from Source

git clone https://github.com/hpcseries/HPCSeriesCore.git
cd HPCSeriesCore

# Build C/Fortran library
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
cd ..

# Install Python bindings
pip install -e .

Requirements: Python 3.8+, NumPy 1.20+, GCC/gfortran 7+, CMake 3.18+

See Build Guide for details.

---

Quick Start

import hpcs
import numpy as np

x = np.random.randn(1_000_000)

# Reductions (2-5x faster than NumPy)
hpcs.sum(x), hpcs.mean(x), hpcs.std(x)

# Rolling operations (50-100x faster than Pandas)
rolling_median = hpcs.rolling_median(x, window=100)

# Anomaly detection
anomalies = hpcs.detect_anomalies_robust(x, threshold=3.0)

# Auto-tuning (run once)
hpcs.calibrate()
hpcs.save_calibration_config()

---

Performance Configuration

Optimal OpenMP Settings

export OMP_DYNAMIC=false
export OMP_PROC_BIND=true
export OMP_PLACES=cores
export OMP_NUM_THREADS=2

Why 2 threads? Empirical testing on AMD EPYC Genoa, Intel Ice Lake, and ARM Graviton3 shows HPCSeries Core saturates memory bandwidth at 2 threads. Using 4+ threads degrades performance by 5-18% due to cache contention.

See Performance Methodology for full analysis.

Additional Tips

• Ensure C-contiguous arrays: np.ascontiguousarray(x)
• Use robust functions (median, robust_zscore) for data with outliers
• Run calibration once: hpcs.calibrate() and hpcs.save_calibration_config()

---

Documentation

Core Documentation

• Performance Methodology - Empirical benchmarks and thread scaling
• AWS Deployment Guide - Production deployment on EC2
• Calibration Guide - Performance auto-tuning
• NUMA Affinity Guide - Multi-socket optimization
• Build & Test Guide - Compilation and testing

Examples & Tutorials

• Jupyter Notebooks - 12 comprehensive tutorials covering:
  • Getting started and basic usage
  • Rolling operations and anomaly detection
  • Climate data, IoT sensors, financial analytics
  • NumPy/Pandas migration guide
  • Kaggle competition examples

See Notebooks README for full list.

---

Version History

v0.7.0 (Current - 2025-12-17)

• Architecture-aware compilation (x86 and ARM)
• AWS deployment infrastructure
• Comprehensive performance validation
• Thread scaling optimization (OMP_NUM_THREADS=2 universal)

See CHANGELOG.md for complete history.

---

Project Structure

HPCSeriesCore/
├── src/                      # C/Fortran/C++ source
│   ├── fortran/              # HPC kernels (OpenMP)
│   └── hpcs_*.c              # SIMD implementations
├── include/                  # C API headers
├── python/hpcs/              # Python bindings (Cython)
├── cmake/                    # CMake modules (architecture detection)
├── notebooks/                # Jupyter tutorials
├── docs/                     # Documentation
├── tests/                    # Test suites
└── bench/                    # Benchmarks

---

Support

• Bug Reports: GitHub Issues
• Discussions: GitHub Discussions
• Changelog: CHANGELOG.md

---

License

MIT License - See LICENSE for details.

---

Citation

If you use HPCSeries Core in your research, please cite:

@software{hpcseries_core_2025,
  title = {HPCSeries Core: High-Performance Statistical Computing for Large-Scale Data Analysis},
  author = {HPCSeries Core Contributors},
  year = {2025},
  month = {12},
  version = {0.7.0},
  url = {https://github.com/hpcseries/HPCSeriesCore},
  license = {MIT}
}

Or use GitHub's "Cite this repository" button (auto-generated from CITATION.cff).

---

⭐ Star us on GitHub if HPCSeries Core accelerates your data analysis!