cwru-plus 0.3.0

An enterprise-grade, ultra-fast pipeline for downloading, filtering, and windowing the CWRU bearing dataset.

Markdown

🚀 CWRU-Plus: The Ultimate Bearing Dataset Pipeline

An enterprise-grade, ultra-fast, and highly customizable Python library for the Case Western Reserve University (CWRU) Bearing Dataset.

---

🌟 Why CWRU-Plus?

Stop wasting days writing loops to parse messy .mat files! CWRU-Plus is built for modern Machine Learning and Deep Learning researchers who need clean, ML-ready tensors instantly.

Whether you are looking for pure simplicity, or you are conducting complex Domain Adaptation research requiring channel-specific signal processing, CWRU-Plus handles it with unprecedented speed.

🔥 Key Superpowers

• 🪄 Magic in 3 Lines: Go from raw files to fully aligned PyTorch/TensorFlow-ready 3D tensors (Samples, Channels, Sequence_Length) in just three lines of code.
• ⚡ Blazing Fast Extraction: Thanks to zero-copy memory mapping (sliding_window_view) and NumPy vectorization, compiling overlapping windows for the entire dataset takes less than 3 seconds.
• 🎛️ Parallel Custom Filtering: Need to apply a 4th-order Butterworth filter to the Drive End (DE) and a median filter to the Fan End (FE)? Our parallel engine processes the entire dataset in under 1 minute.
• ☁️ Cloud Native (Colab & Drive): Built-in daemon threads automatically back up your massive datasets to Google Drive in the background without freezing your Colab runtime.
• 🔓 100% Free & Open-Source: Fully customizable under the MIT License.

---

🚀 Quick Start: The "3-Line" Promise

Forget complex data engineering. Once installed, getting your ML tensors is this simple:

import cwru

# ==============================================================================
# ⚠️ STEP 1: ONE-TIME DATA SETUP (Download & Ingest)
# Run these two lines ONLY ONCE to build your persistent local database.
# After 'dataset.npz' is generated, you can comment out or remove Step 1 entirely.
# ==============================================================================
cwru.download(CWRUfs=12, download_path="data/raw")
cwru.ingest(base_path="data/raw", output_file_path="data/dataset")

# ==============================================================================
# 🚀 STEP 2: FAST PRODUCTION LOADING (Run repeatedly from here)
# Once you have the permanent .npz file, you ONLY need to execute from this line onwards.
# Ingested arrays are loaded instantly into your pipeline in under 3 seconds!
# ==============================================================================
(X, Y), metadata = cwru.load(npz_path="data/dataset.npz", window_size=2048, step_size=512)

print(f"Data Shape: {X.shape} | Labels Shape: {Y.shape}")

---

📂 Offline / Local Directory Import (local_download)

---

If you have already downloaded the .mat files manually (e.g., 105.mat, 112.mat) and they are dumped into a single folder, you don't need to re-download them. CWRU-Plus can scan your local directory, filter out the specific files based on the requested sampling frequency, and safely copy them to a target directory with our standardized, ML-ready naming convention.

Your original files are never altered or deleted during this process.

from cwru import offline_download

# Safely extract and rename ONLY the 48kHz files from your messy downloads folder
success = offline_download(
    source_path="data/cwru_raw",
    target_path="data/CWRU_Standard",
    CWRUfs=48,             # Explicitly target 48kHz dataset links
    replace_files=False    # Do not overwrite if standard file already exists
)

---

⚙️ Installation

You can install CWRU-Plus directly from PyPI via pip (which automatically manages all dependencies like numpy, scipy, requests, and tqdm):

pip install cwru-plus

🛠️ For Developers & Contributors If you want to modify the source code, clone the repository and install it in editable mode:

git clone https://github.com/Razani-Ali/cwru-plus.git
cd cwru-plus
pip install -e .

---

🎛️ Advanced: Channel-Specific Signal Filtering

CWRU-Plus offers Inversion of Control (IoC), allowing you to inject completely different linear or non-linear DSP filters into different sensor channels.

Using our multi-threaded backend, this heavy computation finishes in seconds:

import scipy.signal as sig
import cwru

# Define your custom filters
def de_filter(signal):
    b, a = sig.butter(4, 0.1, btype='low')
    return sig.filtfilt(b, a, signal) # Zero-phase filtering

def fe_filter(signal):
    return sig.medfilt(signal, kernel_size=5) # Non-linear impulsive noise removal

# Map filters to specific channels
my_filters = {
    "DE": de_filter,
    "FE": fe_filter,
    "BA": None # Keep Base Accelerometer raw
}

# Apply across the entire dataset in parallel
cwru.filter(filters=my_filters, input_npz="data/dataset.npz", output_npz="data/filtered.npz")

---

🔬 Advanced: Domain Adaptation & Cross-Domain Splits

If you are conducting Domain Generalization or Domain Adaptation research (e.g., training on specific loads and testing on others), CWRU-Plus makes it incredibly easy.

The cwru.load() function returns perfectly aligned metadata vectors (Severities, HorsePowers, Locations). You can use standard NumPy masking to instantly split your dataset into distinct domains:

import cwru

# Load the entire dataset
(X, Y), (Severities, HorsePowers, Locations, sensors) = cwru.load("data/dataset.npz")

# Define your source and target domains based on HorsePower (HP) loads
source_mask = (HorsePowers <= 2) # Train on 0, 1, and 2 HP
target_mask = (HorsePowers == 3) # Test strictly on 3 HP

# Split the tensors using NumPy masking
X_train, Y_train = X[source_mask], Y[source_mask]
X_test, Y_test = X[target_mask], Y[target_mask]

print(f"Source Domain (0-2 HP) Shape: {X_train.shape}")
print(f"Target Domain (3 HP) Shape: {X_test.shape}")

---

🔬 Leakage-Free Stratified Data Splitting

In rotating machinery fault diagnosis, splitting data windows randomly causes severe Data Leakage. Overlapping windows from the same .mat file end up in both training and testing sets, leading to artificially high accuracies (overfitting) that fail on unseen machines.

The models will optimize for the specific temporal signature of that exact test run rather than learning generic fault features. For robust evaluation, we highly recommend setting num_parts=1 and utilizing our File-Level Stratified Splitting to completely isolate unseen physical experiments.

cwru-plus solves this by introducing a Stratified Group Splitter. It ensures that windows from the same physical file stay together (either 100% in Train, Val, or Test), while strictly maintaining the class distribution frequency across all splits.

Complete ML-Ready Demo (Fancy Indexing)

import cwru

# 1. Load samples along with their original File IDs
(X, Y, file_ids), metadata = cwru.load(
    npz_path="data/CWRU_48k.npz", 
    window_size=2048, 
    step_size=512
)

# 2. Generate perfect, leakage-free indices grouped by physical files
(tr_d, val_d, te_d), (tr_idx, _, _) = cwru.stratified_file_split(
    X=X,
    y=Y, 
    file_ids=file_ids, 
    train_ratio=0.8, 
    val_ratio=0.1, 
    random_seed=42)

# 3. Extract Arrays From Tuple
train_x, train_y = tr_d
val_x, val_y = val_d
test_x, test_y = te_d

# 4. If You Need to Split Meta_Data too
train_HP = metadata[1][tr_idx]

⚠️ A Warning on Temporal Splitting (num_parts) and Non-Stationarity

Some frameworks attempt to avoid file-level leakage by splitting a single long signal into temporal segments over time (using the num_parts argument in module cwru.load). However, due to the non-stationary nature of real-world vibration signals (caused by slight motor load fluctuations, temperature shifts, and transient slips during the experiment), splitting a continuous signal across time still introduces severe distribution leakage between the Train and Validation sets.

🚨 Crucial Notice on CWRU/MAFAULDA Low-File Regime & Skewed Splits

When using Dataset.stratified_file_split(), you might notice deviations in the horizontal split ratios (e.g., the Normal class dominating the Test set while dropped in Train).

This is NOT a bug in the code; it is an inherent physical limitation of the CWRU dataset:

1. Extreme Low-File Counts: Most fault classes (e.g. 'IR007') contain only 4 physical .mat files in total.
2. Imbalancy: The Normal baseline files contain significantly longer continuous signals (yielding thousand more windows) than the damaged bearing files.

Since our algorithm strictly enforces Zero Data Leakage by keeping windows of the same file together, moving one massive Normal file to a small test partition creates a major statistical displacement.

How to handle this in your Research Paper:

• Avoid standard Accuracy: Due to the unavoidable test set imbalance, always evaluate your neural networks using Macro F1-Score or Balanced Accuracy.
• Opt for a 2-way Split: For perfectly proportional distributions, set val_ratio=0.0 to activate the clean 2-way split (allocating 3 files to Train and 1 to Test per class).

---

🎯 Built-in Few-Shot & Meta-Learning Sampler

Modern fault diagnosis research heavily relies on episodic training configurations like MAML or Prototypical Networks. To bridge the gap between raw data and deep learning architectures, CWRU-Plus ships with a high-performance FewShotSampler.

Instead of writing manual indexing loops, you can construct fully synchronized episodic tasks instantly. The sampler automatically manages operational metadata (Severity, HP, Location) and returns clean, integer-mapped arrays explicitly optimized for loss functions like PyTorch's CrossEntropyLoss.

from cwru.sampler import build_few_shot_sampler

# Initialize the sampler over your pre-windowed dataset
sampler = build_few_shot_sampler(
    X_base=X_data, Y_base=Y_labels,
    numeric_to_string={0: 'Normal', 1: 'Inner_Fault'},
    meta_base=(Severity, HP, Location),
    seed=101
)

# Extract a 2-way, 5-shot episodic task in milliseconds
X_task, Y_task, Metadata = sampler.sample(
    target_numeric_classes=(0, 1), 
    samples_per_class=(5, 5)
)

---

☁️ Google Colab Integration & Real-World Benchmarks

Training on the cloud? CWRU-Plus is fully optimized for Google Colab environments. We exploit lightning-fast local ephemeral storage (/content/) for heavy processing, while spawning non-blocking background threads (safe_copy) to silently sync your massive .npz and .zip archives to your Google Drive (/content/drive/MyDrive/).

⏱️ Live Colab Benchmarks (Proven Speed)

Here is the exact execution breakdown measured via %%time in a standard Google Colab environment:

Pipeline Step	Processed Data	CPU Time	Wall Time (Real Waiting)	Key Takeaway
1. Download & Zip	~300 MB Raw .mat Files	20.4 s	1 min 22 s	Parallel fetching. Only ~16s spent on compression!
2. Ingest & Save	All Ingested Signals	1.7 s	1.55 s	Lightning-fast unification into a structured .npz.
3. Parallel Filtering	4th-Order Butter + Median	55.5 s	46.7 s	Over-utilized multi-core filtering (36s raw computation).
4. In-Memory Ingestion	Full Sliding Windows + Masking	660 ms	660 ms	Zero-copy vectorization. ML-ready in less than a second!

Why the CPU Time vs. Wall Time discrepancy? In Step 3, CPU time is 55.5s while Wall time is only 46.7s. This proves our multi-threaded execution is successfully utilizing multiple cores simultaneously, saving you precious waiting time.

👉 Run the interactive, benchmarked End-to-End Pipeline in Colab right now!

---

📊 Output Tensor Formats

The cwru.load() method is highly flexible. Depending on your configuration, it guarantees strict alignment between signals and metadata:

• Standard Flattened: X.shape = (Total_Windows, Channels, Window_Size)
• Time-Partitioned (Cross-Validation Ready): By setting num_parts=10, X becomes (10, Windows_per_Part, Channels, Window_Size). Perfect for preventing Data Leakage in time-series!
• Dynamic Labels: Choose from 'only types', 'types & severity', or full structural strings 'types, severity & locations' like 'OR014@6'.

---

💡 The "Why": Solving the CWRU Bottleneck

As researchers in Condition Monitoring and Fault Diagnosis, we realized that 50-70% of our time was wasted on "Data Engineering" rather than actual Deep Learning research. We built CWRU-Plus to eliminate the three biggest bottlenecks in the field:

1. The Preprocessing Time-Sink ⏳

• The Problem: Traditional scripts use nested for loops and naive file reading to parse hundreds of .mat files, which can take hours.
• The CWRU-Plus Solution: We implemented high-performance Multi-threading for file ingestion and NumPy's zero-copy sliding_window_view for instantaneous window extraction. What used to take hours now takes less than 3 seconds.

2. Data Leakage & Alignment Risks ⚠️

• The Problem: Manually aligning metadata (HP, Fault Type, Severity) with fragmented time-series signals is highly error-prone, often leading to silent data leakage and ruined experiments.
• The CWRU-Plus Solution: Our engine uses an "Atomic Record" strategy. Signals and metadata are locked together into a single dictionary during parallel extraction, guaranteeing 100% strict row-level alignment across the entire dataset.

3. Rigid DSP Pipelines 🧱

• The Problem: Existing repositories hardcode their preprocessing steps, making it a nightmare to apply custom noise removal or filtering.
• The CWRU-Plus Solution: Through Inversion of Control (IoC), CWRU-Plus completely decouples the data loading from the signal processing. You can inject isolated, custom linear or non-linear DSP filters per individual channel (e.g., Drive End vs. Fan End) without touching the core library.

---

🔮 What's Next? We want your feedback!

We built CWRU-Plus to accelerate our own research, but this library is for the community. We want to know what would make your research even faster!

What features would you like to see next?

• Native PyTorch Dataset/DataLoader wrappers?
• Built-in Time-Frequency transformations (STFT, Wavelets, Hilbert-Huang)?
• Support for other bearing datasets (PU, IMS, XJTU-SY)?

👉 Open an Issue on GitHub and tell us what you want us to build next!

---

Why CWRU-Plus? (Comparison with Alternatives)

While legacy tools on PyPI—such as cwru, py-cwru, multivariate-cwru, and bearing-python—laid the foundation for using this dataset, they were built for older Python ecosystems and lack the performance optimization required for modern Deep Learning pipelines.

Below is a technical matrix showing how CWRU-Plus redesigns the entire data engineering layer compared to existing alternatives:

Feature / Capability	Legacy Packages (cwru, py-cwru, etc.)	⚡ CWRU-Plus
Python 3.10+ Compatibility	❌ No (Many crash on modern collections or numpy types)	✅ Yes (Native, production-ready)
Multi-Threaded Downloading	❌ No (Single-threaded, slow legacy sequential mirrors)	✅ Yes (Parallel high-speed downloads)
Smart Cache Management	❌ No (Re-downloads or fails if archives already exist)	✅ Yes (Detects, validates, and skips existing files)
Parallel DSP Filtering (IoC)	❌ No (Sequential preprocessing bottlenecks)	✅ Yes (Multi-core parallelized filter injection)
Atomic Metadata Extraction	❌ No (Only returns raw signals; manual mapping needed)	✅ Yes (Returns synchronized Horse Powers & Fault Severity vectors)
Strict Data Leakage Protection	❌ No (Manual sliding windows often mix train/test frames)	✅ Yes (Guaranteed 100% atomic boundary separation)
Built-in Few-Shot Sampler	❌ No (Requires manual implementation of episodic tasks)	✅ Yes (High-performance episodic task sampler)
Processing Speed	⚠️ Slow (Heavy disk I/O and object creation overhead)	🚀 Blazing Fast (< 3 seconds via zero-copy NumPy views)

🔥🔥🔥

---

Detailed Benchmarks & Technical Edge

---

1. Smart Local Cache Management

Legacy packages usually struggle if you interrupt a download or want to use previously downloaded files, often resulting in corrupted state errors or redundant web requests. CWRU-Plus acts as an intelligent file manager: it automatically scans your ./raw directory, verifies existing files against the official structural grid, and seamlessly proceeds with ingestion without wasting network bandwidth.

2. Synchronized Meta-Tracking (RPM & Severity)

Most alternative packages strip out or ignore the underlying operational context, leaving you with raw signal matrices. For advanced tasks like Domain Adaptation or Regression-based Remaining Useful Life (RUL) estimation, you need the operational metadata. CWRU-Plus keeps Severity and RPM bound to each window instance, returning clean, production-ready vectors out of the box.

3. Native Meta-Learning Support (Few-Shot Sampler)

If you are training Prototypical Networks, Relation Networks, or MAML architectures, you typically have to write hundreds of lines of boilerplate code to handle N-way K-shot episodic sampling. CWRU-Plus introduces a highly optimized FewShotSampler that maps string categories to clean, single-integer target labels ideal for PyTorch CrossEntropyLoss at microscopic latencies (under 5ms per batch execution).

---

🚀 The Next Generation Machinery Engine is Here!

🔥 UPDATE: We have officially expanded our industrial machinery fault diagnosis ecosystem!

While we previously focused heavily on CWRU, we have officially launched MAFAULDA-Plus—a powerhouse data engine designed to handle massive, multi-channel vibration signal processing without crashing your environment.

• 🌐 GitHub Repository: Discover MAFAULDA-Plus on GitHub
• 📦 PyPI Package: pip install mafaulda-plus

If your research is expanding into complex, cross-domain multi-class fault diagnosis with zero-RAM memory constraints, migrate to MAFAULDA-Plus today! 🏁

---

🤝 Contributing & License

Contributions, bug reports, and feature requests are highly welcome! Feel free to open an issue or submit a Pull Request.

This project is open-source and licensed under the MIT License.