RF Modulation Classifier

Nishad Suresh

Abstract

This project implements and evaluates an automatic modulation classification (AMC) pipeline for radio signals, distinguishing five modulation schemes (BPSK, QPSK, 8PSK, AM-DSB, WBFM) from raw in-phase/quadrature (I/Q) samples. The pipeline combines classical signal-processing features -- instantaneous statistics, higher-order cumulants, and spectral peak detection -- with a trained multilayer-perceptron classifier, and characterizes classification accuracy as a function of signal-to-noise ratio (SNR) on held-out data. Training and evaluation use the real RadioML 2016.10a dataset. The classifier reaches 86.70% accuracy at SNR >= 4dB, with an accuracy-vs-SNR curve rising from near-chance below -8dB to a realistic 85-87% plateau at high SNR, consistent with the known accuracy ceiling of classical (non-learned) feature sets on real recorded RF data.

Status: all 5 phases complete (2026-07-18), trained and evaluated on the real RadioML 2016.10a dataset.

1. Motivation

Automatic modulation classification is a well-established problem in radio-frequency signal processing and cognitive radio: given a raw stream of I/Q samples, identify which modulation scheme produced them without prior knowledge of the transmitter. This project implements the full pipeline end to end -- signal loading, feature engineering, classifier training and selection, an accuracy-vs-noise characterization evaluated on genuinely held-out data, and a live-replay demonstration -- rather than treating any single stage in isolation.

2. Data

Training and evaluation use the public RadioML 2016.10a dataset (DeepSig Inc., CC BY-NC-SA 4.0): 11 modulation classes across SNRs from -20 dB to +18 dB, obtained via the Kaggle mirror (nolasthitnotomorrow/radioml2016-deepsigcom). This work targets the 5-class subset (BPSK/QPSK/8PSK/AM-DSB/WBFM); src/dataset.py filters the full 11-class file down to these 5 automatically. Class selection matches the project's original scope (the digital PSK family plus two analog modes); see that file for full source and license documentation.

The dataset's official DeepSig host was unavailable (HTTP 502) and IEEE DataPort required a paid subscription at the time this project began, so src/dataset.py also includes a synthetic I/Q generator, used to validate the pipeline before real data was obtained. It is retained as a fallback should data/ ever be empty, and is no longer used for the results reported below.

3. Setup

python -m venv .venv
source .venv/bin/activate  # or .venv\Scripts\activate on Windows
pip install -r requirements.txt

# get the data (either works):
kaggle datasets download -d nolasthitnotomorrow/radioml2016-deepsigcom -p data/ && \
python -c "import zipfile; zipfile.ZipFile('data/radioml2016-deepsigcom.zip').extractall('data/')"
# or manually download RML2016.10a.tar.bz2 / the extracted .pkl into data/

4. Methodology and Results

#	Phase	Acceptance test	Result (real data)
1	Repo + RadioML loader + spectrogram viewer	Loads I/Q data; plots one labeled spectrogram per class	✅ results/spectrogram_*.png
2	Feature library (cumulants, spectral peaks, instantaneous stats)	Classes visibly separate in a 2-D feature scatter	✅ results/feature_scatter_pca.png
3	Classifier + evaluation on held-out recordings	≥85% accuracy at reasonable SNR	✅ 86.70% (MLP, SNR≥4dB, n=10,000 held out) — results/confusion_matrix.png, results/metrics.json
4	SNR-vs-accuracy curve	Monotonic-ish curve, the headline result	✅ results/snr_vs_accuracy.png
5	Live-replay demo	End-to-end demo runs	✅ 11/12 correct (SNR≥4dB, matches training range) — results/live_replay_demo.png

python run_phase1.py        # spectrogram viewer
python run_phase2.py        # feature scatter
python src/train.py         # classifier + confusion matrix + metrics.json
python run_phase4.py        # SNR-vs-accuracy curve
python src/demo.py          # live-replay demo (needs src/train.py run first)
python tests/test_train.py  # regression test guarding the no-leakage fix (fast, uses synthetic data)

4.1 Feature-space class separation

4.2 Confusion matrix (best model: MLP, 86.70% at SNR≥4dB)

4.3 Accuracy vs. SNR

Accuracy is near-chance (~20%, 5 classes) below -8dB, rising smoothly and plateauing around 85-87% above +2dB. This plateau, rather than an approach to 100%, is expected for real recorded signals: unlike an idealized synthetic generator, real RadioML captures include genuine channel imperfections (timing/frequency offsets, hardware nonlinearities) that impose a realistic ceiling on classical-feature accuracy.

4.4 Live-replay demo

This simulates a receiver replaying captures and predicting the modulation of each in real time, sampled from the same SNR range (≥4dB) used for training and evaluation; 11/12 correct is consistent with the 86.70% headline accuracy at this sample size.

Switching to a live RTL-SDR feed requires a one-line change (see the docstring in src/demo.py): swap the replay_source() generator for one pulling samples from pyrtlsdr instead of the dataset. Everything downstream (feature extraction, prediction, plotting) is unchanged.

5. Methodology Review: Two Issues Found and Corrected

Every source file was passed through an independent quality-review pass (a fresh reviewer with no attachment to the code, distinct from the context that wrote it). Two issues were identified that affected the validity of the headline accuracy figure, and are documented here rather than silently patched.

Test-set leakage in model selection. src/train.py originally selected the "best" model by comparing accuracy on X_test, then reported that same X_test accuracy as the held-out result -- a leakage of the test set into model selection. This was corrected by selecting via 5-fold cross-validation on the training set only, so X_test is now touched exactly once, for the final reported number. The corrected accuracy came out at 86.70%, identical to the pre-fix value, indicating the leak was not materially inflating this particular result, but the methodology is now correct by construction rather than correct by coincidence. src/train.py now exposes get_train_test_split() as the single source of truth for the split, and tests/test_train.py is a regression test guarding against this recurring.

A demo that was not actually evaluating held-out data. src/demo.py's "held-out" claim did not hold: it sampled independently from the full dataset rather than from train.py's real test partition, so replayed captures could statistically overlap with training data. This was corrected by having it call the same get_train_test_split() used by train.py, so the demo now draws only from genuinely unseen examples.

A smaller correctness issue was also fixed in src/features.py: an intermediate cumulant variable computed the standard-literature M41 formula but was labeled m42 (and vice versa). The final c42 output was numerically correct throughout, but the naming was reversed and one line was dead code. Full detail is preserved in each file's git history.

6. Limitations

8PSK/QPSK confusion. This appears consistently in both the feature scatter and the confusion matrix. These two classes are genuinely difficult to separate with hand-engineered features at moderate SNR, consistent with published AMC literature rather than indicating an implementation defect.

Classical feature set. The features used here are deliberately classical (cumulants, spectral peaks, instantaneous statistics) rather than a learned representation such as a CNN over raw I/Q, a scope choice made to keep the project interpretable and tractable to build, at some cost to peak accuracy relative to deep-learning AMC approaches.

Partial class coverage. 5 of the dataset's 11 classes are used, selected to match the project's original scope. Extending to the full 11-class problem is a natural next step but a substantially harder classification task.

7. Summary

This project implements a radio-signal modulation classifier trained and evaluated on the real public RadioML dataset, reaching 86.70% accuracy at SNR ≥ 4dB, and characterizes how that accuracy degrades with decreasing SNR, from near-chance below -8dB to a realistic 85-87% plateau at high SNR.

References

Sources used to design, validate, and cross-check this project's methodology:

[1] T. J. O'Shea and N. West, "Radio Machine Learning Dataset Generation with GNU Radio," Proceedings of the GNU Radio Conference, vol. 1, 2016. https://pubs.gnuradio.org/index.php/grcon/article/view/11 -- origin of the RadioML dataset used here.

[2] T. J. O'Shea, J. Corgan, T. C. Clancy, "Convolutional Radio Modulation Recognition Networks," Engineering Applications of Neural Networks (EANN 2016), Springer, pp. 213-226. https://doi.org/10.1007/978-3-319-44188-7_16

[3] A. Swami and B. M. Sadler, "Hierarchical Digital Modulation Classification Using Cumulants," IEEE Transactions on Communications, vol. 48, no. 3, 2000, pp. 416-429. https://doi.org/10.1109/26.837045 -- foundation for the cumulant features in src/features.py.

[4] "Deep Learning for Automatic Modulation Classification: A Review," Electronics, MDPI, vol. 15, no. 10, 2026. https://www.mdpi.com/2079-9292/15/10/2163

[5] "Deep Learning-Based Automatic Modulation Classification for OFDM Signals: From Synthetic Training to OTA Evaluation," Sensors, MDPI, 2026. https://doi.org/10.3390/s26102945

[6] "Cross-Validated Cross-Channel Self-Attention and Denoising for Automatic Modulation Classification," arXiv:2604.10054. https://arxiv.org/pdf/2604.10054

[7] "Cloud-enabled automatic modulation classification using deep feature fusion and Moth-Flame Optimized ELM approach," Scientific Reports, Nature, 2025. https://www.nature.com/articles/s41598-025-30753-4