aero-rul

How many flights does this engine have left, and how far can that answer be trusted?

aero-rul predicts the remaining useful life (RUL) of turbofan engines on NASA's N-CMAPSS dataset and attaches a prediction interval to every estimate. The interval's coverage is not assumed: it is measured on engines the model has never seen.

Result (DS02, three held-out engines, five seeds). A 90 % conformal interval contains the true RUL in 90.4 % of test flights. Plain quantile regression, asked for the same 90 %, delivers 82.2 % with wider intervals. Point error: 10.1 flights RMSE.

Why this project

Airlines plan engine maintenance around the flights an engine has left. Too early wastes a serviceable engine; too late risks an in-service failure. A point estimate is not enough to plan with. A planner needs a range, and a statement of how often such ranges are right.

The project is built around four rules:

• An engine is either in training or in evaluation, never both. Consecutive flights of one engine are nearly identical; a row-level split lets the model memorise.
• Uncertainty is measured, not assumed. Coverage is reported pooled and per engine.
• No label leakage. The simulator's hidden health parameters, its health-state flag and the flight counter are never inputs.
• Comparisons are fair or not made. Same budget, same validation, several seeds.

Results

All numbers: N-CMAPSS DS02, trained on engines 2, 5, 10, 16, 18, 20, tested on 11, 14, 15, mean ± std over five seeds. Reproduce with the commands under Usage.

Point prediction

RMSE and MAE in flights. The NASA score penalises late predictions (more life predicted than remains) harder than early ones; lower is better. CV is leave-one-engine-out on the six training engines.

History features help. Adding the trailing mean and slope over each engine's last ten flights lowered LightGBM's test RMSE from 10.59 ± 0.25 to 10.05 ± 0.43 and its CV NASA score per flight from 1.91 to 1.59. Ridge got worse with the same features (CV RMSE 12.91 to 17.06): 146 correlated features on 446 flights overwhelm a lightly regularised linear model (alpha = 1), while the tree ensemble absorbs them.

Prediction intervals

• Conformal calibration works. Cross-conformal lands on its 90 % target with the narrowest intervals. Uncalibrated quantile regression promises 90 % and delivers 82 %.
• Coverage is marginal, not per engine. At 90 %, cross-conformal covers 96.6 % of engine 11, 96.4 % of engine 15, but only 80.3 % of engine 14, the engine with the largest point error. Conformal guarantees hold on average over engines; a planner should know which engine is the outlier.
• CQR over-covers here. Conformalized quantile regression is valid (coverage at least the target) but asked for 90 % it covers 98.5 %, and at both levels its intervals are about 50 % wider than cross-conformal's. Its calibration scores come from held-out training engines, which are harder than the test engines; with six engines that gap dominates.

What the model looks at

The exhaust-side temperature T50 (low-pressure turbine outlet) dominates, followed by core speed Nc and T48 (high-pressure turbine outlet). Rising turbine temperature for the same thrust is the classic sign of engine wear in practice (EGT margin). Fourteen of the fifteen most important features are ten-flight averages: the model reads trends, not single flights. SHAP explains the model, not the physics; correlated sensors share credit.

Hyperparameter search: ABC vs random search vs TPE

Three optimisers tune LightGBM's seven main hyperparameters under identical conditions: 60 candidate evaluations each, the same search space, leave-one-engine-out CV on the training engines, five seeds. ABC is the canonical Artificial Bee Colony algorithm (Karaboga) from my own package abcopt.

• On the search objective, TPE wins and ABC comes last. With 60 evaluations and ten food sources ABC completes only a few employed/onlooker cycles; it is reported as is.
• None of it carries over to new engines. Every optimiser lowered CV RMSE below the defaults, and every tuned model is worse on the test engines than the untuned one (about 11.1 vs 10.05). With six training engines, the search fits the validation engines rather than engine wear. The test differences between optimisers are well inside one standard deviation.
• So the shipped model uses default parameters. On data this small, a better validation score is not evidence of a better model.

Install

pip install aerorul

or for development:

git clone https://github.com/yusufkorkmazyigit/aero-rul
cd aero-rul
pip install -e ".[dev]"

Data

Download Turbofan Engine Degradation Simulation Data Set-2 from the NASA Prognostics Data Repository and place N-CMAPSS_DS02-006.h5 (2.4 GB) in data/raw/. Only this file is used.

Usage

aerorul prepare        # 6.5 M one-second rows -> 648 flights (one row each)
aerorul evaluate       # baselines, leave-one-engine-out CV and test scores
aerorul uncertainty    # prediction intervals and their coverage
aerorul explain        # SHAP importance per signal
aerorul benchmark      # ABC vs random search vs TPE, about 15 minutes on 16 threads

Results go to results/ as Markdown tables, CSV files and figures. Every command reads configs/default.yaml; --config other.yaml runs another experiment.

How it works

1. Per-flight features. Each signal (4 flight conditions, 14 sensors) is summarised over every flight by its mean, standard deviation, minimum and maximum.
2. History features. For each engine, the trailing mean and slope of those summaries over its last ten flights. Only past flights of the same engine are used.
3. Models. Ridge and LightGBM, evaluated with leave-one-engine-out CV and on the test engines.
4. Intervals. Split conformal, cross-conformal, quantile regression and CQR, with coverage checked pooled and per engine.

The method page explains each choice, with formulas and references.

Limitations

• Three test engines are few. Results are given per engine as well as pooled.
• N-CMAPSS is a high-fidelity simulation, not fleet data.
• Conformal intervals have constant width; normalised conformal scores that narrow the interval near end of life are the natural next step.

Development

pytest          # 41 tests on a synthetic file with the N-CMAPSS layout; no download
ruff check .

Citation

See CITATION.cff. The dataset: M. Arias Chao, C. Kulkarni, K. Goebel, O. Fink, Aircraft Engine Run-to-Failure Dataset under Real Flight Conditions for Prognostics and Diagnostics, Data 6(1):5, 2021.

License

MIT. N-CMAPSS is released by NASA under CC0.