cve-risk-engine

Calibrated random forest scoring of CVE exploitability, packaged with the honest limits of the study it comes from.

This is the reference implementation of the scoring layer of

Exploitability-Driven CVE Prioritization for Security Operations: A Recursive Feature Elimination Random Forest Approach — Reza Yuzron Wardana, Hari Purnama. Manuscript in preparation. Code, data and evaluation artifacts: CVE-RiskEngine-RF

What this package is not. It is not a claim that machine learning beats EPSS. On the study's held-out temporal test partition the forest ranks better than disclosure-time EPSS, but the difference in average precision is not statistically established, the raw scores are not probabilities, and a CVSS-only score stays close behind. The package ships those facts as metadata and refuses to return an uncalibrated number as a probability.

Install

pip install CVE-RiskEngine-RF
# optional HTTP service
pip install "CVE-RiskEngine-RF[api]"

Use

from cve_risk_engine import CveRiskEngine

engine = CveRiskEngine.load()
result = engine.score(
    cvss_vector="CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
    cvss_score=9.8,
    epss_score=0.00061,   # EPSS at or before the disclosure date, not today's value
    cve_id="CVE-2025-14611",
    cwe="CWE-798",
)

{
  "cve_id": "CVE-2025-14611",
  "raw_model_score": 0.897937,
  "calibrated_probability": 0.012108,
  "cvss_score": 9.8,
  "epss_score": 0.00061,
  "final_risk_score": 0.4961,
  "risk_level": "medium",
  "priority_band": "critical",
  "priority_score": 0.012108,
  "priority_action": "Patch within 7 days",
  "recommendation": "Patch within 30 days",
  "features": {...}
}

risk_level and priority_band disagree on purpose. The first is the CVSS severity scale, the second is the operational band. See "Prioritize with priority_band" below.

Command line:

cve-risk-engine info
cve-risk-engine score --vector "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H" \
                      --cvss 9.8 --epss 0.00061 --cwe CWE-798 --id CVE-2025-14611
cve-risk-engine batch records.csv --out scored.csv

HTTP:

uvicorn cve_risk_engine.api:app --port 8000
# GET /health  GET /model  POST /score  POST /score/batch

Read the two numbers differently

What the model actually does

Corpus: 44,474 CVEs published in 2025 with a CVSS v3.1 vector. Target: CISA KEV entry within 90 days of disclosure, 109 positives (0.245%); CVEs already KEV-listed at disclosure are excluded, so no record is right-censored. Temporal split: train 25,951 (70 positives) to 2025-07-31, validation 7,363 (18) Aug–Sep, test 11,160 (21) Oct–Dec. Features: 7 of 13 candidates retained by RFECV on the training partition. Imbalance: SMOTE 1:1 chosen on validation, applied inside the training folds only.

Ranking differences on average precision and MCC are inside their bootstrap intervals. The recall gain is real but comes at roughly twice the queue length.

KEV membership is not a risk band, and high/critical are unreachable

A natural question is whether a CVE that is listed in CISA KEV comes back as high or critical. It does not, and the engine never accepts KEV as an input: KEV is the evaluation label of the study, never a feature, so a served decision is reproducible from the model alone. On the test partition the 21 KEV-listed CVEs come back as 15 medium and 6 low, none above medium, with final_risk_score between 0.3058 and 0.5061. A KEV CVE at CVSS 7.5 lands in low; so does one at CVSS 6.1.

The reason is the one in the second limitation below: at a 0.245% base rate, calibrated_probability is at most a few percent, so FinalRisk is dominated by the CVSS term and saturates around 0.53. Read the band as a severity-flavoured label, not as an exploitation verdict, and never as a KEV oracle. Rank by final_risk_score and cut by queue capacity; that is the operating point the study evaluates.

Two limitations you must design around

1. Raw scores are not probabilities. They are ranked correctly but are badly calibrated (Brier 0.0629). Always read calibrated_probability, never raw_model_score, when a number is shown to a human.

2. The high and critical bands are unreachable. The bounds come from the CVSS v3.1 severity scale on the unit interval, but once the score is calibrated on a 0.245% base rate, final_risk_score cannot exceed about 0.53 even at CVSS 10.0. On the test partition the high and critical bands hold zero records; everything lands in low (8,421 records, 6 exploited) or medium (2,739 records, 15 exploited). Precision still rises monotonically between them, but a banding scheme usable under calibrated probabilities has to be defined on validation quantiles of the score, not on the CVSS scale. For that reason the metadata ships capacity_cutoffs_validation (the score above which the top 0.5%, 1%, 2%, 5% and 10% of the validation backlog fall) and the package README recommends cutting by capacity. Treating risk_level as an absolute severity rating is wrong.

Prioritize with priority_band, not with risk_level

The CVSS-scale bounds were derived and verified on the raw score of the pre-revision model, whose range reached 0.98: on that scale the high band held 20 records and critical held 2, with precision rising monotonically to 0.50. They were then carried over to the calibrated score without being re-derived, and the scale had moved from 0 to 0.98 down to 0 to 0.013. That is why the upper half of the scale is now empty and why an exploited CVE can come back as low.

priority_band is the replacement. Its bounds are quantiles of calibrated_probability on the validation partition, shipped in metadata.json:

The bounds are not an assumption. Three candidate scales were measured against a rule fixed in advance, that per-band precision must fall monotonically from critical to low and the property must survive out of sample:

Fine cuts fail on every scale because the validation partition holds only 18 positives, so a 0.5% cut is 38 records and one record flips the estimate. FinalRisk fails everywhere because its top is CVSS dominated and its four-decimal rounding creates large tie mass. calibrated_probability and raw_model_score rank identically, so the readable one was chosen.

Applied unchanged to the held-out temporal test partition (11,160 records, 21 positives, base rate 0.188%):

Precision falls monotonically, critical is 7.4 times the base rate and low is 0.2 times it, and 14 of the 21 exploited records land in the top two bands. The same partition scored with the CVSS-scale bounds puts zero exploited records in the top two bands. Every number here is reproduced in metadata.json under priority_bands.

Blended-score sensitivity on the calibrated probability, for reference: test

average precision rises with the model weight (0.0295 at 0.5/0.5, 0.0331 at 0.7/0.3, 0.0347 at 0.8/0.2) against 0.0159 for the model alone and 0.0102 for CVSS alone, while the operational cost of a 500-record queue is identical (1,590, with 10 of the 21 exploited CVEs caught) for every blend from 0.5 upward. Equal weights are used because they sit in the middle of that flat region.

Bundled artifact

cve_risk_engine/artifacts/ holds the classifier, the Platt map, the CWE frequency map fitted on the training partition, and a metadata file recording the hyperparameters, the target definition, the split windows, the test metrics, the paired intervals, the input SHA-256 digests and the limitation above. The artifact reproduces the manuscript's test numbers exactly (average precision 0.015903, ROC-AUC 0.880724), and tests/reference_scores.csv freezes 821 test records including all 21 positives so anyone can re-verify that claim without the study repo:

python -m pytest tests -q

Rebuilding the artifact

The artifact is exported from the study pipeline, never hand-edited:

python scripts/export_artifact.py \
    --study-repo /path/to/CVE-RiskEngine-RF \
    --out src/cve_risk_engine/artifacts
python scripts/build_reference_scores.py \
    --study-repo /path/to/CVE-RiskEngine-RF \
    --out tests/reference_scores.csv

The first script asserts that the exported model's test average precision and ROC-AUC are bit-identical to revision/outputs/results.json; if they drift, the export fails rather than shipping a lookalike.

Changelog

0.1.2 — adds the operational band, and closes the four findings of the independent audit of 0.1.1 plus one raised by a later regression and comparison run.

• Added priority_band, priority_score and priority_action, defined on validation quantiles of calibrated_probability. risk_level is unchanged, so existing consumers keep the same numbers, but it is now documented as the CVSS severity scale and explicitly not an exploitability verdict.
• epss_score and cvss_score are now range checked in CveRiskEngine.score. The core path previously accepted -0.1, 1.5 and nan and returned a confident looking score for all three. This was audit finding F-09.
• The OpenAPI schema version now comes from __version__ instead of a hardcoded 0.1.0, and the project name in the api.py docstring and ImportError is the real distribution name, so the instruction it prints can actually be followed. Findings F-01 and F-02.
• metadata.json now states the decision threshold on the scale it belongs to (raw_model_score 0.049528, not a calibrated probability), because reusing it on the calibrated scale returns an empty queue. Finding F-06.
• metadata.json now records the six-decimal rounding of returned scores and the consequence for anyone recomputing the published metrics from package output: ROC-AUC moves from 0.880724 to 0.881031 and the queue moves from 3,228 to 3,229. Finding F-10.
• The CLI batch output carries priority_band and priority_action; the HTTP response carries both plus priority_score.

0.1.1 — first published release: calibrated model, reference scores, CLI and optional HTTP service.

Licence

MIT. The bundled artifact is trained on derived metadata from NVD, CISA and FIRST; the upstream terms of those sources apply to the data.