cve-risk-engine

Calibrated random forest scoring of CVE exploitability, packaged with the honest limits of the study it comes from.

This is the reference implementation of the scoring layer of

Exploitability-Driven CVE Prioritization for Security Operations: A Recursive Feature Elimination Random Forest Approach — Reza Yuzron Wardana, Hari Purnama. Manuscript in preparation. Code, data and evaluation artifacts: CVE-RiskEngine-RF

What this package is not. It is not a claim that machine learning beats EPSS. On the study's held-out temporal test partition the forest ranks better than disclosure-time EPSS, but the difference in average precision is not statistically established, the raw scores are not probabilities, and a CVSS-only score stays close behind. The package ships those facts as metadata and refuses to return an uncalibrated number as a probability.

Scope: early warning, not a risk register

This distinction decides how every number in this README must be read, so it comes first.

The engine answers exactly one question: how likely is this CVE to be listed in CISA KEV within 90 days of its disclosure, judged only from what was available at disclosure time. It is an early-warning and prioritization artifact. It is not a description of a CVE's current state, and it is not a risk register.

The practical consequence, stated plainly because it surprises people: a CVE that KEV has already confirmed as exploited can come back with a low or medium band. That is not a claim that the CVE is low risk. It is a scope boundary. At the moment the model is meant to be run, the outcome of that CVE was still unknown, and the model placed a bet. When the bet was placed the answer key did not exist yet.

Two rules follow, and both matter:

1. Only score records whose exploitation outcome is still unknown. Applying this model to a record that is already in KEV turns a prediction into a restatement of a known answer.
2. Never wire KEV into the scoring path to force such records upward. KEV is the evaluation label, never an input. A model that only says "critical" after KEV confirms it adds nothing over KEV itself, and this study exists precisely to criticize exploitation signals that react only after confirmation.

If a deployment needs a risk register rather than an early-warning feed, mark KEV-listed records as a separate state at the decision layer and keep that marker out of the model and out of its metrics. The consequence to accept on this side of the line is that recall is not perfect: on the study's test partition the highest band covers 5.8% of the backlog and holds 1.4% exploited records against a 0.188% base rate, so most confirmed-exploited records are found, and some are missed.

Install

pip install CVE-RiskEngine-RF
# optional HTTP service
pip install "CVE-RiskEngine-RF[api]"

Use

from cve_risk_engine import CveRiskEngine

engine = CveRiskEngine.load()
result = engine.score(
    cvss_vector="CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
    cvss_score=9.8,
    epss_score=0.00061,   # EPSS at or before the disclosure date, not today's value
    cve_id="CVE-2025-14611",
    cwe="CWE-798",
)

{
  "cve_id": "CVE-2025-14611",
  "priority_band": "critical",
  "priority_action": "Patch within 7 days",
  "priority_score": 0.012108,
  "priority_band_scale": "calibrated_probability",
  "calibrated_probability": 0.012108,
  "raw_model_score": 0.897937,
  "cvss_score": 9.8,
  "epss_score": 0.00061,
  "final_risk_score": 0.4961,
  "risk_level": "medium",
  "risk_level_note": "legacy severity label ...",
  "recommendation": "Patch within 30 days",
  "features": {...}
}

Read priority_band. It is the band. It comes first in the returned mapping on purpose, and metadata.json records it under primary_band so a consumer can discover that programmatically instead of being told in prose. Everything above it in the output is the operational answer and the scale it is computed on.

risk_level is kept only for continuity. It is the CVSS severity scale applied to final_risk_score, it is not an exploitation verdict, and its bounds above 0.70 are unreachable once the probability is calibrated, so a record that CISA has confirmed as exploited can still read medium or even low there. Do not filter, sort or report by it. The field carries its own risk_level_note saying exactly that, so a caller who prints the whole mapping cannot miss it. See "Prioritize with priority_band" below.

Command line:

cve-risk-engine info
cve-risk-engine score --vector "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H" \
                      --cvss 9.8 --epss 0.00061 --cwe CWE-798 --id CVE-2025-14611
cve-risk-engine batch records.csv --out scored.csv

HTTP:

uvicorn cve_risk_engine.api:app --port 8000
# GET /health  GET /model  POST /score  POST /score/batch

Read the two numbers differently

What the model actually does

Corpus: 44,474 CVEs published in 2025 with a CVSS v3.1 vector. Target: CISA KEV entry within 90 days of disclosure, 109 positives (0.245%); CVEs already KEV-listed at disclosure are excluded, so no record is right-censored. Temporal split: train 25,951 (70 positives) to 2025-07-31, validation 7,363 (18) Aug–Sep, test 11,160 (21) Oct–Dec. Features: 7 of 13 candidates retained by RFECV on the training partition. Imbalance: SMOTE 1:1 chosen on validation, applied inside the training folds only.

Ranking differences on average precision and MCC are inside their bootstrap intervals. The recall gain is real but comes at roughly twice the queue length.

KEV membership is not a risk band, and high/critical are unreachable

A natural question is whether a CVE that is listed in CISA KEV comes back as high or critical. It does not, and the engine never accepts KEV as an input: KEV is the evaluation label of the study, never a feature, so a served decision is reproducible from the model alone. On the test partition the 21 KEV-listed CVEs come back as 15 medium and 6 low, none above medium, with final_risk_score between 0.3058 and 0.5061. A KEV CVE at CVSS 7.5 lands in low; so does one at CVSS 6.1.

The reason is the one in the second limitation below: at a 0.245% base rate, calibrated_probability is at most a few percent, so FinalRisk is dominated by the CVSS term and saturates around 0.53. Read the band as a severity-flavoured label, not as an exploitation verdict, and never as a KEV oracle. Rank by final_risk_score and cut by queue capacity; that is the operating point the study evaluates.

Two limitations you must design around

1. Raw scores are not probabilities. They are ranked correctly but are badly calibrated (Brier 0.0629). Always read calibrated_probability, never raw_model_score, when a number is shown to a human.

2. The high and critical bands are unreachable. The bounds come from the CVSS v3.1 severity scale on the unit interval, but once the score is calibrated on a 0.245% base rate, final_risk_score cannot exceed about 0.53 even at CVSS 10.0. On the test partition the high and critical bands hold zero records; everything lands in low (8,421 records, 6 exploited) or medium (2,739 records, 15 exploited). Precision still rises monotonically between them, but a banding scheme usable under calibrated probabilities has to be defined on validation quantiles of the score, not on the CVSS scale. For that reason the metadata ships capacity_cutoffs_validation (the score above which the top 0.5%, 1%, 2%, 5% and 10% of the validation backlog fall) and the package README recommends cutting by capacity. Treating risk_level as an absolute severity rating is wrong.

Prioritize with priority_band, not with risk_level

The CVSS-scale bounds were derived and verified on the raw score of the pre-revision model, whose range reached 0.98: on that scale the high band held 20 records and critical held 2, with precision rising monotonically to 0.50. They were then carried over to the calibrated score without being re-derived, and the scale had moved from 0 to 0.98 down to 0 to 0.013. That is why the upper half of the scale is now empty and why an exploited CVE can come back as low.

priority_band is the replacement. Its bounds are quantiles of calibrated_probability on the validation partition, shipped in metadata.json:

The bounds are not an assumption. Three candidate scales were measured against a rule fixed in advance, that per-band precision must fall monotonically from critical to low and the property must survive out of sample:

Fine cuts fail on every scale because the validation partition holds only 18 positives, so a 0.5% cut is 38 records and one record flips the estimate. FinalRisk fails everywhere because its top is CVSS dominated and its four-decimal rounding creates large tie mass. calibrated_probability and raw_model_score rank identically, so the readable one was chosen.

Applied unchanged to the held-out temporal test partition (11,160 records, 21 positives, base rate 0.188%):

Precision falls monotonically, critical is 7.4 times the base rate and low is 0.2 times it, and 14 of the 21 exploited records land in the top two bands. The same partition scored with the CVSS-scale bounds puts zero exploited records in the top two bands. Every number here is reproduced in metadata.json under priority_bands.

Blended-score sensitivity on the calibrated probability, for reference: test

average precision rises with the model weight (0.0295 at 0.5/0.5, 0.0331 at 0.7/0.3, 0.0347 at 0.8/0.2) against 0.0159 for the model alone and 0.0102 for CVSS alone, while the operational cost of a 500-record queue is identical (1,590, with 10 of the 21 exploited CVEs caught) for every blend from 0.5 upward. Equal weights are used because they sit in the middle of that flat region.

Bundled artifact

cve_risk_engine/artifacts/ holds the classifier, the Platt map, the CWE frequency map fitted on the training partition, and a metadata file recording the hyperparameters, the target definition, the split windows, the test metrics, the paired intervals, the input SHA-256 digests and the limitation above. The artifact reproduces the manuscript's test numbers exactly (average precision 0.015903, ROC-AUC 0.880724), and tests/reference_scores.csv freezes 821 test records including all 21 positives so anyone can re-verify that claim without the study repo:

python -m pytest tests -q

Rebuilding the artifact

The artifact is exported from the study pipeline, never hand-edited:

python scripts/export_artifact.py \
    --study-repo /path/to/CVE-RiskEngine-RF \
    --out src/cve_risk_engine/artifacts
python scripts/build_reference_scores.py \
    --study-repo /path/to/CVE-RiskEngine-RF \
    --out tests/reference_scores.csv

The first script asserts that the exported model's test average precision and ROC-AUC are bit-identical to revision/outputs/results.json; if they drift, the export fails rather than shipping a lookalike.

Changelog

0.1.4 — promotes the operational band to the primary output, and marks the severity label as non-primary. No score, threshold or metric changed.

• score() now returns priority_band, priority_action, priority_score and priority_band_scale before any of the severity fields, so a caller printing the whole mapping sees the decision field first, and risk_level carries its own risk_level_note stating that it is a legacy severity label rather than an exploitation verdict.
• metadata.json gains primary_band, which names priority_band as the field a consumer should act on, and records under not_primary why risk_level must not be used to filter or sort. describe() returns it.
• README leads with priority_band, and the severity label is documented as continuity only. The reason is not stylistic: the audit and the frozen comparison protocol both found that the severity label places confirmed-exploited records in low or medium, while the operational band puts 120 of those 171 sampled records in its top two bands.
• Verified: parity with 0.1.1, 0.1.2 and 0.1.3 is unchanged at 0.0e+00 on raw, calibrated and blended scores over the frozen 400-row sample, with zero shifted severity labels.

0.1.3 — documentation only, no change to any score, threshold or metric. It records the artifact's scope inside the artifact itself.

• metadata.json gains a scope block and an extended intended_use, and describe() returns it: the engine is an early-warning and prioritization tool, not a risk register, and it is only defined for records whose exploitation outcome is still unknown. A CVE already listed in KEV is a known outcome, so scoring it is out of scope and its band must not be read as a statement about current risk.
• Two tests pin that wording, so the scope statement cannot quietly disappear.
• Verified: the package still scores identically to 0.1.1 and 0.1.2 on all 400 rows of the frozen comparison sample, largest difference 0.0e+00 on raw, calibrated and blended scores, zero rows without a match, zero shifted severity bands.

0.1.2 — adds the operational band, and closes the four findings of the independent audit of 0.1.1 plus one raised by a later regression and comparison run.

• Added priority_band, priority_score and priority_action, defined on validation quantiles of calibrated_probability. risk_level is unchanged, so existing consumers keep the same numbers, but it is now documented as the CVSS severity scale and explicitly not an exploitability verdict.
• epss_score and cvss_score are now range checked in CveRiskEngine.score. The core path previously accepted -0.1, 1.5 and nan and returned a confident looking score for all three. This was audit finding F-09.
• The OpenAPI schema version now comes from __version__ instead of a hardcoded 0.1.0, and the project name in the api.py docstring and ImportError is the real distribution name, so the instruction it prints can actually be followed. Findings F-01 and F-02.
• metadata.json now states the decision threshold on the scale it belongs to (raw_model_score 0.049528, not a calibrated probability), because reusing it on the calibrated scale returns an empty queue. Finding F-06.
• metadata.json now records the six-decimal rounding of returned scores and the consequence for anyone recomputing the published metrics from package output: ROC-AUC moves from 0.880724 to 0.881031 and the queue moves from 3,228 to 3,229. Finding F-10.
• The CLI batch output carries priority_band and priority_action; the HTTP response carries both plus priority_score.

0.1.1 — first published release: calibrated model, reference scores, CLI and optional HTTP service.

Licence

MIT. The bundled artifact is trained on derived metadata from NVD, CISA and FIRST; the upstream terms of those sources apply to the data.