- 2025 one-time sample holdout · higher PR-AUC and lower Brier are
- better · not population performance
+ 2025 one-time sample holdout · 3,971 non-audit episodes from 3,734
+ vehicles, 487 non-pass · higher PR-AUC and lower Brier are better ·
+ not population performance
@@ -882,6 +957,11 @@ build 30-day episodes
+
+ Ranked by score, the top 10% of episodes are non-pass 31.2% of
+ the time against a 12.3% base rate — about
+ 2.5× lift, roughly 1 in 3 instead of 1 in 8.
+
Logistic beat both baselines and the boosted-tree benchmark. We avoid
the word “accuracy,” which hides class imbalance and calibration.
@@ -892,7 +972,7 @@ build 30-day episodes
Model decision
Keep calibrated logistic regression
@@ -905,6 +985,11 @@ build 30-day episodes
Benchmark only: histogram gradient boosting — not a
second final model.
+
+ Selected on 2023 tune, where logistic led
+ (0.282 vs 0.274 PR-AUC). 2024 only fits
+ Platt scaling, so it cannot select; 2025 confirmed the choice.
+
The tree did not improve the declared sample metrics enough to
justify its complexity.
@@ -913,6 +998,88 @@ build 30-day episodes
+
+
+
Calibration
+
The probabilities mean what they say
+
+
+
+
+
+
+
+ Each dot is one decile of the 2025 holdout —
+ predicted probability across, observed non-pass rate up.
+
+
+ The dashed line is perfect calibration. Deciles
+ rise monotonically and track it across the range.
+