diff --git a/presentation/index.html b/presentation/index.html index 2640ed4..c752f71 100644 --- a/presentation/index.html +++ b/presentation/index.html @@ -661,6 +661,80 @@ cursor: pointer; } + /* ---------- Lift callout ---------- */ + .lift { + margin: 0.9rem 0 0.5rem; + padding: 0.8rem 1.15rem; + max-width: 60ch; + background: var(--teal-100); + border-left: 4px solid var(--teal-600); + border-radius: var(--radius-sm); + color: var(--teal-800); + font-size: 0.94em; + line-height: 1.7; + } + + .lift b { + font-family: var(--font-display); + font-size: 1.3em; + font-weight: 600; + letter-spacing: -0.01em; + } + + /* ---------- Calibration plot ---------- */ + .calib { + display: block; + width: 100%; + max-width: 28rem; + height: auto; + } + + .calib__grid line { + stroke: var(--sand-200); + stroke-width: 1; + } + + .calib__ideal { + stroke: var(--ink-500); + stroke-width: 1.5; + stroke-dasharray: 6 5; + } + + .calib__trace { + fill: none; + stroke: var(--teal-600); + stroke-width: 2; + stroke-linejoin: round; + opacity: 0.55; + } + + .calib__dot { + fill: var(--teal-700); + stroke: var(--white); + stroke-width: 1.5; + } + + .calib__tick { + font-family: var(--font-sans); + font-size: 13px; + fill: var(--ink-500); + font-variant-numeric: tabular-nums; + } + + .calib__axis { + font-family: var(--font-sans); + font-size: 13px; + font-weight: 600; + fill: var(--ink-650); + } + + .calib__note { + font-family: var(--font-sans); + font-size: 12.5px; + font-style: italic; + fill: var(--ink-500); + } + /* ---------- Print / PDF backup ---------- */ @media print { body { @@ -839,13 +913,14 @@ build 30-day episodes

Model comparison

@@ -882,6 +957,11 @@ build 30-day episodes
- 2025 one-time sample holdout · higher PR-AUC and lower Brier are - better · not population performance + 2025 one-time sample holdout · 3,971 non-audit episodes from 3,734 + vehicles, 487 non-pass · higher PR-AUC and lower Brier are better · + not population performance
+

+ Ranked by score, the top 10% of episodes are non-pass 31.2% of + the time against a 12.3% base rate — about + 2.5× lift, roughly 1 in 3 instead of 1 in 8. +

Logistic beat both baselines and the boosted-tree benchmark. We avoid the word “accuracy,” which hides class imbalance and calibration. @@ -892,7 +972,7 @@ build 30-day episodes

Model decision

Keep calibrated logistic regression

@@ -905,6 +985,11 @@ build 30-day episodes Benchmark only: histogram gradient boosting — not a second final model. +
  • + Selected on 2023 tune, where logistic led + (0.282 vs 0.274 PR-AUC). 2024 only fits + Platt scaling, so it cannot select; 2025 confirmed the choice. +
  • The tree did not improve the declared sample metrics enough to justify its complexity. @@ -913,6 +998,88 @@ build 30-day episodes
  • + +
    +

    Calibration

    +

    The probabilities mean what they say

    +
    +
    + + Reliability of the calibrated logistic model on the 2025 one-time sample holdout + Predicted non-pass probability by decile plotted against the observed non-pass rate. Points track the diagonal from about 0.05 to about 0.35. + + + + + + + + + + + + + + + + + + + + + + + + + + 0.0 + 0.0 + 0.1 + 0.1 + 0.2 + 0.2 + 0.3 + 0.3 + 0.4 + 0.4 + + Predicted probability (decile mean) + Observed non-pass rate + perfect calibration + +
    +
    +
      +
    • + Each dot is one decile of the 2025 holdout — + predicted probability across, observed non-pass rate up. +
    • +
    • + The dashed line is perfect calibration. Deciles + rise monotonically and track it across the range. +
    • +
    • + Lowest decile: predicted 0.047, observed + 0.040. Highest: predicted 0.348, + observed 0.312. +
    • +
    • + This is what the Brier score of 0.1011 + summarizes in a single number. +
    • +
    +

    + Mid-range bins wobble — each holds roughly 400 episodes, so a few + outcomes move a point. +

    +
    +
    +
    +
    -
    1 / 10
    +
    1 / 11