From 180d58e64617be34861fb094f60ca4e0c0e68941 Mon Sep 17 00:00:00 2001 From: kevinBell Date: Thu, 27 Aug 2026 16:58:38 -0600 Subject: [PATCH] Add calibration slide and sharpen model-comparison evidence in deck MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes to the presentation, all sourced from existing artifacts: Slide 5 (model comparison): caption now states the evaluation set (3,971 non-audit episodes from 3,734 vehicles, 487 non-pass), and a callout reports top-10% precision of 31.2% against the 12.3% base rate — about 2.5x lift. The two headline metrics alone gave the audience no way to judge whether 0.261 PR-AUC is good. Slide 6 (model decision): record that candidate selection happened on the 2023 tune partition, where logistic led 0.282 to 0.274, and that 2024 fits Platt scaling only and so cannot select. The dashboard demo displays 2024 numbers where the tree leads on both metrics; without this line the deck has no answer to the obvious question. New slide 7 (calibration): reliability plot of the 2025 holdout deciles against the perfect-calibration diagonal, plotted from artifacts/private/holdout/baseline_v1/calibration_bins.csv. Nothing in the deck or dashboard previously showed calibration, though it is the centerpiece claim of the final model. Values verified against artifacts/private/holdout/*/metrics.json and calibration_bins.csv. Deck remains self-contained; print/PDF backup exports 11 pages. Co-Authored-By: Claude Opus 5 --- presentation/index.html | 177 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 172 insertions(+), 5 deletions(-) diff --git a/presentation/index.html b/presentation/index.html index 2640ed4..c752f71 100644 --- a/presentation/index.html +++ b/presentation/index.html @@ -661,6 +661,80 @@ cursor: pointer; } + /* ---------- Lift callout ---------- */ + .lift { + margin: 0.9rem 0 0.5rem; + padding: 0.8rem 1.15rem; + max-width: 60ch; + background: var(--teal-100); + border-left: 4px solid var(--teal-600); + border-radius: var(--radius-sm); + color: var(--teal-800); + font-size: 0.94em; + line-height: 1.7; + } + + .lift b { + font-family: var(--font-display); + font-size: 1.3em; + font-weight: 600; + letter-spacing: -0.01em; + } + + /* ---------- Calibration plot ---------- */ + .calib { + display: block; + width: 100%; + max-width: 28rem; + height: auto; + } + + .calib__grid line { + stroke: var(--sand-200); + stroke-width: 1; + } + + .calib__ideal { + stroke: var(--ink-500); + stroke-width: 1.5; + stroke-dasharray: 6 5; + } + + .calib__trace { + fill: none; + stroke: var(--teal-600); + stroke-width: 2; + stroke-linejoin: round; + opacity: 0.55; + } + + .calib__dot { + fill: var(--teal-700); + stroke: var(--white); + stroke-width: 1.5; + } + + .calib__tick { + font-family: var(--font-sans); + font-size: 13px; + fill: var(--ink-500); + font-variant-numeric: tabular-nums; + } + + .calib__axis { + font-family: var(--font-sans); + font-size: 13px; + font-weight: 600; + fill: var(--ink-650); + } + + .calib__note { + font-family: var(--font-sans); + font-size: 12.5px; + font-style: italic; + fill: var(--ink-500); + } + /* ---------- Print / PDF backup ---------- */ @media print { body { @@ -839,13 +913,14 @@ build 30-day episodes

Model comparison

@@ -882,6 +957,11 @@ build 30-day episodes
- 2025 one-time sample holdout · higher PR-AUC and lower Brier are - better · not population performance + 2025 one-time sample holdout · 3,971 non-audit episodes from 3,734 + vehicles, 487 non-pass · higher PR-AUC and lower Brier are better · + not population performance
+

+ Ranked by score, the top 10% of episodes are non-pass 31.2% of + the time against a 12.3% base rate — about + 2.5× lift, roughly 1 in 3 instead of 1 in 8. +

Logistic beat both baselines and the boosted-tree benchmark. We avoid the word “accuracy,” which hides class imbalance and calibration. @@ -892,7 +972,7 @@ build 30-day episodes

Model decision

Keep calibrated logistic regression

@@ -905,6 +985,11 @@ build 30-day episodes Benchmark only: histogram gradient boosting — not a second final model. +
  • + Selected on 2023 tune, where logistic led + (0.282 vs 0.274 PR-AUC). 2024 only fits + Platt scaling, so it cannot select; 2025 confirmed the choice. +
  • The tree did not improve the declared sample metrics enough to justify its complexity. @@ -913,6 +998,88 @@ build 30-day episodes
    • + +
    +

    Calibration

    +

    The probabilities mean what they say

    +
    +
    + + Reliability of the calibrated logistic model on the 2025 one-time sample holdout + Predicted non-pass probability by decile plotted against the observed non-pass rate. Points track the diagonal from about 0.05 to about 0.35. + + + + + + + + + + + + + + + + + + + + + + + + + + 0.0 + 0.0 + 0.1 + 0.1 + 0.2 + 0.2 + 0.3 + 0.3 + 0.4 + 0.4 + + Predicted probability (decile mean) + Observed non-pass rate + perfect calibration + +
    +
    +
      +
    • + Each dot is one decile of the 2025 holdout — + predicted probability across, observed non-pass rate up. +
      • +
    • + The dashed line is perfect calibration. Deciles + rise monotonically and track it across the range. +
      • +
    • + Lowest decile: predicted 0.047, observed + 0.040. Highest: predicted 0.348, + observed 0.312. +
      • +
    • + This is what the Brier score of 0.1011 + summarizes in a single number. +
      • +
    +

    + Mid-range bins wobble — each holds roughly 400 episodes, so a few + outcomes move a point. +

    +
    +
    +
    +
    -
    1 / 10
    +
    1 / 11