Add calibration slide and sharpen model-comparison evidence in deck #2

2026-08-27T22:59:39Z

kevinbell commented

2026-08-27 22:59:39 +00:00

Three changes to presentation/index.html ahead of tomorrow's talk. Every value is sourced from artifacts already in the repo — no new analysis, no re-running of the model, and the 2025 gate stays closed.

Slide 5 — model comparison

The caption now states what the table was evaluated on: 3,971 non-audit episodes from 3,734 vehicles, 487 non-pass. A callout under the table adds the practical read:

Ranked by score, the top 10% of episodes are non-pass 31.2% of the time against a 12.3% base rate — about 2.5× lift, roughly 1 in 3 instead of 1 in 8.

PR-AUC and Brier alone gave the audience no way to judge whether 0.261 is good, and the table gave no N.

Slide 6 — model decision

Adds one bullet recording selection provenance:

Selected on 2023 tune, where logistic led (0.282 vs 0.274 PR-AUC). 2024 only fits Platt scaling, so it cannot select; 2025 confirmed the choice.

This matters for the live demo. The dashboard's Model & benchmark page shows the 2024 calibrate partition, where the boosted tree leads on both metrics (PR-AUC 0.275 vs 0.257, Brier 0.0928 vs 0.0936). Without this line the deck asserts the tree "did not improve the declared sample metrics" and then puts contradicting numbers on screen three minutes later. The speaker note spells out the rebuttal: 2024 is a calibration-fit partition, so selecting on it would be circular.

New slide 7 — calibration

Inline SVG reliability plot of the 2025 holdout deciles against the perfect-calibration diagonal, plotted from artifacts/private/holdout/baseline_v1/calibration_bins.csv. Deciles rise monotonically and track the line: lowest predicted 0.047 / observed 0.040, highest predicted 0.348 / observed 0.312.

"Calibrated" appears on four slides and in the close, but calibration was never shown anywhere — the dashboard has no reliability chart either. The top decile's observed rate is arithmetically the same quantity as the top-10% precision on slide 5, so the two slides reinforce each other.

Verification

All numeric claims re-checked against artifacts/private/holdout/*/metrics.json and calibration_bins.csv; the 10 rendered dot coordinates diff exactly against the CSV.
Deck is 11 slides, tag balance clean, counter seeds correctly.
Guardrail band added to the calibration slide, per the presentation rule that every results slide carries it.
P still exports a working 11-page PDF backup.
No private data enters the deck: the plot uses aggregate decile rates only, no row-level values or identifiers.

Not included

Backup screenshot slides for demo failure — these need captures of the running dashboard.
docs/presentation_outline.md now drifts from the deck (it describes 9 slides plus the demo, with timings the deck no longer carries). Happy to reconcile it in a follow-up if that doc is a graded artifact.

Three changes to `presentation/index.html` ahead of tomorrow's talk. Every value is sourced from artifacts already in the repo — no new analysis, no re-running of the model, and the 2025 gate stays closed. ## Slide 5 — model comparison The caption now states what the table was evaluated on: **3,971 non-audit episodes from 3,734 vehicles, 487 non-pass**. A callout under the table adds the practical read: > Ranked by score, the top 10% of episodes are non-pass **31.2%** of the time against a **12.3%** base rate — about **2.5× lift**, roughly 1 in 3 instead of 1 in 8. PR-AUC and Brier alone gave the audience no way to judge whether 0.261 is good, and the table gave no N. ## Slide 6 — model decision Adds one bullet recording selection provenance: > **Selected on 2023 tune**, where logistic led (`0.282` vs `0.274` PR-AUC). 2024 only fits Platt scaling, so it cannot select; 2025 confirmed the choice. This matters for the live demo. The dashboard's Model & benchmark page shows the 2024 calibrate partition, where the boosted tree leads on *both* metrics (PR-AUC 0.275 vs 0.257, Brier 0.0928 vs 0.0936). Without this line the deck asserts the tree "did not improve the declared sample metrics" and then puts contradicting numbers on screen three minutes later. The speaker note spells out the rebuttal: 2024 is a calibration-fit partition, so selecting on it would be circular. ## New slide 7 — calibration Inline SVG reliability plot of the 2025 holdout deciles against the perfect-calibration diagonal, plotted from `artifacts/private/holdout/baseline_v1/calibration_bins.csv`. Deciles rise monotonically and track the line: lowest predicted 0.047 / observed 0.040, highest predicted 0.348 / observed 0.312. "Calibrated" appears on four slides and in the close, but calibration was never shown anywhere — the dashboard has no reliability chart either. The top decile's observed rate is arithmetically the same quantity as the top-10% precision on slide 5, so the two slides reinforce each other. ## Verification - All numeric claims re-checked against `artifacts/private/holdout/*/metrics.json` and `calibration_bins.csv`; the 10 rendered dot coordinates diff exactly against the CSV. - Deck is 11 slides, tag balance clean, counter seeds correctly. - Guardrail band added to the calibration slide, per the presentation rule that every results slide carries it. - `P` still exports a working 11-page PDF backup. - No private data enters the deck: the plot uses aggregate decile rates only, no row-level values or identifiers. ## Not included - Backup screenshot slides for demo failure — these need captures of the running dashboard. - `docs/presentation_outline.md` now drifts from the deck (it describes 9 slides plus the demo, with timings the deck no longer carries). Happy to reconcile it in a follow-up if that doc is a graded artifact.

kevinbell added 1 commit 2026-08-27 22:59:40 +00:00

Add calibration slide and sharpen model-comparison evidence in deck 180d58e646

Three changes to the presentation, all sourced from existing artifacts:

Slide 5 (model comparison): caption now states the evaluation set
(3,971 non-audit episodes from 3,734 vehicles, 487 non-pass), and a
callout reports top-10% precision of 31.2% against the 12.3% base
rate — about 2.5x lift. The two headline metrics alone gave the
audience no way to judge whether 0.261 PR-AUC is good.

Slide 6 (model decision): record that candidate selection happened on
the 2023 tune partition, where logistic led 0.282 to 0.274, and that
2024 fits Platt scaling only and so cannot select. The dashboard demo
displays 2024 numbers where the tree leads on both metrics; without
this line the deck has no answer to the obvious question.

New slide 7 (calibration): reliability plot of the 2025 holdout
deciles against the perfect-calibration diagonal, plotted from
artifacts/private/holdout/baseline_v1/calibration_bins.csv. Nothing in
the deck or dashboard previously showed calibration, though it is the
centerpiece claim of the final model.

Values verified against artifacts/private/holdout/*/metrics.json and
calibration_bins.csv. Deck remains self-contained; print/PDF backup
exports 11 pages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

kevinbell merged commit c6b36fa235 into main

2026-08-27 23:00:44 +00:00

kevinbell deleted branch deck-calibration-slide

2026-08-27 23:00:44 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Centralized_I-M_Database/SummerProject2026#2