6.7 KiB
Development run results
Run date: 2026-07-15
These results validate the engineering and modeling pipeline. They are not population estimates: the development extract samples inspection-table pages and then retrieves complete histories for up to 10,000 vehicles, which over-represents vehicles with more inspection records.
Label audit
The raw Utah County feed leaves overall_result blank for 3,747,862 records.
An aggregate-only server audit found 3,469,102 Utah obd/OBD records with a
controlled pass, fail, reject, or abort value in obd_result. Across non-Utah
feeds where both fields were recognized, the two fields agreed on binary pass
versus non-pass 99.30% of the time, but their fail-versus-reject categories were
not interchangeable.
Label contract v3 therefore permits the Utah OBD value only as a provenance-
tagged binary proxy. TSI, other/C, B, blank, and unknown values stay
unlabeled. Four-class analysis must use overall_result exclusively. This also
matches the official program distinction between a readiness rejection and a
failed inspection; see the Utah inspection requirements
and program definition of rejection.
In the development extract, the rule restored 11,334 labels and left 1,281 events unlabeled. The label-source field remains private audit metadata and is not a model feature.
Pipeline reconciliation
| Stage | Rows |
|---|---|
| Source events written after VIN validation | 83,552 |
| Clean events after duplicate/conflict handling | 80,190 |
| Inspection episodes | 69,588 |
| Eligible returning targets | 44,659 |
The sample contains 9,996 retained vehicle tokens. The extraction manifest, compressed-file SHA-256, mart manifest, Parquet SHA-256, and row counts all reconcile. Every mart invariant reports zero violations. Raw VIN and raw OBD result are absent from the staging output schema, feature mart, and model features.
| Temporal partition | Eligible targets | Never-fit audit targets |
|---|---|---|
| Train, 2016–2022 | 31,745 | 3,129 |
| Tune, 2023 | 4,972 | 526 |
| Calibrate, 2024 | 2,633 | 277 |
| Locked test, 2025 | 4,451 | 480 |
| Shadow, 2026 partial | 858 | 106 |
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. The normal trainers do not calculate 2025 outcomes or metrics without the explicit locked-evaluation flag.
Baseline results before the locked test
The table below uses only non-audit vehicles. Average precision is the project's PR-AUC summary. The calibrated logistic row is shown only on the partition used to fit the calibrator and is therefore a calibration diagnostic, not an independent final estimate.
| Partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|---|---|---|---|---|---|
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic | 0.282 | 0.1012 | 0.707 | 0.315 |
| 2024 calibrate | Training prevalence | 0.113 | 0.1002 | 0.500 | 0.113 |
| 2024 calibrate | Previous episode, literal | 0.148 | 0.1677 | 0.577 | 0.253 |
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
| 2024 calibrate | Logistic + Platt | 0.257 | 0.0936 | 0.690 | 0.308 |
The selected logistic regularization was C=0.03. It converged in 2,390 of the
5,000 allowed iterations; Platt calibration converged in five iterations. All
five candidates and all stored probabilities passed explicit convergence and
finite-value checks.
The literal previous-outcome baseline is useful as a ranking sanity check but produces overconfident zero/one probabilities, explaining its poor Brier and log loss. The transparent logistic model is the current development leader: on the 2023 tuning partition it more than doubles prevalence PR-AUC and raises top-10% precision from 12.5% to 31.5%.
One-time development holdout
The nonlinear live-verification command invoked the explicit 2025 gate after its feature contract and four-candidate grid had already been fixed and its candidate had been selected only on 2023. No test-driven model change was made. At that point the development specification was frozen and the already-fixed logistic model was evaluated once for a direct comparison. These are development-sample holdout results, not final population claims.
| 2025 non-audit model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|---|---|---|---|---|
| Training prevalence | 0.123 | 0.1076 | 0.500 | 0.123 |
| Previous episode, literal | 0.158 | 0.1731 | 0.576 | 0.268 |
| Logistic + Platt | 0.261 | 0.1011 | 0.693 | 0.312 |
| Histogram gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
The holdout contains 3,971 non-audit episodes from 3,734 vehicles. The never-fit audit contains another 480 episodes from 450 vehicles; calibrated logistic PR-AUC is 0.253 and Brier is 0.0859 there. The logistic model beats both simple baselines on the project's headline metrics and remains the chosen development model. The tree did not provide a decisive pre-test improvement that justified its added complexity, and no further 2025-informed tuning is permitted.
Remaining gates
- Run a complete, contiguous bounded extraction for publishable population aggregates; the page-sampled development cohort cannot support dashboard prevalence or county rankings.
- Treat any later full-data 2025 result as confirmatory rather than a pristine unseen test, because the development sample's holdout has now been opened.
- Run the frozen subgroup/source-era report and clustered uncertainty analysis on the complete extraction.
- Replace the checked-in development-preview bundle with complete-data aggregates only after the population-publication review passes.
Sanitized dashboard preview
The repository now includes a fail-closed exporter and a static four-view
dashboard shell. The preview exporter reads only train, tune, and calibration
rows dated before 2025. It rejects any model manifest that says the holdout was
evaluated, rejects any metric row outside the three approved partitions, and
marks every asset development_preview=true and
population_estimate_allowed=false.
Published cells must clear minimum episode and distinct-vehicle thresholds, including both binary-class complements. Supports are rounded, direct and pseudonymous identifiers stay private, and a checksum manifest covers the approved JSON bundle. Private model manifests bind the exact metrics files, and the public manifest exposes a deterministic release ID plus model versions for provenance. The site disables the estimator because no privacy-reviewed prediction lookup exists.