# Development run results

Run date: 2026-07-15

These results validate the engineering and modeling pipeline. They are **not
population estimates**: the development extract samples inspection-table pages
and then retrieves complete histories for up to 10,000 vehicles, which
over-represents vehicles with more inspection records.

## Label audit

The raw Utah County feed leaves `overall_result` blank for 3,747,862 records.
An aggregate-only server audit found 3,469,102 Utah `obd`/`OBD` records with a
controlled pass, fail, reject, or abort value in `obd_result`. Across non-Utah
feeds where both fields were recognized, the two fields agreed on binary pass
versus non-pass 99.30% of the time, but their fail-versus-reject categories were
not interchangeable.

Label contract v3 therefore permits the Utah OBD value only as a provenance-
tagged binary proxy. TSI, `other/C`, `B`, blank, and unknown values stay
unlabeled. Four-class analysis must use `overall_result` exclusively. This also
matches the official program distinction between a readiness rejection and a
failed inspection; see the [Utah inspection requirements](https://dmv.utah.gov/register/inspections/)
and [program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).

In the development extract, the rule restored 11,334 labels and left 1,281
events unlabeled. The label-source field remains private audit metadata and is
not a model feature.

## Pipeline reconciliation

| Stage | Rows |
| --- | ---: |
| Source events written after VIN validation | 83,552 |
| Clean events after duplicate/conflict handling | 80,190 |
| Inspection episodes | 69,588 |
| Eligible returning targets | 44,659 |

The sample contains 9,996 retained vehicle tokens. The extraction manifest,
compressed-file SHA-256, mart manifest, Parquet SHA-256, and row counts all
reconcile. Every mart invariant reports zero violations. Raw VIN and raw OBD
result are absent from the staging output schema, feature mart, and model
features.

| Temporal partition | Eligible targets | Never-fit audit targets |
| --- | ---: | ---: |
| Train, 2016–2022 | 31,745 | 3,129 |
| Tune, 2023 | 4,972 | 526 |
| Calibrate, 2024 | 2,633 | 277 |
| Locked test, 2025 | 4,451 | 480 |
| Shadow, 2026 partial | 858 | 106 |

Audit-bucket vehicles are excluded from fitting, tuning, and calibration. The
normal trainers do not calculate 2025 outcomes or metrics without the explicit
locked-evaluation flag.

## Baseline results before the locked test

The table below uses only non-audit vehicles. Average precision is the project's
PR-AUC summary. The calibrated logistic row is shown only on the partition used
to fit the calibrator and is therefore a calibration diagnostic, not an
independent final estimate.

| Partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2024 calibrate | Training prevalence | 0.113 | 0.1002 | 0.500 | 0.113 |
| 2024 calibrate | Previous episode, literal | 0.148 | 0.1677 | 0.577 | 0.253 |
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |

The selected logistic regularization was `C=0.03`. It converged in 2,390 of the
5,000 allowed iterations; Platt calibration converged in five iterations. All
five candidates and all stored probabilities passed explicit convergence and
finite-value checks.

The literal previous-outcome baseline is useful as a ranking sanity check but
produces overconfident zero/one probabilities, explaining its poor Brier and log
loss. The transparent logistic model is the current development leader: on the
2023 tuning partition it more than doubles prevalence PR-AUC and raises top-10%
precision from 12.5% to 31.5%.

## One-time development holdout

The nonlinear live-verification command invoked the explicit 2025 gate after
its feature contract and four-candidate grid had already been fixed and its
candidate had been selected only on 2023. No test-driven model change was made.
At that point the development specification was frozen and the already-fixed
logistic model was evaluated once for a direct comparison. These are
development-sample holdout results, not final population claims.

| 2025 non-audit model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | ---: | ---: | ---: | ---: |
| Training prevalence | 0.123 | 0.1076 | 0.500 | 0.123 |
| Previous episode, literal | 0.158 | 0.1731 | 0.576 | 0.268 |
| Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
| Histogram gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |

The holdout contains 3,971 non-audit episodes from 3,734 vehicles. The
never-fit audit contains another 480 episodes from 450 vehicles; calibrated
logistic PR-AUC is 0.253 and Brier is 0.0859 there. The logistic model beats
both simple baselines on the project's headline metrics and remains the chosen
development model. The tree did not provide a decisive pre-test improvement
that justified its added complexity, and no further 2025-informed tuning is
permitted.

## Remaining gates

1. Run a complete, contiguous bounded extraction for publishable population
   aggregates; the page-sampled development cohort cannot support dashboard
   prevalence or county rankings.
2. Treat any later full-data 2025 result as confirmatory rather than a pristine
   unseen test, because the development sample's holdout has now been opened.
3. Run the frozen subgroup/source-era report and clustered uncertainty
   analysis on the complete extraction.
4. Replace the checked-in development-preview bundle with complete-data
   aggregates only after the population-publication review passes.

## Sanitized dashboard preview

The repository now includes a fail-closed exporter and a static four-view
dashboard shell. The preview exporter reads only train, tune, and calibration
rows dated before 2025. It rejects any model manifest that says the holdout was
evaluated, rejects any metric row outside the three approved partitions, and
marks every asset `development_preview=true` and
`population_estimate_allowed=false`.

Published cells must clear minimum episode and distinct-vehicle thresholds,
including both binary-class complements. Supports are rounded, direct and
pseudonymous identifiers stay private, and a checksum manifest covers the
approved JSON bundle. Private model manifests bind the exact metrics files, and
the public manifest exposes a deterministic release ID plus model versions for
provenance. The site disables the estimator because no privacy-reviewed
prediction lookup exists.