# Development run results

Run date: 2026-07-15; presentation wording reviewed 2026-07-21

> **Scope of every value below:** the private, page-sampled 10,000-vehicle
> development cohort. These are sample results, not population estimates. Page
> sampling over-represents vehicles with more inspection records. Nothing in
> this document supports statewide/county prevalence or county rankings.

## Label audit

The binary target is pass versus non-pass, where non-pass combines recognized
fail, reject, and abort outcomes. Blanks and unrecognized values are unlabeled,
not passes.

For the older Utah County feed, the documented source/program/test rule permits
a controlled OBD result to fill a blank overall result as a binary proxy. It
does not support a four-class interpretation. In the private 10,000-vehicle
sample, the rule restored 11,334 labels and left 1,281 events unlabeled. These
sample counts are not population estimates. Label provenance stays in private
audit metadata and is not a predictor.

## Pipeline reconciliation

This table reports private 10,000-vehicle development-sample pipeline counts,
not population totals.

| Sample pipeline stage | Rows |
| --- | ---: |
| Source events written after identifier validation | 83,552 |
| Clean events after duplicate/conflict handling | 80,190 |
| Inspection episodes | 69,588 |
| Eligible returning-vehicle targets | 44,659 |

The sample retained 9,996 private vehicle tokens. Extraction and mart manifests,
compressed-file and Parquet checksums, and stage counts reconcile. Every mart
invariant reports zero violations. Raw identifiers and raw OBD values are absent
from staging output, feature mart predictors, and model features.

The following partition counts are also from the private 10,000-vehicle
development sample and are not population estimates.

| Sample temporal partition | Eligible targets | Never-fit audit targets |
| --- | ---: | ---: |
| Train, 2016-2022 | 31,745 | 3,129 |
| Tune, 2023 | 4,972 | 526 |
| Calibrate, 2024 | 2,633 | 277 |
| One-time holdout, 2025 | 4,451 | 480 |
| Shadow, 2026 partial | 858 | 106 |

Audit-bucket vehicles are excluded from fitting, tuning, and calibration. Normal
training does not calculate 2025 outcomes or metrics without the explicit
locked-evaluation flag.

## Pre-2025 development diagnostics

The table uses non-audit rows from the private 10,000-vehicle development
sample. Average precision is the project's PR-AUC summary. The calibrated 2024
row is evaluated on the same sample partition used to fit Platt scaling, so it
is a calibration diagnostic rather than independent final performance. None of
these values is a population estimate.

| Sample partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2024 calibrate | Training prevalence | 0.113 | 0.1002 | 0.500 | 0.113 |
| 2024 calibrate | Previous episode, literal | 0.148 | 0.1677 | 0.577 | 0.253 |
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |

The selected logistic regularization is `C=0.03`. It converged in 2,390 of the
5,000 allowed iterations; Platt calibration converged in five iterations. These
are development-run diagnostics. All candidates and stored probabilities passed
explicit convergence and finite-value checks.

On the 2023 sample partition, logistic regression more than doubled prevalence
PR-AUC and raised top-10% precision from 12.5% to 31.5%. This describes the
private development sample only.

## One-time development-sample holdout

The nonlinear verification command invoked the explicit 2025 gate only after
the feature contract and benchmark grid had been frozen and candidate choices
had been made from pre-2025 sample data. No model was changed in response. These
are one-time development-sample holdout results, not pristine future-test or
population estimates.

| 2025 private-sample model | Role | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | --- | ---: | ---: | ---: | ---: |
| Training prevalence | Baseline | 0.123 | 0.1076 | 0.500 | 0.123 |
| Previous episode, literal | Baseline | 0.158 | 0.1731 | 0.576 | 0.268 |
| Logistic + Platt | **Final model** | **0.261** | **0.1011** | **0.693** | **0.312** |
| Histogram gradient boosting + Platt | Benchmark only | 0.238 | 0.1024 | 0.690 | 0.292 |

The sample holdout contains 3,971 non-audit episodes from 3,734 vehicles. Its
never-fit audit adds 480 episodes from 450 vehicles; calibrated logistic PR-AUC
is 0.253 and Brier score is 0.0859 there. All are private-sample diagnostics,
not population estimates or external validation.

The calibrated logistic regression remains the final development model. It
beats both simple baselines on the headline metrics and beats the boosted-tree
benchmark in the one-time comparison. The benchmark did not justify its added
complexity. No further 2025-informed tuning is permitted.

## Sanitized dashboard preview

The static preview uses only suppression-cleared sample aggregates and
development diagnostics dated before 2025. It intentionally excludes the
one-time holdout metrics. Every asset declares `development_preview=true` and
`population_estimate_allowed=false`; the manifest declares
`locked_test_metrics_published=false`.

Published cells must clear minimum episode, distinct-vehicle, and binary-class
thresholds. Supports are rounded, suppressed rows are omitted, and a checksum
manifest covers the approved JSON bundle. The site fails closed if a contract
check fails and contains no vehicle-level prediction service.

No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
credential, operational record, or row-level prediction is in the public
bundle.

## Boundary of the finished prototype

The repository completes a presentation-ready development prototype, not a
population study or production service. A complete population-frame extraction,
fresh external/future validation, individualized prediction lookup, DMV
enrichment, multiclass modeling, and operational monitoring are outside this
prototype's deliverables. They would require separate approval, analysis, and
privacy review; they are not implied by the current results.