226 lines
10 KiB
Markdown
226 lines
10 KiB
Markdown
# Utah Vehicle Health: final development-prototype report
|
|
|
|
Prepared July 21, 2026
|
|
|
|
> **Result scope:** Every analytical count, rate, chart, and model metric in this
|
|
> report comes from the private, page-sampled 10,000-vehicle development cohort.
|
|
> These are sample results, not population estimates. The cohort cannot support
|
|
> statewide or county prevalence, county rankings, causal conclusions, or
|
|
> decisions about an individual vehicle.
|
|
|
|
## Executive summary
|
|
|
|
Utah Vehicle Health asks one focused question:
|
|
|
|
> Using only information available before an inspection episode begins, how
|
|
> well can a calibrated logistic regression estimate whether a returning
|
|
> vehicle's next episode will have a first-attempt non-pass outcome?
|
|
|
|
The finished prototype builds leakage-safe inspection episodes, creates
|
|
point-in-time history features, evaluates models chronologically, and publishes
|
|
only suppression-reviewed sample aggregates to a static dashboard. Non-pass
|
|
combines recognized fail, reject, and abort outcomes; blank or unrecognized
|
|
results are not treated as passes.
|
|
|
|
The final model is regularized logistic regression with Platt calibration. On
|
|
the private sample's one-time 2025 holdout, it achieved PR-AUC 0.261 and Brier
|
|
score 0.1011. The calibrated histogram gradient-boosted benchmark achieved
|
|
PR-AUC 0.238 and Brier score 0.1024 on the same sample holdout. Logistic
|
|
regression remains the final model because it was both more transparent and
|
|
better on the declared headline metrics in this development comparison.
|
|
|
|
Those values demonstrate that the pipeline and model comparison work on this
|
|
development sample. They do not establish population performance, production
|
|
readiness, or external validity.
|
|
|
|
## Research scope
|
|
|
|
The prediction unit is the first attempt of the next inspection episode for a
|
|
returning vehicle. Consecutive attempts no more than 30 days apart form one
|
|
episode. This prevents a fail followed quickly by a retest from becoming two
|
|
nominally independent prediction targets. A vehicle is eligible only after a
|
|
prior episode has been observed.
|
|
|
|
The binary outcome is:
|
|
|
|
- pass; or
|
|
- non-pass, combining fail, reject, and abort.
|
|
|
|
Reject and abort can reflect process or readiness conditions, so this report
|
|
does not equate non-pass with mechanical failure. Cold-start prediction,
|
|
four-class modeling, fail-only sensitivity analysis, DMV enrichment,
|
|
station-level analysis, and individual prediction are outside the completed
|
|
prototype.
|
|
|
|
## Development data
|
|
|
|
The private extraction samples inspection-table pages, selects up to 10,000
|
|
vehicles, and then retrieves their complete available inspection histories. It
|
|
retained 9,996 private vehicle tokens after validation. The resulting sample
|
|
contains 83,552 source events, 69,588 episodes, and 44,659 eligible returning-
|
|
vehicle targets. These figures are pipeline-reconciliation counts for the
|
|
private sample, not estimates of Utah inspection volume.
|
|
|
|
The sample design intentionally makes end-to-end development manageable, but it
|
|
over-represents vehicles with more inspection records. Participating feeds also
|
|
change over time and do not cover all 29 Utah counties. The dashboard therefore
|
|
labels every value as a development-sample aggregate and uses county only to
|
|
communicate covered-feed context—not to publish population rankings.
|
|
|
|
The extraction and mart manifests bind hashes, configurations, and row counts.
|
|
Private vehicle linkage uses a one-way keyed token. Direct identifiers and raw
|
|
source payloads remain outside the analytical exports and public site.
|
|
|
|
## Leakage-safe method
|
|
|
|
Every feature exists before the target episode begins. The final model uses
|
|
vehicle age, prior episode and attempt counts, time since prior history, prior
|
|
non-pass rate, prior first/final outcomes, previously observed make/model,
|
|
public county context, and season.
|
|
|
|
The model excludes the target attempt's outcome and diagnostics, later attempts
|
|
from the target episode, station or technician information, future records,
|
|
full-history leakage, direct or pseudonymous identifiers, and preprocessing
|
|
learned from later partitions. DMV enrichment and rich current-test OBD fields
|
|
are not part of the final model.
|
|
|
|
The chronology is fixed:
|
|
|
|
| Period | Role in the private 10,000-vehicle development sample |
|
|
| --- | --- |
|
|
| 2010-2015 | Historical context only |
|
|
| 2016-2022 | Fit preprocessing and model parameters |
|
|
| 2023 | Select logistic regularization and compare fixed candidates |
|
|
| 2024 | Fit Platt calibration and inspect calibration behavior |
|
|
| 2025 | One-time development-sample holdout |
|
|
| 2026 partial | Shadow context only |
|
|
|
|
Random row splitting is prohibited. A separate set of private vehicle buckets
|
|
is never used for fitting and serves as a stress test.
|
|
|
|
## Model comparison and decision
|
|
|
|
The declared comparison includes training prevalence, the literal previous-
|
|
episode outcome, regularized logistic regression, and histogram gradient
|
|
boosting. Logistic regression uses `C=0.03`; Platt scaling is fit on the 2024
|
|
sample partition. The tree exists only to test whether nonlinearity provides a
|
|
meaningful development gain.
|
|
|
|
The table reports non-audit rows from the private, page-sampled 10,000-vehicle
|
|
development cohort. It is not population performance.
|
|
|
|
| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
|
|
| --- | --- | ---: | ---: | ---: | ---: |
|
|
| 2023 tune | Prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
|
|
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
|
|
| 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
|
|
| 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
|
|
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
|
|
|
|
On the 2023 sample partition, logistic regression more than doubled the
|
|
prevalence baseline's PR-AUC and raised precision among the highest-scored 10%
|
|
from 12.5% to 31.5%. On the one-time 2025 sample holdout, calibrated logistic
|
|
also exceeded the simple baselines and the boosted-tree benchmark on PR-AUC and
|
|
Brier score.
|
|
|
|
The explicit 2025 gate was opened once after model specifications were frozen
|
|
from pre-2025 data. No model was changed in response. That evaluation is a
|
|
development holdout, not a pristine future test, and no further 2025-informed
|
|
tuning is allowed.
|
|
|
|
The final decision is therefore straightforward: keep calibrated logistic
|
|
regression. It satisfies the development objective with a simpler, more
|
|
explainable form, while the nonlinear benchmark did not show a gain that
|
|
justified extra complexity.
|
|
|
|
## Static dashboard
|
|
|
|
The presentation-ready dashboard is a dependency-free static site with four
|
|
views:
|
|
|
|
1. **Overview** shows rounded support, sample outcome patterns, vehicle-age
|
|
patterns, and county feed coverage.
|
|
2. **Sample cohorts** shows only supported make/model sample aggregates;
|
|
selected source abbreviations and trim labels are combined for new model
|
|
families while earlier published cohorts keep stable definitions. It does
|
|
not issue reliability ratings or recommendations.
|
|
3. **Model & benchmark** identifies calibrated logistic regression as final and
|
|
the boosted tree as benchmark-only. It shows pre-2025, 2024 calibration-fit
|
|
checks—not the opened 2025 holdout comparison.
|
|
4. **Data & methods** explains the target, chronology, evidence status,
|
|
limitations, and privacy boundary.
|
|
|
|
The finished dashboard contains no vehicle lookup, personal input form, or
|
|
row-level prediction service.
|
|
|
|
The dashboard loads purpose-built JSON rather than private analytical rows. It
|
|
checks schemas, cross-file consistency, publication flags, and checksums. If the
|
|
bundle fails validation, the interface shows an unavailable state instead of
|
|
fallback estimates.
|
|
|
|
Every published cell requires minimum support for episodes, distinct vehicles,
|
|
and both binary outcomes. Supports are rounded and suppressed rows are omitted
|
|
entirely. The checked-in assets use only pre-2025 sample aggregates and
|
|
diagnostics; the one-time 2025 values in this report are not in the browser
|
|
bundle.
|
|
|
|
## Privacy and release boundary
|
|
|
|
The public boundary is intentionally narrow:
|
|
|
|
```text
|
|
private read-only source
|
|
-> private local extraction and model pipeline
|
|
-> suppression-reviewed aggregate JSON
|
|
-> dashboard-only static Bolt project
|
|
```
|
|
|
|
The public dashboard and presentation materials contain no VIN, plate, ZIP,
|
|
station, technician identifier, private vehicle token, raw JSON, credential,
|
|
operational record, or row-level prediction. The browser never connects to the
|
|
source database.
|
|
|
|
Bolt publication must use a separate project or repository containing only the
|
|
contents of `dashboard/`, with `index.html` at its root. The full source
|
|
repository must not be imported into Bolt. The reviewed procedure is in
|
|
[bolt_deployment.md](bolt_deployment.md).
|
|
|
|
## Limitations
|
|
|
|
- The private page sample is not population-representative.
|
|
- Results cover returning vehicles in participating feeds, not all Utah
|
|
vehicles or counties.
|
|
- Geography, source era, program mix, and time are entangled.
|
|
- Reject and abort are heterogeneous non-pass outcomes.
|
|
- The 2025 comparison is a one-time development holdout, not external
|
|
validation.
|
|
- Sample calibration and ranking do not establish operational usefulness.
|
|
- Observed cohort differences are descriptive associations, not causal effects
|
|
or recommendations.
|
|
- The absence of protected attributes does not establish fairness.
|
|
|
|
## Conclusion
|
|
|
|
The prototype answers its development question: a leakage-safe, calibrated
|
|
logistic regression produces materially better discrimination and probability-error metrics
|
|
than simple baselines in the private 10,000-vehicle sample, and it performs
|
|
better than the boosted-tree benchmark in the one-time holdout comparison. The
|
|
more important product result is the disciplined boundary around that finding:
|
|
one target, chronological evaluation, transparent model selection, persistent
|
|
sample labeling, fail-closed aggregate publication, and no row-level exposure.
|
|
|
|
This is a completed development prototype—not a population estimate, county
|
|
ranking, production model, diagnosis, or individual decision tool.
|
|
|
|
## Supporting documentation
|
|
|
|
- [Project charter](project_charter.md)
|
|
- [Modeling protocol](modeling_protocol.md)
|
|
- [Detailed development results](development_results.md)
|
|
- [Model card](model_card.md)
|
|
- [Dashboard specification](dashboard_spec.md)
|
|
- [Private-to-public architecture](architecture.mmd)
|
|
- [10-minute presentation outline](presentation_outline.md)
|
|
- [Dashboard demo script](demo_script.md)
|
|
- [Dashboard-only Bolt deployment](bolt_deployment.md)
|