7.4 KiB
Utah Vehicle Health model card
Last updated: 2026-07-21
Status: final development-prototype model; not approved for production or population claims
Every count, rate, and performance value in this model card comes from the private, page-sampled 10,000-vehicle development cohort. These are sample results, not population estimates.
Model summary
Utah Vehicle Health estimates whether the first attempt of a returning
vehicle's next emissions-inspection episode will be a non-pass. The final model
is regularized logistic regression (C=0.03) followed by Platt probability
calibration. Histogram gradient boosting is retained only as a benchmark.
This is an inspection-history model. It is not a model of overall mechanical health, safety, roadworthiness, or legal compliance.
Intended use
The development model supports:
- validating leakage-safe longitudinal feature engineering;
- comparing a transparent final model with simple and nonlinear benchmarks;
- explaining calibration and chronological evaluation; and
- powering aggregate, suppression-reviewed development-preview diagnostics.
Do not use it to:
- decide anything about an individual vehicle, owner, registration, inspection, county, program, station, or technician;
- diagnose a vehicle or guarantee an inspection outcome;
- rank counties, programs, stations, vehicle cohorts, or people;
- make causal claims from observed sample associations;
- report statewide/county prevalence or production performance; or
- serve a prediction lookup or row-level score.
Prediction unit and target
The prediction is made immediately before a new inspection episode begins. Attempts no more than 30 days apart form one episode. The supervised target is the episode's first attempt, and a vehicle must have at least one prior completed episode.
The binary target is recognized pass (0) versus recognized fail, reject, or
abort (1, collectively non-pass). Blank, null, and unrecognized results
remain unlabeled. Reject and abort may reflect process/readiness conditions, so
non-pass must not be paraphrased as mechanical failure.
A recognized overall result is preferred. The documented older Utah County OBD exception may fill a blank overall result under a narrow source/program/test contract. That value is a binary proxy only. Its provenance remains private audit metadata and is not a predictor. See the modeling protocol.
Development data
The extraction page-samples inspection records and retrieves complete histories for up to 10,000 vehicles. After validation, the private run retained 9,996 vehicle tokens, 83,552 source events, 69,588 episodes, and 44,659 eligible returning targets. These sample counts describe pipeline reconciliation only; they do not imply population coverage.
The sampling design over-represents vehicles with more inspection records. Feed coverage changes by source and time and does not include all 29 Utah counties. Accordingly, the model card makes no statewide prevalence claim, no county ranking, and no claim that sample subgroup differences generalize.
Features and exclusions
The final model uses only pre-episode inspection-history information: vehicle age, prior episode/attempt counts, time since prior history, prior non-pass rate, prior first/final outcomes, previously observed make/model, public county context, and season.
The model excludes current-test results and diagnostics, later attempts, station and technician information, direct/pseudonymous identifiers, future records, full-history leakage, DMV enrichment, and preprocessing learned from evaluation periods. It does not use VIN, plate, ZIP, private token, raw JSON, credentials, or operational records. The complete contract is in modeling_protocol.md.
Chronological evaluation
| Period | Development-sample role |
|---|---|
| 2010-2015 | Historical context only |
| 2016-2022 | Fit preprocessing and logistic parameters |
| 2023 | Select regularization and compare fixed candidates |
| 2024 | Fit Platt scaling and inspect calibration behavior |
| 2025 | One-time development-sample holdout |
| 2026 partial | Shadow context only |
The explicit 2025 gate was opened once after specifications were frozen from pre-2025 data. No model was changed in response. This makes 2025 a one-time development holdout, not a pristine future test. No further 2025-informed tuning is permitted.
Development-sample performance
The following values use non-audit rows from the private 10,000-vehicle development sample. They are not population performance estimates.
| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
|---|---|---|---|---|---|
| 2023 tune | Training-prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic before calibration | 0.282 | 0.1012 | 0.707 | 0.315 |
| 2025 one-time holdout | Calibrated logistic, final | 0.261 | 0.1011 | 0.693 | 0.312 |
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
Platt scaling was fit on the 2024 development-sample calibration partition. Its same-partition PR-AUC of 0.257 and Brier score of 0.0936 are calibration diagnostics, not independent final performance.
The final logistic model was retained because it beat the two simple baselines on headline development metrics and outperformed the more complex tree in the one-time sample holdout. The benchmark did not justify added complexity.
The never-fit private-vehicle stress-test subset contained 480 eligible 2025 sample episodes from 450 vehicles; calibrated logistic PR-AUC was 0.253 and Brier score was 0.0859. These are also private-sample diagnostics, not population or external-validation results.
Limitations
- Results apply only to the sampled returning-vehicle cohort with recognizable labels and sufficient prior history.
- Page sampling is not population-representative.
- Source, program, time, and geography are entangled.
- Reject and abort are heterogeneous non-pass outcomes.
- Make/model aliases and incomplete feed coverage can distort cohorts.
- The one-time 2025 holdout is not an external validation dataset.
- No protected attributes are modeled, but that does not establish fairness.
- Sample calibration does not establish production calibration.
Privacy and public release
Private analytical artifacts remain local and Git-ignored. Dashboard files contain only rounded aggregate cells that pass episode, distinct-vehicle, and binary-class suppression. The browser validates publication flags and checksums and fails closed.
No VIN, plate, ZIP, station, technician identifier, private token, raw JSON, credential, operational record, or row-level prediction belongs in a public asset. The finished dashboard has no vehicle-level prediction service. See the dashboard specification.
Reproducibility
The pipeline uses fixed chronological boundaries and a fixed seed. Private manifests bind input lineage, configuration, convergence checks, row reconciliation, and checksums. Normal training commands keep the 2025 gate closed unless the explicit evaluation flag is supplied.
The detailed sample audit is in development_results.md, and the narrative conclusion is in final_report.md.