SummerProject2026/docs/model_card.md

Utah Vehicle Health model card

Last updated: 2026-07-15

Status: development prototype; not approved for production or population claims

Model summary

Utah Vehicle Health estimates the probability that the first attempt of a returning vehicle's next emissions-inspection episode will be a non-pass. The current selected model is a regularized logistic regression followed by Platt probability calibration. It is an inspection-history model, not a mechanical health, safety, roadworthiness, or legal-compliance model.

This model card summarizes the current page-sampled development run. The full research and product contracts are in the project charter, modeling protocol, and dashboard specification.

Intended use and prohibited use

Intended uses are to validate the data-engineering and modeling pipeline, compare leakage-safe model candidates, study aggregate patterns, and support a clearly marked development dashboard preview.

Do not use the model to:

• make decisions about an individual vehicle, owner, registration, inspection, or station;
• diagnose a vehicle, guarantee an inspection result, or infer safety or roadworthiness;
• rank or penalize people, counties, programs, or inspection stations;
• make causal claims from observed associations; or
• report statewide prevalence, county rankings, or production performance from the current development sample.

The public estimator remains disabled because no privacy-reviewed prediction lookup has been approved.

Prediction unit and target

The prediction is made immediately before a new inspection episode begins. An episode groups consecutive attempts no more than 30 days apart, and only its first attempt is the supervised target. A vehicle must have at least one prior completed episode to enter the returning-vehicle cohort.

The binary target is:

• 0: recognized pass;
• 1: recognized fail, reject, or abort; and
• unlabeled: blank, null, or unrecognized results.

A recognized overall result is preferred. For the older Utah County OBD feed, a controlled OBD result may fill a blank overall result only under the narrow source/program/test contract documented in the modeling protocol. That proxy is approved only for binary pass versus non-pass; it must not support a four-class interpretation. Label provenance is private audit metadata and is not a predictor.

Development data and representativeness

The current extract samples inspection-table pages and then retrieves complete histories for up to 10,000 vehicles. This over-represents vehicles with more inspection records. The run contains 83,552 retained source events, 69,588 episodes, and 44,659 eligible returning targets, but those counts do not make the sample population-representative.

Coverage is limited to participating county/source feeds and changes over time. It does not represent all 29 Utah counties. Source-system transitions, missing years, and partial periods can resemble real changes in risk. See the data inventory and development results for the audited scope.

Features and leakage exclusions

The selected model uses information available before the target episode:

• vehicle age;
• prior episode and attempt counts;
• days since the prior episode and prior adverse outcome;
• prior non-pass rate;
• prior first and final outcomes;
• previously observed make and model;
• public county; and
• target season.

Categorical mappings and all preprocessing are fit on training data only. The current model does not yet include DMV enrichment or rich same-test OBD fields.

Excluded inputs include direct or pseudonymous identifiers, station information, the target attempt's result or diagnostics, later attempts in the target episode, future records, full-history aggregates, label provenance, and preprocessing learned from evaluation periods. The complete exclusion contract is in modeling_protocol.md.

Chronology and holdout status

The fixed temporal design is:

Period	Role
2010-2015	Historical context only
2016-2022	Fit preprocessing and models
2023	Select hyperparameters and compare candidates
2024	Fit probability calibration and inspect calibration behavior
2025	One-time development holdout
2026 partial	Shadow monitoring only

During live verification, the explicit 2025 gate was opened once after both candidate specifications and the tree search grid had already been fixed from pre-2025 data. No model was changed in response. The already-fixed logistic model was then evaluated once for a direct comparison. Consequently, this page-sampled 2025 cohort is no longer a pristine unseen test, and no further 2025-informed tuning is permitted. Any complete-data 2025 analysis must be described as confirmatory. The detailed audit trail is in development_results.md.

Candidate comparison and selected model

The candidates were training prevalence, the literal previous-episode outcome, regularized logistic regression, and histogram gradient boosting. Results below use non-audit development rows.

Evaluation	Model	PR-AUC	Brier	ROC-AUC	Top-10% precision
2023 tuning	Training prevalence	0.125	0.1092	0.500	0.125
2023 tuning	Previous episode	0.166	0.1723	0.582	0.285
2023 tuning	Logistic	0.282	0.1012	0.707	0.315
2025 one-time holdout	Logistic + Platt	0.261	0.1011	0.693	0.312
2025 one-time holdout	Gradient boosting + Platt	0.238	0.1024	0.690	0.292

The selected logistic model uses C=0.03. It satisfied explicit convergence and finite-value checks. Platt scaling was fit on the 2024 calibration cohort; on that same cohort, PR-AUC was 0.257 and Brier score was 0.0936. Those 2024 values are calibration diagnostics, not independent final performance.

The logistic model remains selected because it beat both simple baselines and the more complex tree on the one-time development holdout while remaining more transparent. The separate never-fit vehicle audit also supported the logistic model, but the development sample is too limited for final generalization or fairness claims.

Coverage, fairness, and privacy limitations

• The model applies only to returning vehicles with recognizable labels and sufficient prior history; cold-start behavior is not established.
• Geography and source era are entangled. Performance may shift when a feed, county program, vehicle mix, or label process changes.
• Reject and abort outcomes can reflect readiness or process issues rather than mechanical failure.
• No protected attributes are modeled, but their absence does not establish fairness. Subgroup sample sizes, errors, and calibration still require review.
• Make/model normalization and complete-data subgroup analysis are unfinished.
• The current probability calibration has not yet received independent, vehicle-clustered uncertainty analysis or external validation.

Private analytical artifacts remain local. Public assets contain only reviewed, rounded aggregates that satisfy episode, distinct-vehicle, and binary-class suppression thresholds. Direct identifiers, private linkage values, raw rows, and row-level predictions are outside the public contract. See dashboard_spec.md.

Monitoring

Before any deployment, monitoring must cover source-level volume, label recognition, outcome prevalence, source/program mix, missingness, unseen categories, score distributions, PR-AUC, Brier score, and calibration. Reviews must explicitly separate the Salt Lake source transition, newer rich feeds, DMV coverage gaps, and the partial 2026 period. Alert thresholds and a response plan remain to be defined.

Reproducibility

The pipeline uses fixed temporal boundaries and a fixed random seed. Python dependencies are pinned in requirements.txt. Private manifests record input lineage, software/model configuration, convergence, row reconciliation, and checksums without publishing private paths or values. SQL transforms and validation queries are versioned under sql, model runners are under scripts, and automated checks are under tests. Normal training commands leave the 2025 evaluation gate closed unless an explicit flag is supplied.

Remaining approval gates

1. Run a complete, contiguous, bounded extraction and rebuild the frozen pipeline without page-sampling bias.
2. Treat complete-data 2025 results as confirmation and reserve a genuinely new period or external dataset for future unseen evaluation.
3. Complete subgroup/source-era reporting, vehicle-clustered uncertainty, calibration diagnostics, and fairness review.
4. Finish make/model normalization, episode-gap sensitivity checks, and the preregistered inspection-only versus DMV-enhanced ablation.
5. Pass privacy and publication review before replacing the development-preview aggregates or enabling any prediction lookup.
6. Define monitoring thresholds, ownership, rollback criteria, and a model update policy before operational use.