198 lines
9.2 KiB
Markdown
198 lines
9.2 KiB
Markdown
# Utah Vehicle Health model card
|
|
|
|
Last updated: 2026-07-15
|
|
|
|
Status: **development prototype; not approved for production or population claims**
|
|
|
|
## Model summary
|
|
|
|
Utah Vehicle Health estimates the probability that the first attempt of a
|
|
returning vehicle's next emissions-inspection episode will be a non-pass. The
|
|
current selected model is a regularized logistic regression followed by Platt
|
|
probability calibration. It is an inspection-history model, not a mechanical
|
|
health, safety, roadworthiness, or legal-compliance model.
|
|
|
|
This model card summarizes the current page-sampled development run. The full
|
|
research and product contracts are in the [project charter](project_charter.md),
|
|
[modeling protocol](modeling_protocol.md), and
|
|
[dashboard specification](dashboard_spec.md).
|
|
|
|
## Intended use and prohibited use
|
|
|
|
Intended uses are to validate the data-engineering and modeling pipeline,
|
|
compare leakage-safe model candidates, study aggregate patterns, and support a
|
|
clearly marked development dashboard preview.
|
|
|
|
Do not use the model to:
|
|
|
|
- make decisions about an individual vehicle, owner, registration, inspection,
|
|
or station;
|
|
- diagnose a vehicle, guarantee an inspection result, or infer safety or
|
|
roadworthiness;
|
|
- rank or penalize people, counties, programs, or inspection stations;
|
|
- make causal claims from observed associations; or
|
|
- report statewide prevalence, county rankings, or production performance from
|
|
the current development sample.
|
|
|
|
The public estimator remains disabled because no privacy-reviewed prediction
|
|
lookup has been approved.
|
|
|
|
## Prediction unit and target
|
|
|
|
The prediction is made immediately before a new inspection episode begins. An
|
|
episode groups consecutive attempts no more than 30 days apart, and only its
|
|
first attempt is the supervised target. A vehicle must have at least one prior
|
|
completed episode to enter the returning-vehicle cohort.
|
|
|
|
The binary target is:
|
|
|
|
- `0`: recognized pass;
|
|
- `1`: recognized fail, reject, or abort; and
|
|
- unlabeled: blank, null, or unrecognized results.
|
|
|
|
A recognized overall result is preferred. For the older Utah County OBD feed,
|
|
a controlled OBD result may fill a blank overall result only under the narrow
|
|
source/program/test contract documented in the
|
|
[modeling protocol](modeling_protocol.md#source-specific-label-contract). That
|
|
proxy is approved only for binary pass versus non-pass; it must not support a
|
|
four-class interpretation. Label provenance is private audit metadata and is
|
|
not a predictor.
|
|
|
|
## Development data and representativeness
|
|
|
|
The current extract samples inspection-table pages and then retrieves complete
|
|
histories for up to 10,000 vehicles. This over-represents vehicles with more
|
|
inspection records. The run contains 83,552 retained source events, 69,588
|
|
episodes, and 44,659 eligible returning targets, but those counts do not make
|
|
the sample population-representative.
|
|
|
|
Coverage is limited to participating county/source feeds and changes over time.
|
|
It does not represent all 29 Utah counties. Source-system transitions, missing
|
|
years, and partial periods can resemble real changes in risk. See the
|
|
[data inventory](data_inventory.md) and
|
|
[development results](development_results.md) for the audited scope.
|
|
|
|
## Features and leakage exclusions
|
|
|
|
The selected model uses information available before the target episode:
|
|
|
|
- vehicle age;
|
|
- prior episode and attempt counts;
|
|
- days since the prior episode and prior adverse outcome;
|
|
- prior non-pass rate;
|
|
- prior first and final outcomes;
|
|
- previously observed make and model;
|
|
- public county; and
|
|
- target season.
|
|
|
|
Categorical mappings and all preprocessing are fit on training data only. The
|
|
current model does not yet include DMV enrichment or rich same-test OBD fields.
|
|
|
|
Excluded inputs include direct or pseudonymous identifiers, station
|
|
information, the target attempt's result or diagnostics, later attempts in the
|
|
target episode, future records, full-history aggregates, label provenance, and
|
|
preprocessing learned from evaluation periods. The complete exclusion contract
|
|
is in [modeling_protocol.md](modeling_protocol.md#leakage-exclusions).
|
|
|
|
## Chronology and holdout status
|
|
|
|
The fixed temporal design is:
|
|
|
|
| Period | Role |
|
|
| --- | --- |
|
|
| 2010-2015 | Historical context only |
|
|
| 2016-2022 | Fit preprocessing and models |
|
|
| 2023 | Select hyperparameters and compare candidates |
|
|
| 2024 | Fit probability calibration and inspect calibration behavior |
|
|
| 2025 | One-time development holdout |
|
|
| 2026 partial | Shadow monitoring only |
|
|
|
|
During live verification, the explicit 2025 gate was opened once after both
|
|
candidate specifications and the tree search grid had already been fixed from
|
|
pre-2025 data. No model was changed in response. The already-fixed logistic
|
|
model was then evaluated once for a direct comparison. Consequently, this
|
|
page-sampled 2025 cohort is no longer a pristine unseen test, and no further
|
|
2025-informed tuning is permitted. Any complete-data 2025 analysis must be
|
|
described as confirmatory. The detailed audit trail is in
|
|
[development_results.md](development_results.md#one-time-development-holdout).
|
|
|
|
## Candidate comparison and selected model
|
|
|
|
The candidates were training prevalence, the literal previous-episode outcome,
|
|
regularized logistic regression, and histogram gradient boosting. Results below
|
|
use non-audit development rows.
|
|
|
|
| Evaluation | Model | PR-AUC | Brier | ROC-AUC | Top-10% precision |
|
|
| --- | --- | ---: | ---: | ---: | ---: |
|
|
| 2023 tuning | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
|
|
| 2023 tuning | Previous episode | 0.166 | 0.1723 | 0.582 | 0.285 |
|
|
| 2023 tuning | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
|
|
| 2025 one-time holdout | Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
|
|
| 2025 one-time holdout | Gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
|
|
|
|
The selected logistic model uses `C=0.03`. It satisfied explicit convergence
|
|
and finite-value checks. Platt scaling was fit on the 2024 calibration cohort;
|
|
on that same cohort, PR-AUC was 0.257 and Brier score was 0.0936. Those 2024
|
|
values are calibration diagnostics, not independent final performance.
|
|
|
|
The logistic model remains selected because it beat both simple baselines and
|
|
the more complex tree on the one-time development holdout while remaining more
|
|
transparent. The separate never-fit vehicle audit also supported the logistic
|
|
model, but the development sample is too limited for final generalization or
|
|
fairness claims.
|
|
|
|
## Coverage, fairness, and privacy limitations
|
|
|
|
- The model applies only to returning vehicles with recognizable labels and
|
|
sufficient prior history; cold-start behavior is not established.
|
|
- Geography and source era are entangled. Performance may shift when a feed,
|
|
county program, vehicle mix, or label process changes.
|
|
- Reject and abort outcomes can reflect readiness or process issues rather than
|
|
mechanical failure.
|
|
- No protected attributes are modeled, but their absence does not establish
|
|
fairness. Subgroup sample sizes, errors, and calibration still require review.
|
|
- Make/model normalization and complete-data subgroup analysis are unfinished.
|
|
- The current probability calibration has not yet received independent,
|
|
vehicle-clustered uncertainty analysis or external validation.
|
|
|
|
Private analytical artifacts remain local. Public assets contain only reviewed,
|
|
rounded aggregates that satisfy episode, distinct-vehicle, and binary-class
|
|
suppression thresholds. Direct identifiers, private linkage values, raw rows,
|
|
and row-level predictions are outside the public contract. See
|
|
[dashboard_spec.md](dashboard_spec.md#publication-controls).
|
|
|
|
## Monitoring
|
|
|
|
Before any deployment, monitoring must cover source-level volume, label
|
|
recognition, outcome prevalence, source/program mix, missingness, unseen
|
|
categories, score distributions, PR-AUC, Brier score, and calibration. Reviews
|
|
must explicitly separate the Salt Lake source transition, newer rich feeds, DMV
|
|
coverage gaps, and the partial 2026 period. Alert thresholds and a response plan
|
|
remain to be defined.
|
|
|
|
## Reproducibility
|
|
|
|
The pipeline uses fixed temporal boundaries and a fixed random seed. Python
|
|
dependencies are pinned in [requirements.txt](../requirements.txt). Private
|
|
manifests record input lineage, software/model configuration, convergence, row
|
|
reconciliation, and checksums without publishing private paths or values. SQL
|
|
transforms and validation queries are versioned under [sql](../sql), model
|
|
runners are under [scripts](../scripts), and automated checks are under
|
|
[tests](../tests). Normal training commands leave the 2025 evaluation gate
|
|
closed unless an explicit flag is supplied.
|
|
|
|
## Remaining approval gates
|
|
|
|
1. Run a complete, contiguous, bounded extraction and rebuild the frozen
|
|
pipeline without page-sampling bias.
|
|
2. Treat complete-data 2025 results as confirmation and reserve a genuinely new
|
|
period or external dataset for future unseen evaluation.
|
|
3. Complete subgroup/source-era reporting, vehicle-clustered uncertainty,
|
|
calibration diagnostics, and fairness review.
|
|
4. Finish make/model normalization, episode-gap sensitivity checks, and the
|
|
preregistered inspection-only versus DMV-enhanced ablation.
|
|
5. Pass privacy and publication review before replacing the development-preview
|
|
aggregates or enabling any prediction lookup.
|
|
6. Define monitoring thresholds, ownership, rollback criteria, and a model
|
|
update policy before operational use.
|