SummerProject2026/docs/model_card.md

# Utah Vehicle Health model card

Last updated: 2026-07-21

Status: **final development-prototype model; not approved for production or
population claims**

> Every count, rate, and performance value in this model card comes from the
> private, page-sampled 10,000-vehicle development cohort. These are sample
> results, not population estimates.

## Model summary

Utah Vehicle Health estimates whether the first attempt of a returning
vehicle's next emissions-inspection episode will be a non-pass. The final model
is regularized logistic regression (`C=0.03`) followed by Platt probability
calibration. Histogram gradient boosting is retained only as a benchmark.

This is an inspection-history model. It is not a model of overall mechanical
health, safety, roadworthiness, or legal compliance.

## Intended use

The development model supports:

- validating leakage-safe longitudinal feature engineering;
- comparing a transparent final model with simple and nonlinear benchmarks;
- explaining calibration and chronological evaluation; and
- powering aggregate, suppression-reviewed development-preview diagnostics.

Do not use it to:

- decide anything about an individual vehicle, owner, registration, inspection,
  county, program, station, or technician;
- diagnose a vehicle or guarantee an inspection outcome;
- rank counties, programs, stations, vehicle cohorts, or people;
- make causal claims from observed sample associations;
- report statewide/county prevalence or production performance; or
- serve a prediction lookup or row-level score.

## Prediction unit and target

The prediction is made immediately before a new inspection episode begins.
Attempts no more than 30 days apart form one episode. The supervised target is
the episode's first attempt, and a vehicle must have at least one prior completed
episode.

The binary target is recognized pass (`0`) versus recognized fail, reject, or
abort (`1`, collectively **non-pass**). Blank, null, and unrecognized results
remain unlabeled. Reject and abort may reflect process/readiness conditions, so
non-pass must not be paraphrased as mechanical failure.

A recognized overall result is preferred. The documented older Utah County OBD
exception may fill a blank overall result under a narrow source/program/test
contract. That value is a binary proxy only. Its provenance remains private
audit metadata and is not a predictor. See the
[modeling protocol](modeling_protocol.md#binary-label-contract).

## Development data

The extraction page-samples inspection records and retrieves complete histories
for up to 10,000 vehicles. After validation, the private run retained 9,996
vehicle tokens, 83,552 source events, 69,588 episodes, and 44,659 eligible
returning targets. These sample counts describe pipeline reconciliation only;
they do not imply population coverage.

The sampling design over-represents vehicles with more inspection records. Feed
coverage changes by source and time and does not include all 29 Utah counties.
Accordingly, the model card makes no statewide prevalence claim, no county
ranking, and no claim that sample subgroup differences generalize.

## Features and exclusions

The final model uses only pre-episode inspection-history information: vehicle
age, prior episode/attempt counts, time since prior history, prior non-pass rate,
prior first/final outcomes, previously observed make/model, public county
context, and season.

The model excludes current-test results and diagnostics, later attempts, station
and technician information, direct/pseudonymous identifiers, future records,
full-history leakage, DMV enrichment, and preprocessing learned from evaluation
periods. It does not use VIN, plate, ZIP, private token, raw JSON, credentials,
or operational records. The complete contract is in
[modeling_protocol.md](modeling_protocol.md#leakage-and-privacy-exclusions).

## Chronological evaluation

| Period | Development-sample role |
| --- | --- |
| 2010-2015 | Historical context only |
| 2016-2022 | Fit preprocessing and logistic parameters |
| 2023 | Select regularization and compare fixed candidates |
| 2024 | Fit Platt scaling and inspect calibration behavior |
| 2025 | One-time development-sample holdout |
| 2026 partial | Shadow context only |

The explicit 2025 gate was opened once after specifications were frozen from
pre-2025 data. No model was changed in response. This makes 2025 a one-time
development holdout, not a pristine future test. No further 2025-informed tuning
is permitted.

## Development-sample performance

The following values use non-audit rows from the **private 10,000-vehicle
development sample**. They are **not population performance estimates**.

| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Training-prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |

Platt scaling was fit on the 2024 development-sample calibration partition. Its
same-partition PR-AUC of 0.257 and Brier score of 0.0936 are calibration
diagnostics, not independent final performance.

The final logistic model was retained because it beat the two simple baselines
on headline development metrics and outperformed the more complex tree in the
one-time sample holdout. The benchmark did not justify added complexity.

The never-fit private-vehicle stress-test subset contained 480 eligible 2025
sample episodes from 450 vehicles; calibrated logistic PR-AUC was 0.253 and
Brier score was 0.0859. These are also private-sample diagnostics, not
population or external-validation results.

## Limitations

- Results apply only to the sampled returning-vehicle cohort with recognizable
  labels and sufficient prior history.
- Page sampling is not population-representative.
- Source, program, time, and geography are entangled.
- Reject and abort are heterogeneous non-pass outcomes.
- Make/model aliases and incomplete feed coverage can distort cohorts.
- The one-time 2025 holdout is not an external validation dataset.
- No protected attributes are modeled, but that does not establish fairness.
- Sample calibration does not establish production calibration.

## Privacy and public release

Private analytical artifacts remain local and Git-ignored. Dashboard files
contain only rounded aggregate cells that pass episode, distinct-vehicle, and
binary-class suppression. The browser validates publication flags and checksums
and fails closed.

No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
credential, operational record, or row-level prediction belongs in a public
asset. The finished dashboard has no vehicle-level prediction service. See the
[dashboard specification](dashboard_spec.md#suppression-and-privacy-controls).

## Reproducibility

The pipeline uses fixed chronological boundaries and a fixed seed. Private
manifests bind input lineage, configuration, convergence checks, row
reconciliation, and checksums. Normal training commands keep the 2025 gate
closed unless the explicit evaluation flag is supplied.

The detailed sample audit is in
[development_results.md](development_results.md), and the narrative conclusion
is in [final_report.md](final_report.md).