111 lines
5.0 KiB
Markdown
111 lines
5.0 KiB
Markdown
# Utah Vehicle Health project charter
|
|
|
|
## Prototype status
|
|
|
|
**Presentation-ready development prototype.** Every analytical finding and every
|
|
dashboard value comes from the private, page-sampled 10,000-vehicle development
|
|
cohort. These values are **sample results, not population estimates**. They must
|
|
not be used to estimate statewide or county prevalence, create county rankings,
|
|
or make decisions about an individual vehicle.
|
|
|
|
## Product promise
|
|
|
|
Utah Vehicle Health explains how information available before an inspection
|
|
relates to the first-attempt non-pass outcome of a returning vehicle's next
|
|
inspection episode.
|
|
|
|
The product measures an emissions-inspection outcome. It does not measure
|
|
overall vehicle health, mechanical reliability, safety, roadworthiness, or
|
|
legal compliance.
|
|
|
|
## Primary research question
|
|
|
|
> Using only information available before an inspection episode begins, how
|
|
> well can a calibrated logistic regression estimate whether a returning
|
|
> vehicle's next episode will have a first-attempt non-pass outcome?
|
|
|
|
An episode groups consecutive attempts for the same private vehicle token when
|
|
the gap is 30 days or less. The target is the first attempt of a new episode,
|
|
not a rapid retest. A vehicle is eligible only after at least one completed
|
|
prior episode.
|
|
|
|
## Binary target
|
|
|
|
- `0 - pass`: normalized `PASS` or `P`
|
|
- `1 - non-pass`: normalized `FAIL`, `F`, `REJECT`, or `ABORT`
|
|
- unlabeled: blank, null, and unrecognized results
|
|
|
|
Reject and abort can reflect process or readiness conditions rather than a
|
|
mechanical failure. The prototype therefore says **non-pass**, never “vehicle
|
|
failure,” when referring to the combined target. Multiclass outcome modeling
|
|
and fail-only sensitivity analysis are outside the finished prototype scope.
|
|
|
|
## Data and product boundary
|
|
|
|
The development cohort contains complete inspection histories for up to 10,000
|
|
sampled vehicles from participating Utah county/source feeds. Page sampling
|
|
over-represents vehicles with more inspection records, so neither the cohort nor
|
|
its dashboard aggregates are population-representative. The feeds do not cover
|
|
all 29 Utah counties.
|
|
|
|
The final model is inspection-history only. DMV enrichment, rich current-test
|
|
OBD fields, station effects, cold-start prediction, and individualized lookup
|
|
are outside scope. The static dashboard publishes only rounded, suppressed
|
|
sample aggregates and pre-2025 development diagnostics.
|
|
|
|
## Final model and benchmark
|
|
|
|
The final model is regularized logistic regression (`C=0.03`) with Platt
|
|
probability calibration fit on the 2024 development-sample partition. A
|
|
histogram gradient-boosted tree is retained only as a nonlinear benchmark. It
|
|
is not a second final model and is not used to drive the product.
|
|
|
|
## Completed deliverables
|
|
|
|
1. Read-only extraction with private vehicle pseudonymization.
|
|
2. Leakage-safe episode and point-in-time feature mart.
|
|
3. Training-prevalence and previous-episode baselines.
|
|
4. Calibrated logistic regression as the final model.
|
|
5. Histogram gradient boosting as a benchmark only.
|
|
6. Chronological development evaluation with an explicit one-time 2025 gate.
|
|
7. Suppression-reviewed static dashboard assets with fail-closed validation.
|
|
8. Model card, final report, presentation materials, and dashboard-only Bolt
|
|
deployment instructions.
|
|
|
|
## Acceptance criteria
|
|
|
|
- The prediction unit remains a returning vehicle's next-episode first attempt.
|
|
- Model features exist before that episode begins.
|
|
- The calibrated logistic model beats both simple baselines on development-
|
|
sample PR-AUC and Brier score.
|
|
- The boosted tree is presented only as a benchmark.
|
|
- Every displayed value is labeled as a private-sample result and not a
|
|
population estimate.
|
|
- County views communicate feed coverage and sample context; they do not claim
|
|
population rankings or causal county differences.
|
|
- No VIN, plate, ZIP, station, technician identifier, raw JSON, credential,
|
|
operational record, private token, or row-level prediction reaches the
|
|
dashboard or presentation materials.
|
|
- The dashboard contains no vehicle-level estimator, input form, or prediction
|
|
service.
|
|
|
|
## Non-goals
|
|
|
|
- Predicting the result of a first-observed vehicle
|
|
- Diagnosing, certifying, or guaranteeing an individual vehicle outcome
|
|
- Ranking counties, programs, stations, technicians, owners, or vehicles
|
|
- Making causal claims about geography, vehicle makes, or inspection programs
|
|
- Accepting VIN, plate, address, station, or diagnostic inputs
|
|
- Publishing population prevalence from the development sample
|
|
- Shipping a production decision service or live database connection
|
|
- Completing DMV enrichment, four-class modeling, or failure-to-pass journeys
|
|
|
|
## Evaluation status
|
|
|
|
The explicit 2025 gate was opened once after model specifications were frozen
|
|
from pre-2025 data. No 2025-informed model change was made. The 2025 values are
|
|
therefore a one-time **development-sample holdout**, not a pristine future test
|
|
and not population performance. See [development_results.md](development_results.md)
|
|
for the audit trail and [modeling_protocol.md](modeling_protocol.md) for the
|
|
frozen evaluation contract.
|