123 lines
5.2 KiB
Markdown
123 lines
5.2 KiB
Markdown
# Utah Vehicle Health project charter
|
|
|
|
## Working title
|
|
|
|
**Utah Vehicle Health**
|
|
|
|
*What millions of emissions inspections reveal about the next test across the
|
|
Utah county feeds available in this dataset.*
|
|
|
|
## Product promise
|
|
|
|
Explain how vehicle age, type, location, and prior inspection history relate to
|
|
the chance of passing the first attempt of the next emissions-inspection
|
|
episode.
|
|
|
|
The product measures **emissions-inspection outcomes**. It must not describe its
|
|
score as a diagnosis of overall mechanical reliability, roadworthiness, safety,
|
|
or legal compliance.
|
|
|
|
## Primary research question
|
|
|
|
> Using only information available before an inspection episode begins, how
|
|
> accurately and reliably can we estimate whether its first attempt will pass?
|
|
|
|
An episode groups attempts for the same internal vehicle token when the gap
|
|
between consecutive attempts is 30 days or less. The target is the first attempt
|
|
of a new episode, not every rapid retest.
|
|
|
|
## Primary target
|
|
|
|
- `0 — pass`: normalized `PASS` or `P`
|
|
- `1 — non-pass`: normalized `FAIL`, `F`, `REJECT`, or `ABORT`
|
|
- Blank, null, and unrecognized results are unlabeled. They remain in chronology
|
|
and coverage reporting but are excluded as supervised targets.
|
|
|
|
Because reject and abort can reflect process/readiness problems rather than
|
|
mechanical failure, the project will also report:
|
|
|
|
- a four-class pass/fail/reject/abort analysis; and
|
|
- a fail-versus-pass sensitivity analysis that excludes reject and abort.
|
|
|
|
## Scope
|
|
|
|
The returning-vehicle MVP uses universal inspection-history fields and prior
|
|
DMV information where it is point-in-time valid. It covers participating
|
|
inspection county/source feeds, not all 29 Utah counties.
|
|
|
|
The rich OBD/odometer fields are a later, source-specific extension for newer
|
|
`slco`, `davis`, and `cache` records. They are not part of the statewide-style
|
|
historical baseline.
|
|
|
|
## Deliverables
|
|
|
|
1. A reproducible, read-only extraction and private pseudonymization pipeline.
|
|
2. A point-in-time episode/feature mart with an auditable exclusion report.
|
|
3. Prevalence, previous-outcome, and logistic-regression baselines.
|
|
4. One calibrated boosted-tree model and an inspection-only versus DMV-enhanced
|
|
ablation.
|
|
5. A locked temporal evaluation with subgroup and source-era diagnostics.
|
|
6. Versioned, suppressed public aggregates for a four-page Bolt dashboard.
|
|
7. A model card and data/methodology page documenting limitations.
|
|
|
|
## Headline success criteria
|
|
|
|
- Beat both the training-prevalence and previous-episode-outcome baselines on
|
|
2025 PR-AUC and Brier score.
|
|
- Produce calibrated probabilities, not just class labels.
|
|
- Report performance by county/source era, vehicle-age band, history depth, and
|
|
DMV match/staleness.
|
|
- Reproduce all published charts from versioned sanitized outputs.
|
|
- Export no VIN, plate, ZIP, station, raw JSON, operational record, or
|
|
row-level prediction to Bolt.
|
|
- Suppress public cells with fewer than 100 eligible inspections or fewer than
|
|
10 observations in an outcome or its complement, with complementary
|
|
suppression where totals could reveal a hidden cell.
|
|
|
|
## Initial feasibility result
|
|
|
|
Under label contract v3, the fixed-seed aggregate-only query in
|
|
`sql/10_episode_cohort_feasibility.sql` produced 8,916 eligible
|
|
returning-vehicle episodes from 1,730 of 2,000 sampled vehicle histories. The
|
|
first-attempt non-pass rate was 11.80%, and the median gap from the prior
|
|
episode was about 372 days. Of those targets, 1,732 use the documented Utah OBD
|
|
binary proxy because that source leaves its overall-result field blank. This
|
|
supports both the episode definition and a calibrated binary model. The sample
|
|
is for pipeline feasibility, not a population estimate.
|
|
|
|
The 2025 sample showed a longer median gap and different outcome mix, reinforcing
|
|
the need for source-era drift reporting and a locked chronological test.
|
|
|
|
## Development holdout status
|
|
|
|
During live verification, the explicit 2025 gate was opened once on the
|
|
page-sampled development extract after both candidate specifications had been
|
|
fixed from pre-2025 data. No test-informed model change was made. The frozen
|
|
logistic model remains selected, and a later complete-data 2025 analysis will
|
|
be treated as confirmatory rather than described as a pristine unseen test.
|
|
The audit trail and results are recorded in
|
|
[development_results.md](development_results.md).
|
|
|
|
## Non-goals
|
|
|
|
- Diagnosing an individual vehicle
|
|
- Certifying that a vehicle will pass
|
|
- Ranking or accusing inspection stations
|
|
- Identifying owners or accepting VIN/plate input
|
|
- Making causal claims about county programs
|
|
- Treating missing outcomes as passes
|
|
|
|
## Staged build
|
|
|
|
1. **Foundation:** episode definition, label normalization, exclusions, secure
|
|
extraction, and coverage checks.
|
|
2. **Baseline:** inspection-history feature mart and transparent baselines.
|
|
3. **Enrichment:** canonical vehicle dimension and point-in-time DMV features.
|
|
4. **Modeling:** tree model, calibration, locked test, drift and subgroup audits.
|
|
5. **Product:** sanitized aggregate export and Bolt MVP.
|
|
6. **Stretch:** failure-to-pass journeys, multiclass probabilities, and a
|
|
clearly scoped rich-OBD model.
|
|
|
|
See [modeling_protocol.md](modeling_protocol.md) and
|
|
[dashboard_spec.md](dashboard_spec.md) for the detailed contracts.
|