Centralized_I-M_Database/SummerProject2026
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial code

Browse Source
This commit is contained in:
kevinbell
2026-07-15 17:55:53 -06:00
parent 0952a7ffce
commit 05729fc6de
53 changed files with 12965 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/dashboard_spec.md
+97
View File
@@ -0,0 +1,97 @@
# Bolt dashboard specification
## Product language
Use **Utah Vehicle Health** as the brand, but call the modeled quantity
**next-episode non-pass risk**. The estimator is a cohort estimate, not a
diagnosis, certification, or guarantee.
## Four-page MVP
The sections below describe the full product target. The checked-in
development preview intentionally implements a narrower safe subset: binary
pass/non-pass aggregates, supported make/model scorecards, age bands, coverage
quality, and pre-2025 model diagnostics. Four-class charts, uncertainty
intervals, adjusted scorecards, and the prediction lookup remain disabled until
their own reviewed aggregate assets exist.
### Overview
- Eligible inspections, pass rate, non-pass rate, and covered-period KPIs
- Quarterly pass/fail/reject/abort trend with blanks shown separately
- Covered-county map; unavailable counties remain gray
- Non-pass risk versus vehicle age with intervals
- Clear notices for partial periods and limited feed coverage
### Reliability explorer
- Search supported canonical make/model cohorts
- Compare up to three cohorts across vehicle-age bands
- Observed versus model-adjusted risk toggle
- Outcome-mix bar and uncertainty-aware ranked dot plot
- County, make/model, age band, fuel, program, and period filters
- Support size and interval displayed for every estimate
### Next-test risk estimator
Inputs are coarsened, non-identifying attributes: county, supported make/model,
vehicle-age band, fuel, prior episode outcome, time-since-prior band, season,
and approved program category.
Output a calibrated non-pass probability, uncertainty interval, relevant
baseline, and aggregate factor contributions. Never request VIN, plate, exact
address, station, free text, or current-test diagnostics.
### Data and methods
- Coverage timeline and source/year missingness heatmap
- Episode and target definitions
- `slc`/`slco` source-era explanation
- Temporal split, PR-AUC, Brier score, and calibration plot
- Subgroup/source-era performance
- Leakage controls, DMV gaps, partial periods, and limitations
## Public data contract
| Dataset | Safe grain |
| --- | --- |
| `data_manifest` | Data cutoff, deterministic release ID, model versions, definitions and exclusions |
| `overview_period_county` | Quarter/year × public county with rounded support and outcome rates |
| `cohort_scorecard` | Approved make/model × age band, optionally coarsened county/fuel |
| `age_risk_curve` | Approved cohort × age point/band with risk, interval and support |
| `prediction_lookup` | Only supported coarsened input combinations and calibrated outputs |
| `filter_catalog` | Publishable categories and valid combinations |
| `model_diagnostics` | Approved partition-level metrics; locked metrics require a separate release gate |
| `coverage_quality` | Source era × year volume, blank rate, linkage and availability |
Use purpose-built outputs rather than a single high-dimensional browser cube.
Bolt receives only these sanitized, versioned assets—never countydata
credentials or private analytical rows.
## Publication controls
- Suppress cells below 100 eligible inspections.
- Also require at least 100 distinct private vehicle tokens in every published
cell; tokens and distinct counts never enter the public asset.
- Suppress when an outcome or its complement has fewer than 10 records.
- Require at least 10 distinct vehicles contributing each binary class.
- Apply complementary suppression so totals cannot reconstruct hidden cells.
- Combine rare categories, coarsen model years, and round displayed counts.
- Recheck thresholds after every filter combination.
- Do not include suppressed rows in browser bundles, API responses, downloads,
analytics logs, or hidden chart layers.
- Downloads contain only the sanitized summary currently displayed.
## Visual direction
Use a restrained Utah/desert palette: teal pass, red fail, amber reject, purple
abort, and gray missing. Use probability bars, calibrated dot plots, confidence
bands, and cohort comparisons instead of gauges. Do not rely on color alone.
## Stretch pages
- Failure-to-pass journeys with funnels, attempts-to-pass and survival curves
- Four-class outcome probabilities
- Source-scoped OBD early-warning analysis
- Automated aggregate refresh with a dedicated read-only role
- Model-drift monitoring
docs/data_inventory.md
+102
View File
@@ -0,0 +1,102 @@
# Countydata inventory
Inventory date: 2026-07-15. All inspection was performed with read-only
transactions and aggregate queries; no identifiers were exported.
## Accessible databases
- `countydata`: the useful analytical database, approximately 93.4 GB.
- `postgres`: empty local/staging copies of the core vehicle table structure
plus foreign-table links. It is not the analytical source.
- `vattp`: a tiny, unrelated course/event registration and survey application.
It is not suitable for this project and its application records should remain
out of scope.
## DMV logical dataset
The following are one logical dataset split into projections joined one-to-one
by `id`; they are not four independent populations.
| Relation | Approximate role | Key analytical fields |
| --- | --- | --- |
| `data.dmv_tax_json` | Raw JSON and ingest metadata | Full DMV record, file/source metadata |
| `data.dmv_tax` | Core lookup | VIN, registration date, county |
| `data.dmv_tax_search` | Search projection | VIN, registration date, county |
| `data.dmv_tax_vehicle` | Vehicle projection | Make, model, model year, fuel, registration type/place, temporary flag, expiration/emission dates, ZIPs |
Exact current logical row count: **18,009,278**.
- Date bounds are 2011-09-07 through 2024-03-13, but the few pre-2016 rows are
outliers and there are only 117 records in 2021.
- The 29 Utah counties are represented. Salt Lake (35.66%), Utah (17.21%),
Davis (10.29%), and Weber (8.01%) account for most records.
- Canonicalized fuel mix by record is 86.23% gasoline, 7.97% diesel, 2.59%
flexible fuel, 2.06% hybrid, 0.70% electric, and 0.24% plug-in hybrid.
- A vehicle appears repeatedly over time: sampled DMV histories had a median of
five registration records.
## Inspection logical dataset
These are likewise projections of one logical inspection dataset joined by
`id`.
| Relation | Approximate role | Key analytical fields |
| --- | --- | --- |
| `data.inspection_json` | Raw JSON and ingest metadata | Full inspection, detailed OBD/readiness/visual fields where supplied |
| `data.inspection` | Core lookup | VIN, test timestamp, ingest timestamp |
| `data.inspection_search` | Outcome/search projection | County/source, overall and OBD results, test/program type |
| `data.inspection_vehicle` | Vehicle projection | Make, model, year, calibration/certificate, station |
| `data.inspection_obd` | OBD summary | Result-reason code, DTC count, permanent-DTC flag |
| `data.inspection_plate` | Identifier lookup | VIN, plate, test timestamp; sensitive and unnecessary for analytics |
Exact current logical row count: **19,357,287**.
- Four dates in 1990 are outliers. Normal coverage begins in 2010 and continues
through 2026-06-22; 2026 is partial.
- Real source/county labels are `slc`, `slco`, `utah`, `weber`, `davis`, and
`cache`. The two Salt Lake labels represent different source eras and should
not be blindly treated as different counties.
- Overall results are 71.06% pass, 3.61% fail, 3.56% reject, 2.31% abort, and
19.45% blank/null or other near-blank values in the raw `overall_result`
field. Most missing overall results belong to the older Utah County feed; its
audited OBD/OBD rows carry a separate result that is usable only as a
provenance-tagged binary pass-versus-non-pass proxy.
- About 89.5% of records use the OBD program and about 9.2% use TSI.
- DTC count is zero in about 90.4% of records and null in about 1.4%; the
remaining values are class-imbalanced and include rare data-quality outliers.
- Raw JSON can contain odometer, vehicle fuel/type/GVWR/cylinders/engine,
transmission, DTCs, PIDs, MIL/readiness status, communication protocol, and
visual inspection fields. These richer fields are concentrated in the newer
`slco`, `davis`, and `cache` feeds rather than statewide history.
## Longitudinal linkage
VIN is indexed in both logical datasets and makes longitudinal analysis
possible, but must never appear in public outputs.
- 88.2% of 10,000 sampled distinct inspection VINs had at least one DMV match.
- The sampled inspection history median was eight visits per vehicle.
- 74.2% of 10,000 sampled distinct DMV VINs had at least one inspection match.
- Extreme repeat counts exist and require invalid/shared-identifier filtering.
Use a salted one-way internal token if a stable identifier is needed during
feature engineering. Never send VINs or plates to the browser or Bolt.
## Operational and sensitive relations
The `imreports` and `fdw_data` schemas include users, invitations, sessions,
event/upload logs, file contents, client network metadata, and remote ingest
state. They are operational rather than analytical and should be excluded.
Foreign tables in `fdw_countydata` mirror source data and are unnecessary when
the normalized `data` tables are available.
## Primary quality risks
- Missing years and partial periods can masquerade as real trends.
- Make/model and fuel categories need canonicalization.
- Outcome blanks must not be treated as passes.
- County labels also encode source-system changes and therefore potential drift.
- Rich JSON features are missing by design in older feeds, not missing at
random.
- Same-test OBD/result fields cause target leakage in pre-test prediction.
- Direct identifiers and small groups require aggregation and suppression.
docs/development_results.md
+136
View File
@@ -0,0 +1,136 @@
# Development run results
Run date: 2026-07-15
These results validate the engineering and modeling pipeline. They are **not
population estimates**: the development extract samples inspection-table pages
and then retrieves complete histories for up to 10,000 vehicles, which
over-represents vehicles with more inspection records.
## Label audit
The raw Utah County feed leaves `overall_result` blank for 3,747,862 records.
An aggregate-only server audit found 3,469,102 Utah `obd`/`OBD` records with a
controlled pass, fail, reject, or abort value in `obd_result`. Across non-Utah
feeds where both fields were recognized, the two fields agreed on binary pass
versus non-pass 99.30% of the time, but their fail-versus-reject categories were
not interchangeable.
Label contract v3 therefore permits the Utah OBD value only as a provenance-
tagged binary proxy. TSI, `other/C`, `B`, blank, and unknown values stay
unlabeled. Four-class analysis must use `overall_result` exclusively. This also
matches the official program distinction between a readiness rejection and a
failed inspection; see the [Utah inspection requirements](https://dmv.utah.gov/register/inspections/)
and [program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
In the development extract, the rule restored 11,334 labels and left 1,281
events unlabeled. The label-source field remains private audit metadata and is
not a model feature.
## Pipeline reconciliation
| Stage | Rows |
| --- | ---: |
| Source events written after VIN validation | 83,552 |
| Clean events after duplicate/conflict handling | 80,190 |
| Inspection episodes | 69,588 |
| Eligible returning targets | 44,659 |
The sample contains 9,996 retained vehicle tokens. The extraction manifest,
compressed-file SHA-256, mart manifest, Parquet SHA-256, and row counts all
reconcile. Every mart invariant reports zero violations. Raw VIN and raw OBD
result are absent from the staging output schema, feature mart, and model
features.
| Temporal partition | Eligible targets | Never-fit audit targets |
| --- | ---: | ---: |
| Train, 2016–2022 | 31,745 | 3,129 |
| Tune, 2023 | 4,972 | 526 |
| Calibrate, 2024 | 2,633 | 277 |
| Locked test, 2025 | 4,451 | 480 |
| Shadow, 2026 partial | 858 | 106 |
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. The
normal trainers do not calculate 2025 outcomes or metrics without the explicit
locked-evaluation flag.
## Baseline results before the locked test
The table below uses only non-audit vehicles. Average precision is the project's
PR-AUC summary. The calibrated logistic row is shown only on the partition used
to fit the calibrator and is therefore a calibration diagnostic, not an
independent final estimate.
| Partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2024 calibrate | Training prevalence | 0.113 | 0.1002 | 0.500 | 0.113 |
| 2024 calibrate | Previous episode, literal | 0.148 | 0.1677 | 0.577 | 0.253 |
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |
The selected logistic regularization was `C=0.03`. It converged in 2,390 of the
5,000 allowed iterations; Platt calibration converged in five iterations. All
five candidates and all stored probabilities passed explicit convergence and
finite-value checks.
The literal previous-outcome baseline is useful as a ranking sanity check but
produces overconfident zero/one probabilities, explaining its poor Brier and log
loss. The transparent logistic model is the current development leader: on the
2023 tuning partition it more than doubles prevalence PR-AUC and raises top-10%
precision from 12.5% to 31.5%.
## One-time development holdout
The nonlinear live-verification command invoked the explicit 2025 gate after
its feature contract and four-candidate grid had already been fixed and its
candidate had been selected only on 2023. No test-driven model change was made.
At that point the development specification was frozen and the already-fixed
logistic model was evaluated once for a direct comparison. These are
development-sample holdout results, not final population claims.
| 2025 non-audit model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | ---: | ---: | ---: | ---: |
| Training prevalence | 0.123 | 0.1076 | 0.500 | 0.123 |
| Previous episode, literal | 0.158 | 0.1731 | 0.576 | 0.268 |
| Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
| Histogram gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
The holdout contains 3,971 non-audit episodes from 3,734 vehicles. The
never-fit audit contains another 480 episodes from 450 vehicles; calibrated
logistic PR-AUC is 0.253 and Brier is 0.0859 there. The logistic model beats
both simple baselines on the project's headline metrics and remains the chosen
development model. The tree did not provide a decisive pre-test improvement
that justified its added complexity, and no further 2025-informed tuning is
permitted.
## Remaining gates
1. Run a complete, contiguous bounded extraction for publishable population
aggregates; the page-sampled development cohort cannot support dashboard
prevalence or county rankings.
2. Treat any later full-data 2025 result as confirmatory rather than a pristine
unseen test, because the development sample's holdout has now been opened.
3. Run the frozen subgroup/source-era report and clustered uncertainty
analysis on the complete extraction.
4. Replace the checked-in development-preview bundle with complete-data
aggregates only after the population-publication review passes.
## Sanitized dashboard preview
The repository now includes a fail-closed exporter and a static four-view
dashboard shell. The preview exporter reads only train, tune, and calibration
rows dated before 2025. It rejects any model manifest that says the holdout was
evaluated, rejects any metric row outside the three approved partitions, and
marks every asset `development_preview=true` and
`population_estimate_allowed=false`.
Published cells must clear minimum episode and distinct-vehicle thresholds,
including both binary-class complements. Supports are rounded, direct and
pseudonymous identifiers stay private, and a checksum manifest covers the
approved JSON bundle. Private model manifests bind the exact metrics files, and
the public manifest exposes a deterministic release ID plus model versions for
provenance. The site disables the estimator because no privacy-reviewed
prediction lookup exists.
docs/model_card.md
+197
View File
@@ -0,0 +1,197 @@
# Utah Vehicle Health model card
Last updated: 2026-07-15
Status: **development prototype; not approved for production or population claims**
## Model summary
Utah Vehicle Health estimates the probability that the first attempt of a
returning vehicle's next emissions-inspection episode will be a non-pass. The
current selected model is a regularized logistic regression followed by Platt
probability calibration. It is an inspection-history model, not a mechanical
health, safety, roadworthiness, or legal-compliance model.
This model card summarizes the current page-sampled development run. The full
research and product contracts are in the [project charter](project_charter.md),
[modeling protocol](modeling_protocol.md), and
[dashboard specification](dashboard_spec.md).
## Intended use and prohibited use
Intended uses are to validate the data-engineering and modeling pipeline,
compare leakage-safe model candidates, study aggregate patterns, and support a
clearly marked development dashboard preview.
Do not use the model to:
- make decisions about an individual vehicle, owner, registration, inspection,
or station;
- diagnose a vehicle, guarantee an inspection result, or infer safety or
roadworthiness;
- rank or penalize people, counties, programs, or inspection stations;
- make causal claims from observed associations; or
- report statewide prevalence, county rankings, or production performance from
the current development sample.
The public estimator remains disabled because no privacy-reviewed prediction
lookup has been approved.
## Prediction unit and target
The prediction is made immediately before a new inspection episode begins. An
episode groups consecutive attempts no more than 30 days apart, and only its
first attempt is the supervised target. A vehicle must have at least one prior
completed episode to enter the returning-vehicle cohort.
The binary target is:
- `0`: recognized pass;
- `1`: recognized fail, reject, or abort; and
- unlabeled: blank, null, or unrecognized results.
A recognized overall result is preferred. For the older Utah County OBD feed,
a controlled OBD result may fill a blank overall result only under the narrow
source/program/test contract documented in the
[modeling protocol](modeling_protocol.md#source-specific-label-contract). That
proxy is approved only for binary pass versus non-pass; it must not support a
four-class interpretation. Label provenance is private audit metadata and is
not a predictor.
## Development data and representativeness
The current extract samples inspection-table pages and then retrieves complete
histories for up to 10,000 vehicles. This over-represents vehicles with more
inspection records. The run contains 83,552 retained source events, 69,588
episodes, and 44,659 eligible returning targets, but those counts do not make
the sample population-representative.
Coverage is limited to participating county/source feeds and changes over time.
It does not represent all 29 Utah counties. Source-system transitions, missing
years, and partial periods can resemble real changes in risk. See the
[data inventory](data_inventory.md) and
[development results](development_results.md) for the audited scope.
## Features and leakage exclusions
The selected model uses information available before the target episode:
- vehicle age;
- prior episode and attempt counts;
- days since the prior episode and prior adverse outcome;
- prior non-pass rate;
- prior first and final outcomes;
- previously observed make and model;
- public county; and
- target season.
Categorical mappings and all preprocessing are fit on training data only. The
current model does not yet include DMV enrichment or rich same-test OBD fields.
Excluded inputs include direct or pseudonymous identifiers, station
information, the target attempt's result or diagnostics, later attempts in the
target episode, future records, full-history aggregates, label provenance, and
preprocessing learned from evaluation periods. The complete exclusion contract
is in [modeling_protocol.md](modeling_protocol.md#leakage-exclusions).
## Chronology and holdout status
The fixed temporal design is:
| Period | Role |
| --- | --- |
| 2010-2015 | Historical context only |
| 2016-2022 | Fit preprocessing and models |
| 2023 | Select hyperparameters and compare candidates |
| 2024 | Fit probability calibration and inspect calibration behavior |
| 2025 | One-time development holdout |
| 2026 partial | Shadow monitoring only |
During live verification, the explicit 2025 gate was opened once after both
candidate specifications and the tree search grid had already been fixed from
pre-2025 data. No model was changed in response. The already-fixed logistic
model was then evaluated once for a direct comparison. Consequently, this
page-sampled 2025 cohort is no longer a pristine unseen test, and no further
2025-informed tuning is permitted. Any complete-data 2025 analysis must be
described as confirmatory. The detailed audit trail is in
[development_results.md](development_results.md#one-time-development-holdout).
## Candidate comparison and selected model
The candidates were training prevalence, the literal previous-episode outcome,
regularized logistic regression, and histogram gradient boosting. Results below
use non-audit development rows.
| Evaluation | Model | PR-AUC | Brier | ROC-AUC | Top-10% precision |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tuning | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tuning | Previous episode | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tuning | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2025 one-time holdout | Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
| 2025 one-time holdout | Gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
The selected logistic model uses `C=0.03`. It satisfied explicit convergence
and finite-value checks. Platt scaling was fit on the 2024 calibration cohort;
on that same cohort, PR-AUC was 0.257 and Brier score was 0.0936. Those 2024
values are calibration diagnostics, not independent final performance.
The logistic model remains selected because it beat both simple baselines and
the more complex tree on the one-time development holdout while remaining more
transparent. The separate never-fit vehicle audit also supported the logistic
model, but the development sample is too limited for final generalization or
fairness claims.
## Coverage, fairness, and privacy limitations
- The model applies only to returning vehicles with recognizable labels and
sufficient prior history; cold-start behavior is not established.
- Geography and source era are entangled. Performance may shift when a feed,
county program, vehicle mix, or label process changes.
- Reject and abort outcomes can reflect readiness or process issues rather than
mechanical failure.
- No protected attributes are modeled, but their absence does not establish
fairness. Subgroup sample sizes, errors, and calibration still require review.
- Make/model normalization and complete-data subgroup analysis are unfinished.
- The current probability calibration has not yet received independent,
vehicle-clustered uncertainty analysis or external validation.
Private analytical artifacts remain local. Public assets contain only reviewed,
rounded aggregates that satisfy episode, distinct-vehicle, and binary-class
suppression thresholds. Direct identifiers, private linkage values, raw rows,
and row-level predictions are outside the public contract. See
[dashboard_spec.md](dashboard_spec.md#publication-controls).
## Monitoring
Before any deployment, monitoring must cover source-level volume, label
recognition, outcome prevalence, source/program mix, missingness, unseen
categories, score distributions, PR-AUC, Brier score, and calibration. Reviews
must explicitly separate the Salt Lake source transition, newer rich feeds, DMV
coverage gaps, and the partial 2026 period. Alert thresholds and a response plan
remain to be defined.
## Reproducibility
The pipeline uses fixed temporal boundaries and a fixed random seed. Python
dependencies are pinned in [requirements.txt](../requirements.txt). Private
manifests record input lineage, software/model configuration, convergence, row
reconciliation, and checksums without publishing private paths or values. SQL
transforms and validation queries are versioned under [sql](../sql), model
runners are under [scripts](../scripts), and automated checks are under
[tests](../tests). Normal training commands leave the 2025 evaluation gate
closed unless an explicit flag is supplied.
## Remaining approval gates
1. Run a complete, contiguous, bounded extraction and rebuild the frozen
pipeline without page-sampling bias.
2. Treat complete-data 2025 results as confirmation and reserve a genuinely new
period or external dataset for future unseen evaluation.
3. Complete subgroup/source-era reporting, vehicle-clustered uncertainty,
calibration diagnostics, and fairness review.
4. Finish make/model normalization, episode-gap sensitivity checks, and the
preregistered inspection-only versus DMV-enhanced ablation.
5. Pass privacy and publication review before replacing the development-preview
aggregates or enabling any prediction lookup.
6. Define monitoring thresholds, ownership, rollback criteria, and a model
update policy before operational use.
docs/modeling_protocol.md
+141
View File
@@ -0,0 +1,141 @@
# Leakage-safe modeling protocol
## Prediction unit
Score the **first attempt of the next inspection episode** immediately before
check-in. Consecutive attempts for the same private vehicle token belong to the
same episode when they are no more than 30 days apart. Re-run the analysis with
14- and 45-day gaps as sensitivity checks.
This prevents rapid fail/retest sequences from dominating the target. Those
within-episode attempts belong in the separate failure-to-pass journey analysis.
## Eligibility
- Use 2010-2015 events only as historical context; supervised targets begin in
2016.
- Exclude the four 1990 date outliers.
- Require a recognized first-attempt outcome and at least one prior completed
episode for the returning-vehicle model.
- Keep first-observed vehicles as a separate cold-start cohort.
- Normalize `slc` and `slco` to Salt Lake County for geography while preserving
source era for drift reporting.
- Deduplicate exact uploads and quarantine shared/test/placeholder identifiers
using preregistered rules for impossible conflicts or extreme activity.
- Do not require a DMV match; retain explicit match and staleness indicators.
Every exclusion must appear in a cohort-flow report.
## Source-specific label contract
The normalized target prefers a recognized `overall_result` for every source.
The older Utah County feed is the sole exception: when its overall result is
blank, `obd_result` may supply the binary target only when the source is
`utah`, the program is `obd`, the test type is `OBD`, and the controlled value
is pass, fail, reject, or abort. `B`, blank, TSI, `other/C`, and unknown values
remain unlabeled. Every target retains `target_outcome_label_source` as private
audit metadata, and that field is never a predictor.
This is a binary pass-versus-non-pass proxy, not a four-class substitution.
Across non-Utah feeds where both fields are recognized, an aggregate audit found
99.30% agreement on pass versus non-pass, while the fail/reject distinction was
not interchangeable. Utah program rules also distinguish a readiness rejection
from a failed inspection. Future four-class analysis must therefore require
`target_outcome_label_source = 'overall_result'`. See the official
[Utah inspection requirements](https://dmv.utah.gov/register/inspections/) and
[program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
## Point-in-time features
Every historical window ends strictly before the target episode:
- prior episode and attempt counts;
- previous episode first/final outcome;
- expanding and trailing prior pass/fail/reject/abort counts and rates;
- days since the previous episode and prior adverse result;
- attempts required in prior episodes;
- vehicle age and canonical make/model from prior information;
- physical county and target month/season;
- latest DMV record dated before the target, registration count/recency, fuel,
and DMV match/staleness indicators; and
- missingness indicators.
The stable core model is inspection-only. The DMV-enhanced model is an explicit
ablation because DMV history ends in March 2024 and is nearly absent in 2021.
## Leakage exclusions
The MVP must not use:
- the target attempt's overall/OBD result, result reason, DTC count, MIL or
readiness values, PIDs, visual checks, measurements, certificate, or
calibration fields;
- later attempts or eventual outcome from the target episode;
- station or station-level outcome statistics;
- raw VIN, private token, plate, ZIP, or other identifiers as features;
- target-row vehicle attributes when a prior/static source is available;
- future DMV records;
- full-history aggregates; or
- preprocessing, category mappings, target encodings, or imputation learned
from validation/test data.
Timestamp ties must be resolved before lag/window calculations. Cumulative
windows end at the preceding event or episode.
## Fixed evaluation timeline
| Partition | Target dates | Purpose |
| --- | --- | --- |
| Historical context | 2010-2015 | Lag features only |
| Train | 2016-2022 | Fit preprocessing and models |
| Tune | 2023 | Hyperparameters and selection |
| Calibrate | 2024 | Probability calibration and thresholds |
| Locked test | 2025 | Final reported performance |
| Shadow drift | 2026-01-01 to 2026-06-22 | Monitoring only |
The page-sampled development extract's 2025 gate was opened once during live
verification after both candidate specifications were fixed. It is therefore
a one-time development holdout, not a pristine future test. No model change was
made from it; a later complete-data 2025 run is confirmatory. See
[development_results.md](development_results.md) for the audit trail. The table
continues to define the frozen chronology for a complete extraction.
Repeated vehicles may cross ordinary time partitions because returning-vehicle
prediction is the deployment scenario. Separately reserve 10% of keyed vehicle
buckets as a never-fit VIN audit and report its 2025 performance as an unseen-
vehicle stress test. Never use random row splitting.
## Baselines and candidate model
1. Training prevalence
2. Repeat the previous episode's first outcome
3. Regularized logistic regression with age splines and one-hot categoricals
4. A boosted-tree model
5. Inspection-only versus inspection-plus-DMV ablation
Keep validation and test sets at natural prevalence. If pass rows are sampled
for training, preserve sampling probabilities and recalibrate on the untouched
2024 partition.
## Metrics
Headline binary metrics:
- non-pass PR-AUC;
- Brier score and log loss;
- calibration intercept, slope, and reliability curve;
- precision, recall, and lift at fixed review capacities; and
- ROC-AUC as secondary context.
Use vehicle-clustered bootstrap confidence intervals. Report results by county,
source era, vehicle-age band, prior outcome, fuel, history depth, and DMV
match/staleness. Four-class analysis adds class-specific and macro PR-AUC,
multiclass log loss/Brier score, calibration, and a confusion matrix.
## Drift contract
Monitor monthly/source-level volume, label recognition, outcome prevalence,
blank rate, source/program mix, missingness, DMV staleness, unseen categories,
prediction distribution, PR-AUC, Brier score, and calibration. Treat the
`slc`→`slco` transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and
partial 2026 period as explicit stress cases rather than ordinary random drift.
docs/project_charter.md
+122
View File
@@ -0,0 +1,122 @@
# Utah Vehicle Health project charter
## Working title
**Utah Vehicle Health**
*What millions of emissions inspections reveal about the next test across the
Utah county feeds available in this dataset.*
## Product promise
Explain how vehicle age, type, location, and prior inspection history relate to
the chance of passing the first attempt of the next emissions-inspection
episode.
The product measures **emissions-inspection outcomes**. It must not describe its
score as a diagnosis of overall mechanical reliability, roadworthiness, safety,
or legal compliance.
## Primary research question
> Using only information available before an inspection episode begins, how
> accurately and reliably can we estimate whether its first attempt will pass?
An episode groups attempts for the same internal vehicle token when the gap
between consecutive attempts is 30 days or less. The target is the first attempt
of a new episode, not every rapid retest.
## Primary target
- `0 — pass`: normalized `PASS` or `P`
- `1 — non-pass`: normalized `FAIL`, `F`, `REJECT`, or `ABORT`
- Blank, null, and unrecognized results are unlabeled. They remain in chronology
and coverage reporting but are excluded as supervised targets.
Because reject and abort can reflect process/readiness problems rather than
mechanical failure, the project will also report:
- a four-class pass/fail/reject/abort analysis; and
- a fail-versus-pass sensitivity analysis that excludes reject and abort.
## Scope
The returning-vehicle MVP uses universal inspection-history fields and prior
DMV information where it is point-in-time valid. It covers participating
inspection county/source feeds, not all 29 Utah counties.
The rich OBD/odometer fields are a later, source-specific extension for newer
`slco`, `davis`, and `cache` records. They are not part of the statewide-style
historical baseline.
## Deliverables
1. A reproducible, read-only extraction and private pseudonymization pipeline.
2. A point-in-time episode/feature mart with an auditable exclusion report.
3. Prevalence, previous-outcome, and logistic-regression baselines.
4. One calibrated boosted-tree model and an inspection-only versus DMV-enhanced
ablation.
5. A locked temporal evaluation with subgroup and source-era diagnostics.
6. Versioned, suppressed public aggregates for a four-page Bolt dashboard.
7. A model card and data/methodology page documenting limitations.
## Headline success criteria
- Beat both the training-prevalence and previous-episode-outcome baselines on
2025 PR-AUC and Brier score.
- Produce calibrated probabilities, not just class labels.
- Report performance by county/source era, vehicle-age band, history depth, and
DMV match/staleness.
- Reproduce all published charts from versioned sanitized outputs.
- Export no VIN, plate, ZIP, station, raw JSON, operational record, or
row-level prediction to Bolt.
- Suppress public cells with fewer than 100 eligible inspections or fewer than
10 observations in an outcome or its complement, with complementary
suppression where totals could reveal a hidden cell.
## Initial feasibility result
Under label contract v3, the fixed-seed aggregate-only query in
`sql/10_episode_cohort_feasibility.sql` produced 8,916 eligible
returning-vehicle episodes from 1,730 of 2,000 sampled vehicle histories. The
first-attempt non-pass rate was 11.80%, and the median gap from the prior
episode was about 372 days. Of those targets, 1,732 use the documented Utah OBD
binary proxy because that source leaves its overall-result field blank. This
supports both the episode definition and a calibrated binary model. The sample
is for pipeline feasibility, not a population estimate.
The 2025 sample showed a longer median gap and different outcome mix, reinforcing
the need for source-era drift reporting and a locked chronological test.
## Development holdout status
During live verification, the explicit 2025 gate was opened once on the
page-sampled development extract after both candidate specifications had been
fixed from pre-2025 data. No test-informed model change was made. The frozen
logistic model remains selected, and a later complete-data 2025 analysis will
be treated as confirmatory rather than described as a pristine unseen test.
The audit trail and results are recorded in
[development_results.md](development_results.md).
## Non-goals
- Diagnosing an individual vehicle
- Certifying that a vehicle will pass
- Ranking or accusing inspection stations
- Identifying owners or accepting VIN/plate input
- Making causal claims about county programs
- Treating missing outcomes as passes
## Staged build
1. **Foundation:** episode definition, label normalization, exclusions, secure
extraction, and coverage checks.
2. **Baseline:** inspection-history feature mart and transparent baselines.
3. **Enrichment:** canonical vehicle dimension and point-in-time DMV features.
4. **Modeling:** tree model, calibration, locked test, drift and subgroup audits.
5. **Product:** sanitized aggregate export and Bolt MVP.
6. **Stretch:** failure-to-pass journeys, multiclass probabilities, and a
clearly scoped rich-OBD model.
See [modeling_protocol.md](modeling_protocol.md) and
[dashboard_spec.md](dashboard_spec.md) for the detailed contracts.
docs/project_options.md
+161
View File
@@ -0,0 +1,161 @@
# Project options based on the countydata inventory
Inventory date: 2026-07-15
> **Selected:** Option 1, Utah Vehicle Health. The implementation contract is in
> [project_charter.md](project_charter.md).
## What is actually available
The useful analytical source is the `countydata` PostgreSQL database, which is
about 93.4 GB. Its normalized tables are projections of two large logical
datasets, so their row counts should not be added together:
- 18,009,278 DMV registration/tax records with county, registration date,
make, model, model year, fuel type, registration type/place, expiration and
emissions dates, ZIP codes, and temporary-registration status.
- 19,357,287 inspection records with county/source, timestamp, make, model,
model year, station, result, OBD result, test/program type, and OBD summary.
- Raw inspection JSON adds odometer, fuel, engine, transmission, vehicle type,
readiness monitors, visual checks, DTCs, PIDs, and communication protocol for
newer county/source feeds.
The vehicle histories link well without exposing identifiers: 88.2% of a
10,000-VIN inspection sample had DMV history, and the sampled median was eight
inspection visits per vehicle. In the other direction, 74.2% of sampled DMV
vehicles had an inspection match.
Important limitations:
- DMV coverage is strong in 2016-2020 and 2022 through March 2024, but only 117
rows are dated 2021. Treat 2024 as partial.
- Inspection coverage is strong from 2010 through June 2026, but the four rows
dated 1990 are clear date outliers.
- Raw inspection `overall_result` labels are imbalanced: 71.06% pass, 3.61%
fail, 3.56% reject, 2.31% abort, and 19.45% blank/null. Most missing values
come from the older Utah County feed; its separately encoded OBD result can
support a provenance-tagged binary target, but not the four-class analysis.
- Rich OBD/odometer fields are source-dependent. They are essentially absent
from the older `slc`, `utah`, and `weber` feeds and concentrated in `slco`,
`davis`, and `cache` records from late 2024 onward.
- Categories require cleaning: fuel values differ by case and punctuation, and
inspection make values include aliases such as `TOYOTA`/`TOYOT` and
`CHEVROLET`/`CHEVR`.
## Ranked ideas
### 1. Utah Vehicle Health and Reliability Observatory
Build a model that estimates the chance a vehicle will fail, reject, or abort
its next emissions inspection using only information known before that test:
vehicle age, canonical make/model, county, season, fuel, prior test outcomes,
time since the previous test, and longitudinal DMV history.
The dashboard could include:
- make/model/year reliability scorecards with uncertainty intervals;
- risk-versus-age curves and county comparisons;
- prior-failure and repeat-test patterns;
- calibrated individual what-if estimates without accepting or displaying a
VIN; and
- a methodology/data-quality page showing drift and missingness.
This is the strongest overall option because it combines SQL/data engineering,
entity resolution, longitudinal feature engineering, classification or
survival analysis, model explainability, and a compelling public dashboard.
Guardrail: do not use same-test OBD result, DTC count, or overall result as
features when claiming to predict an outcome before the test. That would be
target leakage. Use a time-based holdout and report PR-AUC, calibration, and
Brier score rather than accuracy alone.
### 2. Failure-to-Pass Journey Analyzer
Follow vehicles forward after a fail or reject and model how many attempts and
how much time it takes to achieve a pass. Compare journeys by vehicle age,
canonical make/model, program, county/source, and failure history using
time-to-event or competing-risk methods.
The dashboard could use journey funnels, transition diagrams, survival curves,
and a cohort comparison tool. This is an unusually good fit for the repeated
histories and can become a major page within option 1 instead of a separate app.
### 3. Utah EV and Hybrid Transition Atlas
Deduplicate repeated registrations into vehicle-by-period snapshots, estimate
electric and hybrid share by county and ZIP, measure transitions in the vehicle
fleet, and forecast adoption scenarios.
The dashboard could use a choropleth, adoption curves, county rankings, and a
make/model explorer. This is visually strong and easy to explain, but the DMV
2021 gap and March 2024 endpoint make honest uncertainty and partial-period
handling essential.
### 4. OBD Early-Warning Lab
Scope the analysis to the richer `slco`, `davis`, and `cache` feeds. Use
odometer, vehicle age, fuel, engine, readiness monitors, communication protocol,
MIL state, and prior history to distinguish likely pass, fail, reject, and
abort outcomes and discover common failure signatures.
The dashboard could show diagnostic pathways, readiness-monitor patterns, and
failure signatures by vehicle cohort. It is technically novel, but it is not a
statewide study because rich-feature coverage begins mainly in 2024-2025.
### 5. Risk-Adjusted Inspection Station Consistency
Fit a hierarchical model of station outcomes after controlling for vehicle age,
make/model, program, county, and time. Use empirical-Bayes shrinkage or control
charts to flag unusual reject, abort, or failure rates and detect process drift.
This is excellent applied statistics and anomaly detection. Public results
should anonymize station identifiers, suppress small groups, show uncertainty,
and describe anomalies as review signals rather than evidence of misconduct.
### 6. AutoClean: Government Vehicle Data Entity Resolution
Use the high VIN match rate as weak supervision to learn canonical make/model
mappings between messy inspection strings and cleaner DMV values. Compare
rules, fuzzy matching, and a supervised ranking model, then quantify how much
normalization improves downstream analytics.
The dashboard could show before/after category fragmentation, match confidence,
and correction examples using non-identifying values. This is a particularly
strong data-engineering project and could also serve as the first pipeline stage
for option 1.
### 7. Inspection Demand Forecast and Operations Dashboard
Forecast daily or weekly test volume by county and station using timestamps,
seasonality, holidays, long-term trend, and recent history. Add change-point and
volume-anomaly detection.
The dashboard could show workload forecasts, day/hour heatmaps, forecast
intervals, and historical disruptions. It is feasible and useful, although it
has less machine-learning depth than the reliability project unless forecasting
and anomaly evaluation are developed carefully.
### 8. Vehicle Longevity and Survival Index
Treat the repeated DMV and inspection histories as censored longitudinal data.
Estimate how long makes, models, fuel types, and model-year cohorts remain
active, using Kaplan-Meier curves and a Cox or gradient-boosted survival model.
The dashboard could answer "Which vehicles stay on Utah roads the longest?"
The main methodological challenge is that disappearance from the data can mean
sale, relocation, incomplete coverage, or retirement, so the result must be
described as observed-system retention rather than mechanical lifespan.
## Recommended project shape
Use option 1 as the main story and option 6 as its data-engineering foundation:
1. Build an incremental, read-only SQL extraction and canonical vehicle model.
2. Create one row per eligible upcoming inspection using only prior information.
3. Compare a transparent logistic baseline with a tree model.
4. Validate on a later time period and separately by county/source.
5. Export only aggregate scorecards, curves, and de-identified model outputs.
6. Build the public Bolt dashboard over those safe outputs.
This gives the project a coherent end-to-end narrative across data engineering,
analytics, machine learning, responsible validation, and product design.