Centralized_I-M_Database/SummerProject2026
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

repository is now a presentation-ready development prototype

Browse Source
This commit is contained in:
kevinbell
2026-07-21 15:40:44 -06:00
parent 640165649d
commit 88161a6f16
24 changed files with 1989 additions and 946 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/model_card.md
+118 -154
View File
@@ -1,197 +1,161 @@
# Utah Vehicle Health model card
Last updated: 2026-07-15
Last updated: 2026-07-21
Status: **development prototype; not approved for production or population claims**
Status: **final development-prototype model; not approved for production or
population claims**
> Every count, rate, and performance value in this model card comes from the
> private, page-sampled 10,000-vehicle development cohort. These are sample
> results, not population estimates.
## Model summary
Utah Vehicle Health estimates the probability that the first attempt of a
returning vehicle's next emissions-inspection episode will be a non-pass. The
current selected model is a regularized logistic regression followed by Platt
probability calibration. It is an inspection-history model, not a mechanical
health, safety, roadworthiness, or legal-compliance model.
Utah Vehicle Health estimates whether the first attempt of a returning
vehicle's next emissions-inspection episode will be a non-pass. The final model
is regularized logistic regression (`C=0.03`) followed by Platt probability
calibration. Histogram gradient boosting is retained only as a benchmark.
This model card summarizes the current page-sampled development run. The full
research and product contracts are in the [project charter](project_charter.md),
[modeling protocol](modeling_protocol.md), and
[dashboard specification](dashboard_spec.md).
This is an inspection-history model. It is not a model of overall mechanical
health, safety, roadworthiness, or legal compliance.
## Intended use and prohibited use
## Intended use
Intended uses are to validate the data-engineering and modeling pipeline,
compare leakage-safe model candidates, study aggregate patterns, and support a
clearly marked development dashboard preview.
The development model supports:
Do not use the model to:
- validating leakage-safe longitudinal feature engineering;
- comparing a transparent final model with simple and nonlinear benchmarks;
- explaining calibration and chronological evaluation; and
- powering aggregate, suppression-reviewed development-preview diagnostics.
- make decisions about an individual vehicle, owner, registration, inspection,
or station;
- diagnose a vehicle, guarantee an inspection result, or infer safety or
roadworthiness;
- rank or penalize people, counties, programs, or inspection stations;
- make causal claims from observed associations; or
- report statewide prevalence, county rankings, or production performance from
the current development sample.
Do not use it to:
The public estimator remains disabled because no privacy-reviewed prediction
lookup has been approved.
- decide anything about an individual vehicle, owner, registration, inspection,
county, program, station, or technician;
- diagnose a vehicle or guarantee an inspection outcome;
- rank counties, programs, stations, vehicle cohorts, or people;
- make causal claims from observed sample associations;
- report statewide/county prevalence or production performance; or
- serve a prediction lookup or row-level score.
## Prediction unit and target
The prediction is made immediately before a new inspection episode begins. An
episode groups consecutive attempts no more than 30 days apart, and only its
first attempt is the supervised target. A vehicle must have at least one prior
completed episode to enter the returning-vehicle cohort.
The prediction is made immediately before a new inspection episode begins.
Attempts no more than 30 days apart form one episode. The supervised target is
the episode's first attempt, and a vehicle must have at least one prior completed
episode.
The binary target is:
The binary target is recognized pass (`0`) versus recognized fail, reject, or
abort (`1`, collectively **non-pass**). Blank, null, and unrecognized results
remain unlabeled. Reject and abort may reflect process/readiness conditions, so
non-pass must not be paraphrased as mechanical failure.
- `0`: recognized pass;
- `1`: recognized fail, reject, or abort; and
- unlabeled: blank, null, or unrecognized results.
A recognized overall result is preferred. The documented older Utah County OBD
exception may fill a blank overall result under a narrow source/program/test
contract. That value is a binary proxy only. Its provenance remains private
audit metadata and is not a predictor. See the
[modeling protocol](modeling_protocol.md#binary-label-contract).
A recognized overall result is preferred. For the older Utah County OBD feed,
a controlled OBD result may fill a blank overall result only under the narrow
source/program/test contract documented in the
[modeling protocol](modeling_protocol.md#source-specific-label-contract). That
proxy is approved only for binary pass versus non-pass; it must not support a
four-class interpretation. Label provenance is private audit metadata and is
not a predictor.
## Development data
## Development data and representativeness
The extraction page-samples inspection records and retrieves complete histories
for up to 10,000 vehicles. After validation, the private run retained 9,996
vehicle tokens, 83,552 source events, 69,588 episodes, and 44,659 eligible
returning targets. These sample counts describe pipeline reconciliation only;
they do not imply population coverage.
The current extract samples inspection-table pages and then retrieves complete
histories for up to 10,000 vehicles. This over-represents vehicles with more
inspection records. The run contains 83,552 retained source events, 69,588
episodes, and 44,659 eligible returning targets, but those counts do not make
the sample population-representative.
The sampling design over-represents vehicles with more inspection records. Feed
coverage changes by source and time and does not include all 29 Utah counties.
Accordingly, the model card makes no statewide prevalence claim, no county
ranking, and no claim that sample subgroup differences generalize.
Coverage is limited to participating county/source feeds and changes over time.
It does not represent all 29 Utah counties. Source-system transitions, missing
years, and partial periods can resemble real changes in risk. See the
[data inventory](data_inventory.md) and
[development results](development_results.md) for the audited scope.
## Features and exclusions
## Features and leakage exclusions
The final model uses only pre-episode inspection-history information: vehicle
age, prior episode/attempt counts, time since prior history, prior non-pass rate,
prior first/final outcomes, previously observed make/model, public county
context, and season.
The selected model uses information available before the target episode:
The model excludes current-test results and diagnostics, later attempts, station
and technician information, direct/pseudonymous identifiers, future records,
full-history leakage, DMV enrichment, and preprocessing learned from evaluation
periods. It does not use VIN, plate, ZIP, private token, raw JSON, credentials,
or operational records. The complete contract is in
[modeling_protocol.md](modeling_protocol.md#leakage-and-privacy-exclusions).
- vehicle age;
- prior episode and attempt counts;
- days since the prior episode and prior adverse outcome;
- prior non-pass rate;
- prior first and final outcomes;
- previously observed make and model;
- public county; and
- target season.
## Chronological evaluation
Categorical mappings and all preprocessing are fit on training data only. The
current model does not yet include DMV enrichment or rich same-test OBD fields.
Excluded inputs include direct or pseudonymous identifiers, station
information, the target attempt's result or diagnostics, later attempts in the
target episode, future records, full-history aggregates, label provenance, and
preprocessing learned from evaluation periods. The complete exclusion contract
is in [modeling_protocol.md](modeling_protocol.md#leakage-exclusions).
## Chronology and holdout status
The fixed temporal design is:
| Period | Role |
| Period | Development-sample role |
| --- | --- |
| 2010-2015 | Historical context only |
| 2016-2022 | Fit preprocessing and models |
| 2023 | Select hyperparameters and compare candidates |
| 2024 | Fit probability calibration and inspect calibration behavior |
| 2025 | One-time development holdout |
| 2026 partial | Shadow monitoring only |
| 2016-2022 | Fit preprocessing and logistic parameters |
| 2023 | Select regularization and compare fixed candidates |
| 2024 | Fit Platt scaling and inspect calibration behavior |
| 2025 | One-time development-sample holdout |
| 2026 partial | Shadow context only |
During live verification, the explicit 2025 gate was opened once after both
candidate specifications and the tree search grid had already been fixed from
pre-2025 data. No model was changed in response. The already-fixed logistic
model was then evaluated once for a direct comparison. Consequently, this
page-sampled 2025 cohort is no longer a pristine unseen test, and no further
2025-informed tuning is permitted. Any complete-data 2025 analysis must be
described as confirmatory. The detailed audit trail is in
[development_results.md](development_results.md#one-time-development-holdout).
The explicit 2025 gate was opened once after specifications were frozen from
pre-2025 data. No model was changed in response. This makes 2025 a one-time
development holdout, not a pristine future test. No further 2025-informed tuning
is permitted.
## Candidate comparison and selected model
## Development-sample performance
The candidates were training prevalence, the literal previous-episode outcome,
regularized logistic regression, and histogram gradient boosting. Results below
use non-audit development rows.
The following values use non-audit rows from the **private 10,000-vehicle
development sample**. They are **not population performance estimates**.
| Evaluation | Model | PR-AUC | Brier | ROC-AUC | Top-10% precision |
| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tuning | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tuning | Previous episode | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tuning | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2025 one-time holdout | Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
| 2025 one-time holdout | Gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
| 2023 tune | Training-prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
The selected logistic model uses `C=0.03`. It satisfied explicit convergence
and finite-value checks. Platt scaling was fit on the 2024 calibration cohort;
on that same cohort, PR-AUC was 0.257 and Brier score was 0.0936. Those 2024
values are calibration diagnostics, not independent final performance.
Platt scaling was fit on the 2024 development-sample calibration partition. Its
same-partition PR-AUC of 0.257 and Brier score of 0.0936 are calibration
diagnostics, not independent final performance.
The logistic model remains selected because it beat both simple baselines and
the more complex tree on the one-time development holdout while remaining more
transparent. The separate never-fit vehicle audit also supported the logistic
model, but the development sample is too limited for final generalization or
fairness claims.
The final logistic model was retained because it beat the two simple baselines
on headline development metrics and outperformed the more complex tree in the
one-time sample holdout. The benchmark did not justify added complexity.
## Coverage, fairness, and privacy limitations
The never-fit private-vehicle stress-test subset contained 480 eligible 2025
sample episodes from 450 vehicles; calibrated logistic PR-AUC was 0.253 and
Brier score was 0.0859. These are also private-sample diagnostics, not
population or external-validation results.
- The model applies only to returning vehicles with recognizable labels and
sufficient prior history; cold-start behavior is not established.
- Geography and source era are entangled. Performance may shift when a feed,
county program, vehicle mix, or label process changes.
- Reject and abort outcomes can reflect readiness or process issues rather than
mechanical failure.
- No protected attributes are modeled, but their absence does not establish
fairness. Subgroup sample sizes, errors, and calibration still require review.
- Make/model normalization and complete-data subgroup analysis are unfinished.
- The current probability calibration has not yet received independent,
vehicle-clustered uncertainty analysis or external validation.
## Limitations
Private analytical artifacts remain local. Public assets contain only reviewed,
rounded aggregates that satisfy episode, distinct-vehicle, and binary-class
suppression thresholds. Direct identifiers, private linkage values, raw rows,
and row-level predictions are outside the public contract. See
[dashboard_spec.md](dashboard_spec.md#publication-controls).
- Results apply only to the sampled returning-vehicle cohort with recognizable
labels and sufficient prior history.
- Page sampling is not population-representative.
- Source, program, time, and geography are entangled.
- Reject and abort are heterogeneous non-pass outcomes.
- Make/model aliases and incomplete feed coverage can distort cohorts.
- The one-time 2025 holdout is not an external validation dataset.
- No protected attributes are modeled, but that does not establish fairness.
- Sample calibration does not establish production calibration.
## Monitoring
## Privacy and public release
Before any deployment, monitoring must cover source-level volume, label
recognition, outcome prevalence, source/program mix, missingness, unseen
categories, score distributions, PR-AUC, Brier score, and calibration. Reviews
must explicitly separate the Salt Lake source transition, newer rich feeds, DMV
coverage gaps, and the partial 2026 period. Alert thresholds and a response plan
remain to be defined.
Private analytical artifacts remain local and Git-ignored. Dashboard files
contain only rounded aggregate cells that pass episode, distinct-vehicle, and
binary-class suppression. The browser validates publication flags and checksums
and fails closed.
No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
credential, operational record, or row-level prediction belongs in a public
asset. The finished dashboard has no vehicle-level prediction service. See the
[dashboard specification](dashboard_spec.md#suppression-and-privacy-controls).
## Reproducibility
The pipeline uses fixed temporal boundaries and a fixed random seed. Python
dependencies are pinned in [requirements.txt](../requirements.txt). Private
manifests record input lineage, software/model configuration, convergence, row
reconciliation, and checksums without publishing private paths or values. SQL
transforms and validation queries are versioned under [sql](../sql), model
runners are under [scripts](../scripts), and automated checks are under
[tests](../tests). Normal training commands leave the 2025 evaluation gate
closed unless an explicit flag is supplied.
The pipeline uses fixed chronological boundaries and a fixed seed. Private
manifests bind input lineage, configuration, convergence checks, row
reconciliation, and checksums. Normal training commands keep the 2025 gate
closed unless the explicit evaluation flag is supplied.
## Remaining approval gates
1. Run a complete, contiguous, bounded extraction and rebuild the frozen
pipeline without page-sampling bias.
2. Treat complete-data 2025 results as confirmation and reserve a genuinely new
period or external dataset for future unseen evaluation.
3. Complete subgroup/source-era reporting, vehicle-clustered uncertainty,
calibration diagnostics, and fairness review.
4. Finish make/model normalization, episode-gap sensitivity checks, and the
preregistered inspection-only versus DMV-enhanced ablation.
5. Pass privacy and publication review before replacing the development-preview
aggregates or enabling any prediction lookup.
6. Define monitoring thresholds, ownership, rollback criteria, and a model
update policy before operational use.
The detailed sample audit is in
[development_results.md](development_results.md), and the narrative conclusion
is in [final_report.md](final_report.md).