Centralized_I-M_Database/SummerProject2026
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

repository is now a presentation-ready development prototype

Browse Source
This commit is contained in:
kevinbell
2026-07-21 15:40:44 -06:00
parent 640165649d
commit 88161a6f16
24 changed files with 1989 additions and 946 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/modeling_protocol.md
+95 -103
View File
@@ -1,141 +1,133 @@
# Leakage-safe modeling protocol
## Reporting boundary
This protocol governs the private, page-sampled 10,000-vehicle development
cohort. All resulting counts, rates, comparisons, and performance metrics are
**sample results, not population estimates**. County/source reporting is for
coverage and drift context only; it must not be turned into population county
rankings.
## Prediction unit
Score the **first attempt of the next inspection episode** immediately before
check-in. Consecutive attempts for the same private vehicle token belong to the
same episode when they are no more than 30 days apart. Re-run the analysis with
14- and 45-day gaps as sensitivity checks.
Estimate the probability that the **first attempt of a returning vehicle's next
inspection episode is a non-pass**, immediately before the episode begins.
Consecutive attempts for the same private vehicle token form one episode when
they are no more than 30 days apart. Rapid retests are not separate targets.
This prevents rapid fail/retest sequences from dominating the target. Those
within-episode attempts belong in the separate failure-to-pass journey analysis.
This is the sole research target for the completed prototype. Cold-start,
four-class, fail-only, failure-to-pass, and individualized prediction tasks are
out of scope.
## Eligibility
- Use 2010-2015 events only as historical context; supervised targets begin in
2016.
- Exclude the four 1990 date outliers.
- Require a recognized first-attempt outcome and at least one prior completed
episode for the returning-vehicle model.
- Keep first-observed vehicles as a separate cold-start cohort.
- Exclude the four known 1990 date outliers.
- Require a recognized first-attempt binary outcome and at least one prior
completed episode.
- Keep first-observed vehicles outside the modeled returning-vehicle cohort.
- Normalize `slc` and `slco` to Salt Lake County for geography while preserving
source era for drift reporting.
- Deduplicate exact uploads and quarantine shared/test/placeholder identifiers
using preregistered rules for impossible conflicts or extreme activity.
- Do not require a DMV match; retain explicit match and staleness indicators.
source era for diagnostics.
- Deduplicate exact uploads and quarantine shared, test, or placeholder
identifiers under preregistered conflict/activity rules.
Every exclusion must appear in a cohort-flow report.
Every exclusion appears in the private cohort-flow audit.
## Source-specific label contract
## Binary label contract
The normalized target prefers a recognized `overall_result` for every source.
The older Utah County feed is the sole exception: when its overall result is
blank, `obd_result` may supply the binary target only when the source is
`utah`, the program is `obd`, the test type is `OBD`, and the controlled value
is pass, fail, reject, or abort. `B`, blank, TSI, `other/C`, and unknown values
remain unlabeled. Every target retains `target_outcome_label_source` as private
audit metadata, and that field is never a predictor.
The target is `0` for a recognized pass and `1` for recognized fail, reject, or
abort. Blank, null, and unrecognized values remain unlabeled rather than being
treated as passes.
This is a binary pass-versus-non-pass proxy, not a four-class substitution.
Across non-Utah feeds where both fields are recognized, an aggregate audit found
99.30% agreement on pass versus non-pass, while the fail/reject distinction was
not interchangeable. Utah program rules also distinguish a readiness rejection
from a failed inspection. Future four-class analysis must therefore require
`target_outcome_label_source = 'overall_result'`. See the official
[Utah inspection requirements](https://dmv.utah.gov/register/inspections/) and
[program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
The normalized target prefers a recognized `overall_result`. The older Utah
County feed is the sole exception: when its overall result is blank,
`obd_result` may supply the binary label only when source is `utah`, program is
`obd`, test type is `OBD`, and the controlled value is pass, fail, reject, or
abort. `B`, blank, TSI, `other/C`, and unknown values remain unlabeled. Private
label provenance is retained for audit and is never a predictor or public
field.
This exception is a binary pass/non-pass proxy. It does not justify interpreting
fail, reject, and abort as interchangeable mechanical conditions.
## Point-in-time features
Every historical window ends strictly before the target episode:
Every feature window ends strictly before the target episode. The final
inspection-history feature set includes:
- vehicle age;
- prior episode and attempt counts;
- previous episode first/final outcome;
- expanding and trailing prior pass/fail/reject/abort counts and rates;
- days since the previous episode and prior adverse result;
- attempts required in prior episodes;
- vehicle age and canonical make/model from prior information;
- physical county and target month/season;
- latest DMV record dated before the target, registration count/recency, fuel,
and DMV match/staleness indicators; and
- missingness indicators.
- days since the prior episode and prior adverse outcome;
- prior non-pass rate;
- prior episode first and final outcomes;
- previously observed make and model;
- public county context; and
- target season.
The stable core model is inspection-only. The DMV-enhanced model is an explicit
ablation because DMV history ends in March 2024 and is nearly absent in 2021.
The final prototype does not use DMV enrichment or rich current-test OBD data.
## Leakage exclusions
## Leakage and privacy exclusions
The MVP must not use:
The model and public product exclude:
- the target attempt's overall/OBD result, result reason, DTC count, MIL or
readiness values, PIDs, visual checks, measurements, certificate, or
calibration fields;
- the target attempt's result, reason, DTC count, MIL/readiness state, PIDs,
visual checks, measurements, certificate, or calibration fields;
- later attempts or eventual outcome from the target episode;
- station or station-level outcome statistics;
- raw VIN, private token, plate, ZIP, or other identifiers as features;
- target-row vehicle attributes when a prior/static source is available;
- future DMV records;
- full-history aggregates; or
- preprocessing, category mappings, target encodings, or imputation learned
from validation/test data.
- station, technician, or station-level outcome information;
- raw VIN, private vehicle token, plate, ZIP, exact address, or another direct
or pseudonymous identifier as a feature;
- future records, full-history aggregates, or target-row information that was
not available before the episode;
- preprocessing, category mappings, imputation, or calibration learned from a
later partition; and
- raw JSON, credentials, operational records, and row-level predictions in any
public artifact.
Timestamp ties must be resolved before lag/window calculations. Cumulative
windows end at the preceding event or episode.
Timestamp ties are resolved before lag/window calculations. Cumulative windows
end at the preceding event or episode.
## Fixed evaluation timeline
## Frozen chronology
| Partition | Target dates | Purpose |
| Partition | Target dates | Role in the private 10,000-vehicle development sample |
| --- | --- | --- |
| Historical context | 2010-2015 | Lag features only |
| Train | 2016-2022 | Fit preprocessing and models |
| Tune | 2023 | Hyperparameters and selection |
| Calibrate | 2024 | Probability calibration and thresholds |
| Locked test | 2025 | Final reported performance |
| Shadow drift | 2026-01-01 to 2026-06-22 | Monitoring only |
| Train | 2016-2022 | Fit preprocessing and model parameters |
| Tune | 2023 | Select logistic regularization; compare fixed candidates |
| Calibrate | 2024 | Fit Platt scaling and inspect calibration behavior |
| One-time holdout | 2025 | Development comparison after specifications were frozen |
| Shadow | 2026-01-01 to 2026-06-22 | Partial-period monitoring context only |
The page-sampled development extract's 2025 gate was opened once during live
verification after both candidate specifications were fixed. It is therefore
a one-time development holdout, not a pristine future test. No model change was
made from it; a later complete-data 2025 run is confirmatory. See
[development_results.md](development_results.md) for the audit trail. The table
continues to define the frozen chronology for a complete extraction.
The 2025 gate was opened once during live verification. It is no longer a
pristine unseen test, no 2025-informed tuning is permitted, and every reported
2025 value must say “one-time development-sample holdout.” A later complete-data
2025 analysis would be confirmatory, not a new final test.
Repeated vehicles may cross ordinary time partitions because returning-vehicle
prediction is the deployment scenario. Separately reserve 10% of keyed vehicle
buckets as a never-fit VIN audit and report its 2025 performance as an unseen-
vehicle stress test. Never use random row splitting.
Repeated vehicles may cross chronological partitions because returning-vehicle
prediction is the intended scenario. Separately held-out private vehicle buckets
form a never-fit stress test. Random row splitting is prohibited.
## Baselines and candidate model
## Models and decision rule
1. Training prevalence
2. Repeat the previous episode's first outcome
3. Regularized logistic regression with age splines and one-hot categoricals
4. A boosted-tree model
5. Inspection-only versus inspection-plus-DMV ablation
1. Training prevalence: probability baseline.
2. Previous episode's first outcome: literal history baseline.
3. Regularized logistic regression: selected using the 2023 sample partition,
then Platt-calibrated on the 2024 sample partition; **the final model**.
4. Histogram gradient boosting with Platt calibration: fixed nonlinear
**benchmark only**.
Keep validation and test sets at natural prevalence. If pass rows are sampled
for training, preserve sampling probabilities and recalibrate on the untouched
2024 partition.
The benchmark does not power a separate dashboard experience. Added complexity
would require a clear, predeclared improvement to displace the logistic model;
the one-time 2025 sample comparison did not show that improvement.
## Metrics
## Metrics and interpretation
Headline binary metrics:
Headline metrics are non-pass PR-AUC and Brier score. Log loss, ROC-AUC, and
precision at the highest-scored 10% are supporting diagnostics. Calibration
values computed on the 2024 partition used to fit Platt scaling are calibration
diagnostics, not independent final performance.
- non-pass PR-AUC;
- Brier score and log loss;
- calibration intercept, slope, and reliability curve;
- precision, recall, and lift at fixed review capacities; and
- ROC-AUC as secondary context.
Use vehicle-clustered bootstrap confidence intervals. Report results by county,
source era, vehicle-age band, prior outcome, fuel, history depth, and DMV
match/staleness. Four-class analysis adds class-specific and macro PR-AUC,
multiclass log loss/Brier score, calibration, and a confusion matrix.
## Drift contract
Monitor monthly/source-level volume, label recognition, outcome prevalence,
blank rate, source/program mix, missingness, DMV staleness, unseen categories,
prediction distribution, PR-AUC, Brier score, and calibration. Treat the
`slc`→`slco` transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and
partial 2026 period as explicit stress cases rather than ordinary random drift.
All tables and charts must state that they use the private 10,000-vehicle
development sample and are not population estimates. Report sample support with
every subgroup result. Do not describe observed differences as causal, do not
claim generalization to all Utah vehicles, and do not publish county rankings.