Leakage-safe modeling protocol

Prediction unit

Score the first attempt of the next inspection episode immediately before check-in. Consecutive attempts for the same private vehicle token belong to the same episode when they are no more than 30 days apart. Re-run the analysis with 14- and 45-day gaps as sensitivity checks.

This prevents rapid fail/retest sequences from dominating the target. Those within-episode attempts belong in the separate failure-to-pass journey analysis.

Eligibility

Use 2010-2015 events only as historical context; supervised targets begin in 2016.
Exclude the four 1990 date outliers.
Require a recognized first-attempt outcome and at least one prior completed episode for the returning-vehicle model.
Keep first-observed vehicles as a separate cold-start cohort.
Normalize slc and slco to Salt Lake County for geography while preserving source era for drift reporting.
Deduplicate exact uploads and quarantine shared/test/placeholder identifiers using preregistered rules for impossible conflicts or extreme activity.
Do not require a DMV match; retain explicit match and staleness indicators.

Every exclusion must appear in a cohort-flow report.

Source-specific label contract

The normalized target prefers a recognized overall_result for every source. The older Utah County feed is the sole exception: when its overall result is blank, obd_result may supply the binary target only when the source is utah, the program is obd, the test type is OBD, and the controlled value is pass, fail, reject, or abort. B, blank, TSI, other/C, and unknown values remain unlabeled. Every target retains target_outcome_label_source as private audit metadata, and that field is never a predictor.

This is a binary pass-versus-non-pass proxy, not a four-class substitution. Across non-Utah feeds where both fields are recognized, an aggregate audit found 99.30% agreement on pass versus non-pass, while the fail/reject distinction was not interchangeable. Utah program rules also distinguish a readiness rejection from a failed inspection. Future four-class analysis must therefore require target_outcome_label_source = 'overall_result'. See the official Utah inspection requirements and program definition of rejection.

Point-in-time features

Every historical window ends strictly before the target episode:

prior episode and attempt counts;
previous episode first/final outcome;
expanding and trailing prior pass/fail/reject/abort counts and rates;
days since the previous episode and prior adverse result;
attempts required in prior episodes;
vehicle age and canonical make/model from prior information;
physical county and target month/season;
latest DMV record dated before the target, registration count/recency, fuel, and DMV match/staleness indicators; and
missingness indicators.

The stable core model is inspection-only. The DMV-enhanced model is an explicit ablation because DMV history ends in March 2024 and is nearly absent in 2021.

Leakage exclusions

The MVP must not use:

the target attempt's overall/OBD result, result reason, DTC count, MIL or readiness values, PIDs, visual checks, measurements, certificate, or calibration fields;
later attempts or eventual outcome from the target episode;
station or station-level outcome statistics;
raw VIN, private token, plate, ZIP, or other identifiers as features;
target-row vehicle attributes when a prior/static source is available;
future DMV records;
full-history aggregates; or
preprocessing, category mappings, target encodings, or imputation learned from validation/test data.

Timestamp ties must be resolved before lag/window calculations. Cumulative windows end at the preceding event or episode.

Fixed evaluation timeline

Partition	Target dates	Purpose
Historical context	2010-2015	Lag features only
Train	2016-2022	Fit preprocessing and models
Tune	2023	Hyperparameters and selection
Calibrate	2024	Probability calibration and thresholds
Locked test	2025	Final reported performance
Shadow drift	2026-01-01 to 2026-06-22	Monitoring only

The page-sampled development extract's 2025 gate was opened once during live verification after both candidate specifications were fixed. It is therefore a one-time development holdout, not a pristine future test. No model change was made from it; a later complete-data 2025 run is confirmatory. See development_results.md for the audit trail. The table continues to define the frozen chronology for a complete extraction.

Repeated vehicles may cross ordinary time partitions because returning-vehicle prediction is the deployment scenario. Separately reserve 10% of keyed vehicle buckets as a never-fit VIN audit and report its 2025 performance as an unseen- vehicle stress test. Never use random row splitting.

Baselines and candidate model

Training prevalence
Repeat the previous episode's first outcome
Regularized logistic regression with age splines and one-hot categoricals
A boosted-tree model
Inspection-only versus inspection-plus-DMV ablation

Keep validation and test sets at natural prevalence. If pass rows are sampled for training, preserve sampling probabilities and recalibrate on the untouched 2024 partition.

Metrics

Headline binary metrics:

non-pass PR-AUC;
Brier score and log loss;
calibration intercept, slope, and reliability curve;
precision, recall, and lift at fixed review capacities; and
ROC-AUC as secondary context.

Use vehicle-clustered bootstrap confidence intervals. Report results by county, source era, vehicle-age band, prior outcome, fuel, history depth, and DMV match/staleness. Four-class analysis adds class-specific and macro PR-AUC, multiclass log loss/Brier score, calibration, and a confusion matrix.

Drift contract

Monitor monthly/source-level volume, label recognition, outcome prevalence, blank rate, source/program mix, missingness, DMV staleness, unseen categories, prediction distribution, PR-AUC, Brier score, and calibration. Treat the slc→slco transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and partial 2026 period as explicit stress cases rather than ordinary random drift.

6.3 KiB Raw Blame History