# Leakage-safe modeling protocol

## Prediction unit

Score the **first attempt of the next inspection episode** immediately before
check-in. Consecutive attempts for the same private vehicle token belong to the
same episode when they are no more than 30 days apart. Re-run the analysis with
14- and 45-day gaps as sensitivity checks.

This prevents rapid fail/retest sequences from dominating the target. Those
within-episode attempts belong in the separate failure-to-pass journey analysis.

## Eligibility

- Use 2010-2015 events only as historical context; supervised targets begin in
  2016.
- Exclude the four 1990 date outliers.
- Require a recognized first-attempt outcome and at least one prior completed
  episode for the returning-vehicle model.
- Keep first-observed vehicles as a separate cold-start cohort.
- Normalize `slc` and `slco` to Salt Lake County for geography while preserving
  source era for drift reporting.
- Deduplicate exact uploads and quarantine shared/test/placeholder identifiers
  using preregistered rules for impossible conflicts or extreme activity.
- Do not require a DMV match; retain explicit match and staleness indicators.

Every exclusion must appear in a cohort-flow report.

## Source-specific label contract

The normalized target prefers a recognized `overall_result` for every source.
The older Utah County feed is the sole exception: when its overall result is
blank, `obd_result` may supply the binary target only when the source is
`utah`, the program is `obd`, the test type is `OBD`, and the controlled value
is pass, fail, reject, or abort. `B`, blank, TSI, `other/C`, and unknown values
remain unlabeled. Every target retains `target_outcome_label_source` as private
audit metadata, and that field is never a predictor.

This is a binary pass-versus-non-pass proxy, not a four-class substitution.
Across non-Utah feeds where both fields are recognized, an aggregate audit found
99.30% agreement on pass versus non-pass, while the fail/reject distinction was
not interchangeable. Utah program rules also distinguish a readiness rejection
from a failed inspection. Future four-class analysis must therefore require
`target_outcome_label_source = 'overall_result'`. See the official
[Utah inspection requirements](https://dmv.utah.gov/register/inspections/) and
[program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).

## Point-in-time features

Every historical window ends strictly before the target episode:

- prior episode and attempt counts;
- previous episode first/final outcome;
- expanding and trailing prior pass/fail/reject/abort counts and rates;
- days since the previous episode and prior adverse result;
- attempts required in prior episodes;
- vehicle age and canonical make/model from prior information;
- physical county and target month/season;
- latest DMV record dated before the target, registration count/recency, fuel,
  and DMV match/staleness indicators; and
- missingness indicators.

The stable core model is inspection-only. The DMV-enhanced model is an explicit
ablation because DMV history ends in March 2024 and is nearly absent in 2021.

## Leakage exclusions

The MVP must not use:

- the target attempt's overall/OBD result, result reason, DTC count, MIL or
  readiness values, PIDs, visual checks, measurements, certificate, or
  calibration fields;
- later attempts or eventual outcome from the target episode;
- station or station-level outcome statistics;
- raw VIN, private token, plate, ZIP, or other identifiers as features;
- target-row vehicle attributes when a prior/static source is available;
- future DMV records;
- full-history aggregates; or
- preprocessing, category mappings, target encodings, or imputation learned
  from validation/test data.

Timestamp ties must be resolved before lag/window calculations. Cumulative
windows end at the preceding event or episode.

## Fixed evaluation timeline

| Partition | Target dates | Purpose |
| --- | --- | --- |
| Historical context | 2010-2015 | Lag features only |
| Train | 2016-2022 | Fit preprocessing and models |
| Tune | 2023 | Hyperparameters and selection |
| Calibrate | 2024 | Probability calibration and thresholds |
| Locked test | 2025 | Final reported performance |
| Shadow drift | 2026-01-01 to 2026-06-22 | Monitoring only |

The page-sampled development extract's 2025 gate was opened once during live
verification after both candidate specifications were fixed. It is therefore
a one-time development holdout, not a pristine future test. No model change was
made from it; a later complete-data 2025 run is confirmatory. See
[development_results.md](development_results.md) for the audit trail. The table
continues to define the frozen chronology for a complete extraction.

Repeated vehicles may cross ordinary time partitions because returning-vehicle
prediction is the deployment scenario. Separately reserve 10% of keyed vehicle
buckets as a never-fit VIN audit and report its 2025 performance as an unseen-
vehicle stress test. Never use random row splitting.

## Baselines and candidate model

1. Training prevalence
2. Repeat the previous episode's first outcome
3. Regularized logistic regression with age splines and one-hot categoricals
4. A boosted-tree model
5. Inspection-only versus inspection-plus-DMV ablation

Keep validation and test sets at natural prevalence. If pass rows are sampled
for training, preserve sampling probabilities and recalibrate on the untouched
2024 partition.

## Metrics

Headline binary metrics:

- non-pass PR-AUC;
- Brier score and log loss;
- calibration intercept, slope, and reliability curve;
- precision, recall, and lift at fixed review capacities; and
- ROC-AUC as secondary context.

Use vehicle-clustered bootstrap confidence intervals. Report results by county,
source era, vehicle-age band, prior outcome, fuel, history depth, and DMV
match/staleness. Four-class analysis adds class-specific and macro PR-AUC,
multiclass log loss/Brier score, calibration, and a confusion matrix.

## Drift contract

Monitor monthly/source-level volume, label recognition, outcome prevalence,
blank rate, source/program mix, missingness, DMV staleness, unseen categories,
prediction distribution, PR-AUC, Brier score, and calibration. Treat the
`slc`→`slco` transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and
partial 2026 period as explicit stress cases rather than ordinary random drift.