142 lines
6.3 KiB
Markdown
142 lines
6.3 KiB
Markdown
# Leakage-safe modeling protocol
|
|
|
|
## Prediction unit
|
|
|
|
Score the **first attempt of the next inspection episode** immediately before
|
|
check-in. Consecutive attempts for the same private vehicle token belong to the
|
|
same episode when they are no more than 30 days apart. Re-run the analysis with
|
|
14- and 45-day gaps as sensitivity checks.
|
|
|
|
This prevents rapid fail/retest sequences from dominating the target. Those
|
|
within-episode attempts belong in the separate failure-to-pass journey analysis.
|
|
|
|
## Eligibility
|
|
|
|
- Use 2010-2015 events only as historical context; supervised targets begin in
|
|
2016.
|
|
- Exclude the four 1990 date outliers.
|
|
- Require a recognized first-attempt outcome and at least one prior completed
|
|
episode for the returning-vehicle model.
|
|
- Keep first-observed vehicles as a separate cold-start cohort.
|
|
- Normalize `slc` and `slco` to Salt Lake County for geography while preserving
|
|
source era for drift reporting.
|
|
- Deduplicate exact uploads and quarantine shared/test/placeholder identifiers
|
|
using preregistered rules for impossible conflicts or extreme activity.
|
|
- Do not require a DMV match; retain explicit match and staleness indicators.
|
|
|
|
Every exclusion must appear in a cohort-flow report.
|
|
|
|
## Source-specific label contract
|
|
|
|
The normalized target prefers a recognized `overall_result` for every source.
|
|
The older Utah County feed is the sole exception: when its overall result is
|
|
blank, `obd_result` may supply the binary target only when the source is
|
|
`utah`, the program is `obd`, the test type is `OBD`, and the controlled value
|
|
is pass, fail, reject, or abort. `B`, blank, TSI, `other/C`, and unknown values
|
|
remain unlabeled. Every target retains `target_outcome_label_source` as private
|
|
audit metadata, and that field is never a predictor.
|
|
|
|
This is a binary pass-versus-non-pass proxy, not a four-class substitution.
|
|
Across non-Utah feeds where both fields are recognized, an aggregate audit found
|
|
99.30% agreement on pass versus non-pass, while the fail/reject distinction was
|
|
not interchangeable. Utah program rules also distinguish a readiness rejection
|
|
from a failed inspection. Future four-class analysis must therefore require
|
|
`target_outcome_label_source = 'overall_result'`. See the official
|
|
[Utah inspection requirements](https://dmv.utah.gov/register/inspections/) and
|
|
[program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
|
|
|
|
## Point-in-time features
|
|
|
|
Every historical window ends strictly before the target episode:
|
|
|
|
- prior episode and attempt counts;
|
|
- previous episode first/final outcome;
|
|
- expanding and trailing prior pass/fail/reject/abort counts and rates;
|
|
- days since the previous episode and prior adverse result;
|
|
- attempts required in prior episodes;
|
|
- vehicle age and canonical make/model from prior information;
|
|
- physical county and target month/season;
|
|
- latest DMV record dated before the target, registration count/recency, fuel,
|
|
and DMV match/staleness indicators; and
|
|
- missingness indicators.
|
|
|
|
The stable core model is inspection-only. The DMV-enhanced model is an explicit
|
|
ablation because DMV history ends in March 2024 and is nearly absent in 2021.
|
|
|
|
## Leakage exclusions
|
|
|
|
The MVP must not use:
|
|
|
|
- the target attempt's overall/OBD result, result reason, DTC count, MIL or
|
|
readiness values, PIDs, visual checks, measurements, certificate, or
|
|
calibration fields;
|
|
- later attempts or eventual outcome from the target episode;
|
|
- station or station-level outcome statistics;
|
|
- raw VIN, private token, plate, ZIP, or other identifiers as features;
|
|
- target-row vehicle attributes when a prior/static source is available;
|
|
- future DMV records;
|
|
- full-history aggregates; or
|
|
- preprocessing, category mappings, target encodings, or imputation learned
|
|
from validation/test data.
|
|
|
|
Timestamp ties must be resolved before lag/window calculations. Cumulative
|
|
windows end at the preceding event or episode.
|
|
|
|
## Fixed evaluation timeline
|
|
|
|
| Partition | Target dates | Purpose |
|
|
| --- | --- | --- |
|
|
| Historical context | 2010-2015 | Lag features only |
|
|
| Train | 2016-2022 | Fit preprocessing and models |
|
|
| Tune | 2023 | Hyperparameters and selection |
|
|
| Calibrate | 2024 | Probability calibration and thresholds |
|
|
| Locked test | 2025 | Final reported performance |
|
|
| Shadow drift | 2026-01-01 to 2026-06-22 | Monitoring only |
|
|
|
|
The page-sampled development extract's 2025 gate was opened once during live
|
|
verification after both candidate specifications were fixed. It is therefore
|
|
a one-time development holdout, not a pristine future test. No model change was
|
|
made from it; a later complete-data 2025 run is confirmatory. See
|
|
[development_results.md](development_results.md) for the audit trail. The table
|
|
continues to define the frozen chronology for a complete extraction.
|
|
|
|
Repeated vehicles may cross ordinary time partitions because returning-vehicle
|
|
prediction is the deployment scenario. Separately reserve 10% of keyed vehicle
|
|
buckets as a never-fit VIN audit and report its 2025 performance as an unseen-
|
|
vehicle stress test. Never use random row splitting.
|
|
|
|
## Baselines and candidate model
|
|
|
|
1. Training prevalence
|
|
2. Repeat the previous episode's first outcome
|
|
3. Regularized logistic regression with age splines and one-hot categoricals
|
|
4. A boosted-tree model
|
|
5. Inspection-only versus inspection-plus-DMV ablation
|
|
|
|
Keep validation and test sets at natural prevalence. If pass rows are sampled
|
|
for training, preserve sampling probabilities and recalibrate on the untouched
|
|
2024 partition.
|
|
|
|
## Metrics
|
|
|
|
Headline binary metrics:
|
|
|
|
- non-pass PR-AUC;
|
|
- Brier score and log loss;
|
|
- calibration intercept, slope, and reliability curve;
|
|
- precision, recall, and lift at fixed review capacities; and
|
|
- ROC-AUC as secondary context.
|
|
|
|
Use vehicle-clustered bootstrap confidence intervals. Report results by county,
|
|
source era, vehicle-age band, prior outcome, fuel, history depth, and DMV
|
|
match/staleness. Four-class analysis adds class-specific and macro PR-AUC,
|
|
multiclass log loss/Brier score, calibration, and a confusion matrix.
|
|
|
|
## Drift contract
|
|
|
|
Monitor monthly/source-level volume, label recognition, outcome prevalence,
|
|
blank rate, source/program mix, missingness, DMV staleness, unseen categories,
|
|
prediction distribution, PR-AUC, Brier score, and calibration. Treat the
|
|
`slc`→`slco` transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and
|
|
partial 2026 period as explicit stress cases rather than ordinary random drift.
|