initial code

docs/modeling_protocol.md

@@ -0,0 +1,141 @@
+# Leakage-safe modeling protocol
+
+## Prediction unit
+
+Score the **first attempt of the next inspection episode** immediately before
+check-in. Consecutive attempts for the same private vehicle token belong to the
+same episode when they are no more than 30 days apart. Re-run the analysis with
+14- and 45-day gaps as sensitivity checks.
+
+This prevents rapid fail/retest sequences from dominating the target. Those
+within-episode attempts belong in the separate failure-to-pass journey analysis.
+
+## Eligibility
+
+- Use 2010-2015 events only as historical context; supervised targets begin in
+  2016.
+- Exclude the four 1990 date outliers.
+- Require a recognized first-attempt outcome and at least one prior completed
+  episode for the returning-vehicle model.
+- Keep first-observed vehicles as a separate cold-start cohort.
+- Normalize `slc` and `slco` to Salt Lake County for geography while preserving
+  source era for drift reporting.
+- Deduplicate exact uploads and quarantine shared/test/placeholder identifiers
+  using preregistered rules for impossible conflicts or extreme activity.
+- Do not require a DMV match; retain explicit match and staleness indicators.
+
+Every exclusion must appear in a cohort-flow report.
+
+## Source-specific label contract
+
+The normalized target prefers a recognized `overall_result` for every source.
+The older Utah County feed is the sole exception: when its overall result is
+blank, `obd_result` may supply the binary target only when the source is
+`utah`, the program is `obd`, the test type is `OBD`, and the controlled value
+is pass, fail, reject, or abort. `B`, blank, TSI, `other/C`, and unknown values
+remain unlabeled. Every target retains `target_outcome_label_source` as private
+audit metadata, and that field is never a predictor.
+
+This is a binary pass-versus-non-pass proxy, not a four-class substitution.
+Across non-Utah feeds where both fields are recognized, an aggregate audit found
+99.30% agreement on pass versus non-pass, while the fail/reject distinction was
+not interchangeable. Utah program rules also distinguish a readiness rejection
+from a failed inspection. Future four-class analysis must therefore require
+`target_outcome_label_source = 'overall_result'`. See the official
+[Utah inspection requirements](https://dmv.utah.gov/register/inspections/) and
+[program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
+
+## Point-in-time features
+
+Every historical window ends strictly before the target episode:
+
+- prior episode and attempt counts;
+- previous episode first/final outcome;
+- expanding and trailing prior pass/fail/reject/abort counts and rates;
+- days since the previous episode and prior adverse result;
+- attempts required in prior episodes;
+- vehicle age and canonical make/model from prior information;
+- physical county and target month/season;
+- latest DMV record dated before the target, registration count/recency, fuel,
+  and DMV match/staleness indicators; and
+- missingness indicators.
+
+The stable core model is inspection-only. The DMV-enhanced model is an explicit
+ablation because DMV history ends in March 2024 and is nearly absent in 2021.
+
+## Leakage exclusions
+
+The MVP must not use:
+
+- the target attempt's overall/OBD result, result reason, DTC count, MIL or
+  readiness values, PIDs, visual checks, measurements, certificate, or
+  calibration fields;
+- later attempts or eventual outcome from the target episode;
+- station or station-level outcome statistics;
+- raw VIN, private token, plate, ZIP, or other identifiers as features;
+- target-row vehicle attributes when a prior/static source is available;
+- future DMV records;
+- full-history aggregates; or
+- preprocessing, category mappings, target encodings, or imputation learned
+  from validation/test data.
+
+Timestamp ties must be resolved before lag/window calculations. Cumulative
+windows end at the preceding event or episode.
+
+## Fixed evaluation timeline
+
+| Partition | Target dates | Purpose |
+| --- | --- | --- |
+| Historical context | 2010-2015 | Lag features only |
+| Train | 2016-2022 | Fit preprocessing and models |
+| Tune | 2023 | Hyperparameters and selection |
+| Calibrate | 2024 | Probability calibration and thresholds |
+| Locked test | 2025 | Final reported performance |
+| Shadow drift | 2026-01-01 to 2026-06-22 | Monitoring only |
+
+The page-sampled development extract's 2025 gate was opened once during live
+verification after both candidate specifications were fixed. It is therefore
+a one-time development holdout, not a pristine future test. No model change was
+made from it; a later complete-data 2025 run is confirmatory. See
+[development_results.md](development_results.md) for the audit trail. The table
+continues to define the frozen chronology for a complete extraction.
+
+Repeated vehicles may cross ordinary time partitions because returning-vehicle
+prediction is the deployment scenario. Separately reserve 10% of keyed vehicle
+buckets as a never-fit VIN audit and report its 2025 performance as an unseen-
+vehicle stress test. Never use random row splitting.
+
+## Baselines and candidate model
+
+1. Training prevalence
+2. Repeat the previous episode's first outcome
+3. Regularized logistic regression with age splines and one-hot categoricals
+4. A boosted-tree model
+5. Inspection-only versus inspection-plus-DMV ablation
+
+Keep validation and test sets at natural prevalence. If pass rows are sampled
+for training, preserve sampling probabilities and recalibrate on the untouched
+2024 partition.
+
+## Metrics
+
+Headline binary metrics:
+
+- non-pass PR-AUC;
+- Brier score and log loss;
+- calibration intercept, slope, and reliability curve;
+- precision, recall, and lift at fixed review capacities; and
+- ROC-AUC as secondary context.
+
+Use vehicle-clustered bootstrap confidence intervals. Report results by county,
+source era, vehicle-age band, prior outcome, fuel, history depth, and DMV
+match/staleness. Four-class analysis adds class-specific and macro PR-AUC,
+multiclass log loss/Brier score, calibration, and a confusion matrix.
+
+## Drift contract
+
+Monitor monthly/source-level volume, label recognition, outcome prevalence,
+blank rate, source/program mix, missingness, DMV staleness, unseen categories,
+prediction distribution, PR-AUC, Brier score, and calibration. Treat the
+`slc`→`slco` transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and
+partial 2026 period as explicit stress cases rather than ordinary random drift.