SummerProject2026/docs/modeling_protocol.md

Leakage-safe modeling protocol

Reporting boundary

This protocol governs the private, page-sampled 10,000-vehicle development cohort. All resulting counts, rates, comparisons, and performance metrics are sample results, not population estimates. County/source reporting is for coverage and drift context only; it must not be turned into population county rankings.

Prediction unit

Estimate the probability that the first attempt of a returning vehicle's next inspection episode is a non-pass, immediately before the episode begins. Consecutive attempts for the same private vehicle token form one episode when they are no more than 30 days apart. Rapid retests are not separate targets.

This is the sole research target for the completed prototype. Cold-start, four-class, fail-only, failure-to-pass, and individualized prediction tasks are out of scope.

Eligibility

• Use 2010-2015 events only as historical context; supervised targets begin in 2016.
• Exclude the four known 1990 date outliers.
• Require a recognized first-attempt binary outcome and at least one prior completed episode.
• Keep first-observed vehicles outside the modeled returning-vehicle cohort.
• Normalize slc and slco to Salt Lake County for geography while preserving source era for diagnostics.
• Deduplicate exact uploads and quarantine shared, test, or placeholder identifiers under preregistered conflict/activity rules.

Every exclusion appears in the private cohort-flow audit.

Binary label contract

The target is 0 for a recognized pass and 1 for recognized fail, reject, or abort. Blank, null, and unrecognized values remain unlabeled rather than being treated as passes.

The normalized target prefers a recognized overall_result. The older Utah County feed is the sole exception: when its overall result is blank, obd_result may supply the binary label only when source is utah, program is obd, test type is OBD, and the controlled value is pass, fail, reject, or abort. B, blank, TSI, other/C, and unknown values remain unlabeled. Private label provenance is retained for audit and is never a predictor or public field.

This exception is a binary pass/non-pass proxy. It does not justify interpreting fail, reject, and abort as interchangeable mechanical conditions.

Point-in-time features

Every feature window ends strictly before the target episode. The final inspection-history feature set includes:

• vehicle age;
• prior episode and attempt counts;
• days since the prior episode and prior adverse outcome;
• prior non-pass rate;
• prior episode first and final outcomes;
• previously observed make and model;
• public county context; and
• target season.

The final prototype does not use DMV enrichment or rich current-test OBD data.

Leakage and privacy exclusions

The model and public product exclude:

• the target attempt's result, reason, DTC count, MIL/readiness state, PIDs, visual checks, measurements, certificate, or calibration fields;
• later attempts or eventual outcome from the target episode;
• station, technician, or station-level outcome information;
• raw VIN, private vehicle token, plate, ZIP, exact address, or another direct or pseudonymous identifier as a feature;
• future records, full-history aggregates, or target-row information that was not available before the episode;
• preprocessing, category mappings, imputation, or calibration learned from a later partition; and
• raw JSON, credentials, operational records, and row-level predictions in any public artifact.

Timestamp ties are resolved before lag/window calculations. Cumulative windows end at the preceding event or episode.

Frozen chronology

Partition	Target dates	Role in the private 10,000-vehicle development sample
Historical context	2010-2015	Lag features only
Train	2016-2022	Fit preprocessing and model parameters
Tune	2023	Select logistic regularization; compare fixed candidates
Calibrate	2024	Fit Platt scaling and inspect calibration behavior
One-time holdout	2025	Development comparison after specifications were frozen
Shadow	2026-01-01 to 2026-06-22	Partial-period monitoring context only

The 2025 gate was opened once during live verification. It is no longer a pristine unseen test, no 2025-informed tuning is permitted, and every reported 2025 value must say “one-time development-sample holdout.” A later complete-data 2025 analysis would be confirmatory, not a new final test.

Repeated vehicles may cross chronological partitions because returning-vehicle prediction is the intended scenario. Separately held-out private vehicle buckets form a never-fit stress test. Random row splitting is prohibited.

Models and decision rule

1. Training prevalence: probability baseline.
2. Previous episode's first outcome: literal history baseline.
3. Regularized logistic regression: selected using the 2023 sample partition, then Platt-calibrated on the 2024 sample partition; the final model.
4. Histogram gradient boosting with Platt calibration: fixed nonlinear benchmark only.

The benchmark does not power a separate dashboard experience. Added complexity would require a clear, predeclared improvement to displace the logistic model; the one-time 2025 sample comparison did not show that improvement.

Metrics and interpretation

Headline metrics are non-pass PR-AUC and Brier score. Log loss, ROC-AUC, and precision at the highest-scored 10% are supporting diagnostics. Calibration values computed on the 2024 partition used to fit Platt scaling are calibration diagnostics, not independent final performance.

All tables and charts must state that they use the private 10,000-vehicle development sample and are not population estimates. Report sample support with every subgroup result. Do not describe observed differences as causal, do not claim generalization to all Utah vehicles, and do not publish county rankings.