137 lines
6.7 KiB
Markdown
137 lines
6.7 KiB
Markdown
# Development run results
|
||
|
||
Run date: 2026-07-15
|
||
|
||
These results validate the engineering and modeling pipeline. They are **not
|
||
population estimates**: the development extract samples inspection-table pages
|
||
and then retrieves complete histories for up to 10,000 vehicles, which
|
||
over-represents vehicles with more inspection records.
|
||
|
||
## Label audit
|
||
|
||
The raw Utah County feed leaves `overall_result` blank for 3,747,862 records.
|
||
An aggregate-only server audit found 3,469,102 Utah `obd`/`OBD` records with a
|
||
controlled pass, fail, reject, or abort value in `obd_result`. Across non-Utah
|
||
feeds where both fields were recognized, the two fields agreed on binary pass
|
||
versus non-pass 99.30% of the time, but their fail-versus-reject categories were
|
||
not interchangeable.
|
||
|
||
Label contract v3 therefore permits the Utah OBD value only as a provenance-
|
||
tagged binary proxy. TSI, `other/C`, `B`, blank, and unknown values stay
|
||
unlabeled. Four-class analysis must use `overall_result` exclusively. This also
|
||
matches the official program distinction between a readiness rejection and a
|
||
failed inspection; see the [Utah inspection requirements](https://dmv.utah.gov/register/inspections/)
|
||
and [program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
|
||
|
||
In the development extract, the rule restored 11,334 labels and left 1,281
|
||
events unlabeled. The label-source field remains private audit metadata and is
|
||
not a model feature.
|
||
|
||
## Pipeline reconciliation
|
||
|
||
| Stage | Rows |
|
||
| --- | ---: |
|
||
| Source events written after VIN validation | 83,552 |
|
||
| Clean events after duplicate/conflict handling | 80,190 |
|
||
| Inspection episodes | 69,588 |
|
||
| Eligible returning targets | 44,659 |
|
||
|
||
The sample contains 9,996 retained vehicle tokens. The extraction manifest,
|
||
compressed-file SHA-256, mart manifest, Parquet SHA-256, and row counts all
|
||
reconcile. Every mart invariant reports zero violations. Raw VIN and raw OBD
|
||
result are absent from the staging output schema, feature mart, and model
|
||
features.
|
||
|
||
| Temporal partition | Eligible targets | Never-fit audit targets |
|
||
| --- | ---: | ---: |
|
||
| Train, 2016–2022 | 31,745 | 3,129 |
|
||
| Tune, 2023 | 4,972 | 526 |
|
||
| Calibrate, 2024 | 2,633 | 277 |
|
||
| Locked test, 2025 | 4,451 | 480 |
|
||
| Shadow, 2026 partial | 858 | 106 |
|
||
|
||
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. The
|
||
normal trainers do not calculate 2025 outcomes or metrics without the explicit
|
||
locked-evaluation flag.
|
||
|
||
## Baseline results before the locked test
|
||
|
||
The table below uses only non-audit vehicles. Average precision is the project's
|
||
PR-AUC summary. The calibrated logistic row is shown only on the partition used
|
||
to fit the calibrator and is therefore a calibration diagnostic, not an
|
||
independent final estimate.
|
||
|
||
| Partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
||
| --- | --- | ---: | ---: | ---: | ---: |
|
||
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
|
||
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
|
||
| 2023 tune | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
|
||
| 2024 calibrate | Training prevalence | 0.113 | 0.1002 | 0.500 | 0.113 |
|
||
| 2024 calibrate | Previous episode, literal | 0.148 | 0.1677 | 0.577 | 0.253 |
|
||
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
|
||
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |
|
||
|
||
The selected logistic regularization was `C=0.03`. It converged in 2,390 of the
|
||
5,000 allowed iterations; Platt calibration converged in five iterations. All
|
||
five candidates and all stored probabilities passed explicit convergence and
|
||
finite-value checks.
|
||
|
||
The literal previous-outcome baseline is useful as a ranking sanity check but
|
||
produces overconfident zero/one probabilities, explaining its poor Brier and log
|
||
loss. The transparent logistic model is the current development leader: on the
|
||
2023 tuning partition it more than doubles prevalence PR-AUC and raises top-10%
|
||
precision from 12.5% to 31.5%.
|
||
|
||
## One-time development holdout
|
||
|
||
The nonlinear live-verification command invoked the explicit 2025 gate after
|
||
its feature contract and four-candidate grid had already been fixed and its
|
||
candidate had been selected only on 2023. No test-driven model change was made.
|
||
At that point the development specification was frozen and the already-fixed
|
||
logistic model was evaluated once for a direct comparison. These are
|
||
development-sample holdout results, not final population claims.
|
||
|
||
| 2025 non-audit model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| Training prevalence | 0.123 | 0.1076 | 0.500 | 0.123 |
|
||
| Previous episode, literal | 0.158 | 0.1731 | 0.576 | 0.268 |
|
||
| Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
|
||
| Histogram gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
|
||
|
||
The holdout contains 3,971 non-audit episodes from 3,734 vehicles. The
|
||
never-fit audit contains another 480 episodes from 450 vehicles; calibrated
|
||
logistic PR-AUC is 0.253 and Brier is 0.0859 there. The logistic model beats
|
||
both simple baselines on the project's headline metrics and remains the chosen
|
||
development model. The tree did not provide a decisive pre-test improvement
|
||
that justified its added complexity, and no further 2025-informed tuning is
|
||
permitted.
|
||
|
||
## Remaining gates
|
||
|
||
1. Run a complete, contiguous bounded extraction for publishable population
|
||
aggregates; the page-sampled development cohort cannot support dashboard
|
||
prevalence or county rankings.
|
||
2. Treat any later full-data 2025 result as confirmatory rather than a pristine
|
||
unseen test, because the development sample's holdout has now been opened.
|
||
3. Run the frozen subgroup/source-era report and clustered uncertainty
|
||
analysis on the complete extraction.
|
||
4. Replace the checked-in development-preview bundle with complete-data
|
||
aggregates only after the population-publication review passes.
|
||
|
||
## Sanitized dashboard preview
|
||
|
||
The repository now includes a fail-closed exporter and a static four-view
|
||
dashboard shell. The preview exporter reads only train, tune, and calibration
|
||
rows dated before 2025. It rejects any model manifest that says the holdout was
|
||
evaluated, rejects any metric row outside the three approved partitions, and
|
||
marks every asset `development_preview=true` and
|
||
`population_estimate_allowed=false`.
|
||
|
||
Published cells must clear minimum episode and distinct-vehicle thresholds,
|
||
including both binary-class complements. Supports are rounded, direct and
|
||
pseudonymous identifiers stay private, and a checksum manifest covers the
|
||
approved JSON bundle. Private model manifests bind the exact metrics files, and
|
||
the public manifest exposes a deterministic release ID plus model versions for
|
||
provenance. The site disables the estimator because no privacy-reviewed
|
||
prediction lookup exists.
|