132 lines
6.3 KiB
Markdown
132 lines
6.3 KiB
Markdown
# Development run results
|
|
|
|
Run date: 2026-07-15; presentation wording reviewed 2026-07-21
|
|
|
|
> **Scope of every value below:** the private, page-sampled 10,000-vehicle
|
|
> development cohort. These are sample results, not population estimates. Page
|
|
> sampling over-represents vehicles with more inspection records. Nothing in
|
|
> this document supports statewide/county prevalence or county rankings.
|
|
|
|
## Label audit
|
|
|
|
The binary target is pass versus non-pass, where non-pass combines recognized
|
|
fail, reject, and abort outcomes. Blanks and unrecognized values are unlabeled,
|
|
not passes.
|
|
|
|
For the older Utah County feed, the documented source/program/test rule permits
|
|
a controlled OBD result to fill a blank overall result as a binary proxy. It
|
|
does not support a four-class interpretation. In the private 10,000-vehicle
|
|
sample, the rule restored 11,334 labels and left 1,281 events unlabeled. These
|
|
sample counts are not population estimates. Label provenance stays in private
|
|
audit metadata and is not a predictor.
|
|
|
|
## Pipeline reconciliation
|
|
|
|
This table reports private 10,000-vehicle development-sample pipeline counts,
|
|
not population totals.
|
|
|
|
| Sample pipeline stage | Rows |
|
|
| --- | ---: |
|
|
| Source events written after identifier validation | 83,552 |
|
|
| Clean events after duplicate/conflict handling | 80,190 |
|
|
| Inspection episodes | 69,588 |
|
|
| Eligible returning-vehicle targets | 44,659 |
|
|
|
|
The sample retained 9,996 private vehicle tokens. Extraction and mart manifests,
|
|
compressed-file and Parquet checksums, and stage counts reconcile. Every mart
|
|
invariant reports zero violations. Raw identifiers and raw OBD values are absent
|
|
from staging output, feature mart predictors, and model features.
|
|
|
|
The following partition counts are also from the private 10,000-vehicle
|
|
development sample and are not population estimates.
|
|
|
|
| Sample temporal partition | Eligible targets | Never-fit audit targets |
|
|
| --- | ---: | ---: |
|
|
| Train, 2016-2022 | 31,745 | 3,129 |
|
|
| Tune, 2023 | 4,972 | 526 |
|
|
| Calibrate, 2024 | 2,633 | 277 |
|
|
| One-time holdout, 2025 | 4,451 | 480 |
|
|
| Shadow, 2026 partial | 858 | 106 |
|
|
|
|
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. Normal
|
|
training does not calculate 2025 outcomes or metrics without the explicit
|
|
locked-evaluation flag.
|
|
|
|
## Pre-2025 development diagnostics
|
|
|
|
The table uses non-audit rows from the private 10,000-vehicle development
|
|
sample. Average precision is the project's PR-AUC summary. The calibrated 2024
|
|
row is evaluated on the same sample partition used to fit Platt scaling, so it
|
|
is a calibration diagnostic rather than independent final performance. None of
|
|
these values is a population estimate.
|
|
|
|
| Sample partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
|
| --- | --- | ---: | ---: | ---: | ---: |
|
|
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
|
|
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
|
|
| 2023 tune | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
|
|
| 2024 calibrate | Training prevalence | 0.113 | 0.1002 | 0.500 | 0.113 |
|
|
| 2024 calibrate | Previous episode, literal | 0.148 | 0.1677 | 0.577 | 0.253 |
|
|
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
|
|
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |
|
|
|
|
The selected logistic regularization is `C=0.03`. It converged in 2,390 of the
|
|
5,000 allowed iterations; Platt calibration converged in five iterations. These
|
|
are development-run diagnostics. All candidates and stored probabilities passed
|
|
explicit convergence and finite-value checks.
|
|
|
|
On the 2023 sample partition, logistic regression more than doubled prevalence
|
|
PR-AUC and raised top-10% precision from 12.5% to 31.5%. This describes the
|
|
private development sample only.
|
|
|
|
## One-time development-sample holdout
|
|
|
|
The nonlinear verification command invoked the explicit 2025 gate only after
|
|
the feature contract and benchmark grid had been frozen and candidate choices
|
|
had been made from pre-2025 sample data. No model was changed in response. These
|
|
are one-time development-sample holdout results, not pristine future-test or
|
|
population estimates.
|
|
|
|
| 2025 private-sample model | Role | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
|
| --- | --- | ---: | ---: | ---: | ---: |
|
|
| Training prevalence | Baseline | 0.123 | 0.1076 | 0.500 | 0.123 |
|
|
| Previous episode, literal | Baseline | 0.158 | 0.1731 | 0.576 | 0.268 |
|
|
| Logistic + Platt | **Final model** | **0.261** | **0.1011** | **0.693** | **0.312** |
|
|
| Histogram gradient boosting + Platt | Benchmark only | 0.238 | 0.1024 | 0.690 | 0.292 |
|
|
|
|
The sample holdout contains 3,971 non-audit episodes from 3,734 vehicles. Its
|
|
never-fit audit adds 480 episodes from 450 vehicles; calibrated logistic PR-AUC
|
|
is 0.253 and Brier score is 0.0859 there. All are private-sample diagnostics,
|
|
not population estimates or external validation.
|
|
|
|
The calibrated logistic regression remains the final development model. It
|
|
beats both simple baselines on the headline metrics and beats the boosted-tree
|
|
benchmark in the one-time comparison. The benchmark did not justify its added
|
|
complexity. No further 2025-informed tuning is permitted.
|
|
|
|
## Sanitized dashboard preview
|
|
|
|
The static preview uses only suppression-cleared sample aggregates and
|
|
development diagnostics dated before 2025. It intentionally excludes the
|
|
one-time holdout metrics. Every asset declares `development_preview=true` and
|
|
`population_estimate_allowed=false`; the manifest declares
|
|
`locked_test_metrics_published=false`.
|
|
|
|
Published cells must clear minimum episode, distinct-vehicle, and binary-class
|
|
thresholds. Supports are rounded, suppressed rows are omitted, and a checksum
|
|
manifest covers the approved JSON bundle. The site fails closed if a contract
|
|
check fails and contains no vehicle-level prediction service.
|
|
|
|
No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
|
|
credential, operational record, or row-level prediction is in the public
|
|
bundle.
|
|
|
|
## Boundary of the finished prototype
|
|
|
|
The repository completes a presentation-ready development prototype, not a
|
|
population study or production service. A complete population-frame extraction,
|
|
fresh external/future validation, individualized prediction lookup, DMV
|
|
enrichment, multiclass modeling, and operational monitoring are outside this
|
|
prototype's deliverables. They would require separate approval, analysis, and
|
|
privacy review; they are not implied by the current results.
|