Centralized_I-M_Database/SummerProject2026
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

repository is now a presentation-ready development prototype

Browse Source
This commit is contained in:
kevinbell
2026-07-21 15:40:44 -06:00
parent 640165649d
commit 88161a6f16
24 changed files with 1989 additions and 946 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/development_results.md
+88 -93
View File
@@ -1,67 +1,66 @@
# Development run results
Run date: 2026-07-15
Run date: 2026-07-15; presentation wording reviewed 2026-07-21
These results validate the engineering and modeling pipeline. They are **not
population estimates**: the development extract samples inspection-table pages
and then retrieves complete histories for up to 10,000 vehicles, which
over-represents vehicles with more inspection records.
> **Scope of every value below:** the private, page-sampled 10,000-vehicle
> development cohort. These are sample results, not population estimates. Page
> sampling over-represents vehicles with more inspection records. Nothing in
> this document supports statewide/county prevalence or county rankings.
## Label audit
The raw Utah County feed leaves `overall_result` blank for 3,747,862 records.
An aggregate-only server audit found 3,469,102 Utah `obd`/`OBD` records with a
controlled pass, fail, reject, or abort value in `obd_result`. Across non-Utah
feeds where both fields were recognized, the two fields agreed on binary pass
versus non-pass 99.30% of the time, but their fail-versus-reject categories were
not interchangeable.
The binary target is pass versus non-pass, where non-pass combines recognized
fail, reject, and abort outcomes. Blanks and unrecognized values are unlabeled,
not passes.
Label contract v3 therefore permits the Utah OBD value only as a provenance-
tagged binary proxy. TSI, `other/C`, `B`, blank, and unknown values stay
unlabeled. Four-class analysis must use `overall_result` exclusively. This also
matches the official program distinction between a readiness rejection and a
failed inspection; see the [Utah inspection requirements](https://dmv.utah.gov/register/inspections/)
and [program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
In the development extract, the rule restored 11,334 labels and left 1,281
events unlabeled. The label-source field remains private audit metadata and is
not a model feature.
For the older Utah County feed, the documented source/program/test rule permits
a controlled OBD result to fill a blank overall result as a binary proxy. It
does not support a four-class interpretation. In the private 10,000-vehicle
sample, the rule restored 11,334 labels and left 1,281 events unlabeled. These
sample counts are not population estimates. Label provenance stays in private
audit metadata and is not a predictor.
## Pipeline reconciliation
| Stage | Rows |
This table reports private 10,000-vehicle development-sample pipeline counts,
not population totals.
| Sample pipeline stage | Rows |
| --- | ---: |
| Source events written after VIN validation | 83,552 |
| Source events written after identifier validation | 83,552 |
| Clean events after duplicate/conflict handling | 80,190 |
| Inspection episodes | 69,588 |
| Eligible returning targets | 44,659 |
| Eligible returning-vehicle targets | 44,659 |
The sample contains 9,996 retained vehicle tokens. The extraction manifest,
compressed-file SHA-256, mart manifest, Parquet SHA-256, and row counts all
reconcile. Every mart invariant reports zero violations. Raw VIN and raw OBD
result are absent from the staging output schema, feature mart, and model
features.
The sample retained 9,996 private vehicle tokens. Extraction and mart manifests,
compressed-file and Parquet checksums, and stage counts reconcile. Every mart
invariant reports zero violations. Raw identifiers and raw OBD values are absent
from staging output, feature mart predictors, and model features.
| Temporal partition | Eligible targets | Never-fit audit targets |
The following partition counts are also from the private 10,000-vehicle
development sample and are not population estimates.
| Sample temporal partition | Eligible targets | Never-fit audit targets |
| --- | ---: | ---: |
| Train, 2016–2022 | 31,745 | 3,129 |
| Train, 2016-2022 | 31,745 | 3,129 |
| Tune, 2023 | 4,972 | 526 |
| Calibrate, 2024 | 2,633 | 277 |
| Locked test, 2025 | 4,451 | 480 |
| One-time holdout, 2025 | 4,451 | 480 |
| Shadow, 2026 partial | 858 | 106 |
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. The
normal trainers do not calculate 2025 outcomes or metrics without the explicit
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. Normal
training does not calculate 2025 outcomes or metrics without the explicit
locked-evaluation flag.
## Baseline results before the locked test
## Pre-2025 development diagnostics
The table below uses only non-audit vehicles. Average precision is the project's
PR-AUC summary. The calibrated logistic row is shown only on the partition used
to fit the calibrator and is therefore a calibration diagnostic, not an
independent final estimate.
The table uses non-audit rows from the private 10,000-vehicle development
sample. Average precision is the project's PR-AUC summary. The calibrated 2024
row is evaluated on the same sample partition used to fit Platt scaling, so it
is a calibration diagnostic rather than independent final performance. None of
these values is a population estimate.
| Partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| Sample partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
@@ -71,66 +70,62 @@ independent final estimate.
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |
The selected logistic regularization was `C=0.03`. It converged in 2,390 of the
5,000 allowed iterations; Platt calibration converged in five iterations. All
five candidates and all stored probabilities passed explicit convergence and
finite-value checks.
The selected logistic regularization is `C=0.03`. It converged in 2,390 of the
5,000 allowed iterations; Platt calibration converged in five iterations. These
are development-run diagnostics. All candidates and stored probabilities passed
explicit convergence and finite-value checks.
The literal previous-outcome baseline is useful as a ranking sanity check but
produces overconfident zero/one probabilities, explaining its poor Brier and log
loss. The transparent logistic model is the current development leader: on the
2023 tuning partition it more than doubles prevalence PR-AUC and raises top-10%
precision from 12.5% to 31.5%.
On the 2023 sample partition, logistic regression more than doubled prevalence
PR-AUC and raised top-10% precision from 12.5% to 31.5%. This describes the
private development sample only.
## One-time development holdout
## One-time development-sample holdout
The nonlinear live-verification command invoked the explicit 2025 gate after
its feature contract and four-candidate grid had already been fixed and its
candidate had been selected only on 2023. No test-driven model change was made.
At that point the development specification was frozen and the already-fixed
logistic model was evaluated once for a direct comparison. These are
development-sample holdout results, not final population claims.
The nonlinear verification command invoked the explicit 2025 gate only after
the feature contract and benchmark grid had been frozen and candidate choices
had been made from pre-2025 sample data. No model was changed in response. These
are one-time development-sample holdout results, not pristine future-test or
population estimates.
| 2025 non-audit model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | ---: | ---: | ---: | ---: |
| Training prevalence | 0.123 | 0.1076 | 0.500 | 0.123 |
| Previous episode, literal | 0.158 | 0.1731 | 0.576 | 0.268 |
| Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
| Histogram gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
| 2025 private-sample model | Role | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | --- | ---: | ---: | ---: | ---: |
| Training prevalence | Baseline | 0.123 | 0.1076 | 0.500 | 0.123 |
| Previous episode, literal | Baseline | 0.158 | 0.1731 | 0.576 | 0.268 |
| Logistic + Platt | **Final model** | **0.261** | **0.1011** | **0.693** | **0.312** |
| Histogram gradient boosting + Platt | Benchmark only | 0.238 | 0.1024 | 0.690 | 0.292 |
The holdout contains 3,971 non-audit episodes from 3,734 vehicles. The
never-fit audit contains another 480 episodes from 450 vehicles; calibrated
logistic PR-AUC is 0.253 and Brier is 0.0859 there. The logistic model beats
both simple baselines on the project's headline metrics and remains the chosen
development model. The tree did not provide a decisive pre-test improvement
that justified its added complexity, and no further 2025-informed tuning is
permitted.
The sample holdout contains 3,971 non-audit episodes from 3,734 vehicles. Its
never-fit audit adds 480 episodes from 450 vehicles; calibrated logistic PR-AUC
is 0.253 and Brier score is 0.0859 there. All are private-sample diagnostics,
not population estimates or external validation.
## Remaining gates
1. Run a complete, contiguous bounded extraction for publishable population
aggregates; the page-sampled development cohort cannot support dashboard
prevalence or county rankings.
2. Treat any later full-data 2025 result as confirmatory rather than a pristine
unseen test, because the development sample's holdout has now been opened.
3. Run the frozen subgroup/source-era report and clustered uncertainty
analysis on the complete extraction.
4. Replace the checked-in development-preview bundle with complete-data
aggregates only after the population-publication review passes.
The calibrated logistic regression remains the final development model. It
beats both simple baselines on the headline metrics and beats the boosted-tree
benchmark in the one-time comparison. The benchmark did not justify its added
complexity. No further 2025-informed tuning is permitted.
## Sanitized dashboard preview
The repository now includes a fail-closed exporter and a static four-view
dashboard shell. The preview exporter reads only train, tune, and calibration
rows dated before 2025. It rejects any model manifest that says the holdout was
evaluated, rejects any metric row outside the three approved partitions, and
marks every asset `development_preview=true` and
`population_estimate_allowed=false`.
The static preview uses only suppression-cleared sample aggregates and
development diagnostics dated before 2025. It intentionally excludes the
one-time holdout metrics. Every asset declares `development_preview=true` and
`population_estimate_allowed=false`; the manifest declares
`locked_test_metrics_published=false`.
Published cells must clear minimum episode and distinct-vehicle thresholds,
including both binary-class complements. Supports are rounded, direct and
pseudonymous identifiers stay private, and a checksum manifest covers the
approved JSON bundle. Private model manifests bind the exact metrics files, and
the public manifest exposes a deterministic release ID plus model versions for
provenance. The site disables the estimator because no privacy-reviewed
prediction lookup exists.
Published cells must clear minimum episode, distinct-vehicle, and binary-class
thresholds. Supports are rounded, suppressed rows are omitted, and a checksum
manifest covers the approved JSON bundle. The site fails closed if a contract
check fails and contains no vehicle-level prediction service.
No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
credential, operational record, or row-level prediction is in the public
bundle.
## Boundary of the finished prototype
The repository completes a presentation-ready development prototype, not a
population study or production service. A complete population-frame extraction,
fresh external/future validation, individualized prediction lookup, DMV
enrichment, multiclass modeling, and operational monitoring are outside this
prototype's deliverables. They would require separate approval, analysis, and
privacy review; they are not implied by the current results.