Centralized_I-M_Database/SummerProject2026
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

repository is now a presentation-ready development prototype

Browse Source
This commit is contained in:
kevinbell
2026-07-21 15:40:44 -06:00
parent 640165649d
commit 88161a6f16
24 changed files with 1989 additions and 946 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/final_report.md
+223
View File
@@ -0,0 +1,223 @@
# Utah Vehicle Health: final development-prototype report
Prepared July 21, 2026
> **Result scope:** Every analytical count, rate, chart, and model metric in this
> report comes from the private, page-sampled 10,000-vehicle development cohort.
> These are sample results, not population estimates. The cohort cannot support
> statewide or county prevalence, county rankings, causal conclusions, or
> decisions about an individual vehicle.
## Executive summary
Utah Vehicle Health asks one focused question:
> Using only information available before an inspection episode begins, how
> well can a calibrated logistic regression estimate whether a returning
> vehicle's next episode will have a first-attempt non-pass outcome?
The finished prototype builds leakage-safe inspection episodes, creates
point-in-time history features, evaluates models chronologically, and publishes
only suppression-reviewed sample aggregates to a static dashboard. Non-pass
combines recognized fail, reject, and abort outcomes; blank or unrecognized
results are not treated as passes.
The final model is regularized logistic regression with Platt calibration. On
the private sample's one-time 2025 holdout, it achieved PR-AUC 0.261 and Brier
score 0.1011. The calibrated histogram gradient-boosted benchmark achieved
PR-AUC 0.238 and Brier score 0.1024 on the same sample holdout. Logistic
regression remains the final model because it was both more transparent and
better on the declared headline metrics in this development comparison.
Those values demonstrate that the pipeline and model comparison work on this
development sample. They do not establish population performance, production
readiness, or external validity.
## Research scope
The prediction unit is the first attempt of the next inspection episode for a
returning vehicle. Consecutive attempts no more than 30 days apart form one
episode. This prevents a fail followed quickly by a retest from becoming two
nominally independent prediction targets. A vehicle is eligible only after a
prior episode has been observed.
The binary outcome is:
- pass; or
- non-pass, combining fail, reject, and abort.
Reject and abort can reflect process or readiness conditions, so this report
does not equate non-pass with mechanical failure. Cold-start prediction,
four-class modeling, fail-only sensitivity analysis, DMV enrichment,
station-level analysis, and individual prediction are outside the completed
prototype.
## Development data
The private extraction samples inspection-table pages, selects up to 10,000
vehicles, and then retrieves their complete available inspection histories. It
retained 9,996 private vehicle tokens after validation. The resulting sample
contains 83,552 source events, 69,588 episodes, and 44,659 eligible returning-
vehicle targets. These figures are pipeline-reconciliation counts for the
private sample, not estimates of Utah inspection volume.
The sample design intentionally makes end-to-end development manageable, but it
over-represents vehicles with more inspection records. Participating feeds also
change over time and do not cover all 29 Utah counties. The dashboard therefore
labels every value as a development-sample aggregate and uses county only to
communicate covered-feed context—not to publish population rankings.
The extraction and mart manifests bind hashes, configurations, and row counts.
Private vehicle linkage uses a one-way keyed token. Direct identifiers and raw
source payloads remain outside the analytical exports and public site.
## Leakage-safe method
Every feature exists before the target episode begins. The final model uses
vehicle age, prior episode and attempt counts, time since prior history, prior
non-pass rate, prior first/final outcomes, previously observed make/model,
public county context, and season.
The model excludes the target attempt's outcome and diagnostics, later attempts
from the target episode, station or technician information, future records,
full-history leakage, direct or pseudonymous identifiers, and preprocessing
learned from later partitions. DMV enrichment and rich current-test OBD fields
are not part of the final model.
The chronology is fixed:
| Period | Role in the private 10,000-vehicle development sample |
| --- | --- |
| 2010-2015 | Historical context only |
| 2016-2022 | Fit preprocessing and model parameters |
| 2023 | Select logistic regularization and compare fixed candidates |
| 2024 | Fit Platt calibration and inspect calibration behavior |
| 2025 | One-time development-sample holdout |
| 2026 partial | Shadow context only |
Random row splitting is prohibited. A separate set of private vehicle buckets
is never used for fitting and serves as a stress test.
## Model comparison and decision
The declared comparison includes training prevalence, the literal previous-
episode outcome, regularized logistic regression, and histogram gradient
boosting. Logistic regression uses `C=0.03`; Platt scaling is fit on the 2024
sample partition. The tree exists only to test whether nonlinearity provides a
meaningful development gain.
The table reports non-audit rows from the private, page-sampled 10,000-vehicle
development cohort. It is not population performance.
| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
On the 2023 sample partition, logistic regression more than doubled the
prevalence baseline's PR-AUC and raised precision among the highest-scored 10%
from 12.5% to 31.5%. On the one-time 2025 sample holdout, calibrated logistic
also exceeded the simple baselines and the boosted-tree benchmark on PR-AUC and
Brier score.
The explicit 2025 gate was opened once after model specifications were frozen
from pre-2025 data. No model was changed in response. That evaluation is a
development holdout, not a pristine future test, and no further 2025-informed
tuning is allowed.
The final decision is therefore straightforward: keep calibrated logistic
regression. It satisfies the development objective with a simpler, more
explainable form, while the nonlinear benchmark did not show a gain that
justified extra complexity.
## Static dashboard
The presentation-ready dashboard is a dependency-free static site with four
views:
1. **Overview** shows rounded support, sample outcome patterns, vehicle-age
patterns, and county feed coverage.
2. **Sample cohorts** shows only supported make/model sample aggregates; it does
not issue reliability ratings or recommendations.
3. **Model & benchmark** identifies calibrated logistic regression as final and
the boosted tree as benchmark-only. It shows pre-2025, 2024 calibration-fit
checks—not the opened 2025 holdout comparison.
4. **Data & methods** explains the target, chronology, evidence status,
limitations, and privacy boundary.
The finished dashboard contains no vehicle lookup, personal input form, or
row-level prediction service.
The dashboard loads purpose-built JSON rather than private analytical rows. It
checks schemas, cross-file consistency, publication flags, and checksums. If the
bundle fails validation, the interface shows an unavailable state instead of
fallback estimates.
Every published cell requires minimum support for episodes, distinct vehicles,
and both binary outcomes. Supports are rounded and suppressed rows are omitted
entirely. The checked-in assets use only pre-2025 sample aggregates and
diagnostics; the one-time 2025 values in this report are not in the browser
bundle.
## Privacy and release boundary
The public boundary is intentionally narrow:
```text
private read-only source
-> private local extraction and model pipeline
-> suppression-reviewed aggregate JSON
-> dashboard-only static Bolt project
```
The public dashboard and presentation materials contain no VIN, plate, ZIP,
station, technician identifier, private vehicle token, raw JSON, credential,
operational record, or row-level prediction. The browser never connects to the
source database.
Bolt publication must use a separate project or repository containing only the
contents of `dashboard/`, with `index.html` at its root. The full source
repository must not be imported into Bolt. The reviewed procedure is in
[bolt_deployment.md](bolt_deployment.md).
## Limitations
- The private page sample is not population-representative.
- Results cover returning vehicles in participating feeds, not all Utah
vehicles or counties.
- Geography, source era, program mix, and time are entangled.
- Reject and abort are heterogeneous non-pass outcomes.
- The 2025 comparison is a one-time development holdout, not external
validation.
- Sample calibration and ranking do not establish operational usefulness.
- Observed cohort differences are descriptive associations, not causal effects
or recommendations.
- The absence of protected attributes does not establish fairness.
## Conclusion
The prototype answers its development question: a leakage-safe, calibrated
logistic regression produces materially better discrimination and probability-error metrics
than simple baselines in the private 10,000-vehicle sample, and it performs
better than the boosted-tree benchmark in the one-time holdout comparison. The
more important product result is the disciplined boundary around that finding:
one target, chronological evaluation, transparent model selection, persistent
sample labeling, fail-closed aggregate publication, and no row-level exposure.
This is a completed development prototype—not a population estimate, county
ranking, production model, diagnosis, or individual decision tool.
## Supporting documentation
- [Project charter](project_charter.md)
- [Modeling protocol](modeling_protocol.md)
- [Detailed development results](development_results.md)
- [Model card](model_card.md)
- [Dashboard specification](dashboard_spec.md)
- [Private-to-public architecture](architecture.mmd)
- [10-minute presentation outline](presentation_outline.md)
- [Dashboard demo script](demo_script.md)
- [Dashboard-only Bolt deployment](bolt_deployment.md)