repository is now a presentation-ready development prototype
This commit is contained in:
parent
640165649d
commit
88161a6f16
214
README.md
214
README.md
@ -1,61 +1,57 @@
|
||||
# Summer Project 2026
|
||||
# Utah Vehicle Health
|
||||
|
||||
kevinBell@Linux.com
|
||||
Kevin Bell's summer 2026 development prototype for the first-attempt outcome of
|
||||
a returning vehicle's next emissions-inspection episode.
|
||||
|
||||
https://kevinbell.dev/
|
||||
## Prototype status
|
||||
|
||||
Kevin Bell's summer 2026 data project using state of Utah county vehicle-registration
|
||||
and emissions-inspection data.
|
||||
The repository implements one research question:
|
||||
|
||||
See the [data inventory](docs/data_inventory.md) and the data-backed
|
||||
[project shortlist](docs/project_options.md).
|
||||
> Using only information available before an inspection episode begins, how
|
||||
> well can a calibrated logistic regression estimate whether a returning
|
||||
> vehicle's next episode will have a first-attempt non-pass outcome?
|
||||
|
||||
## Selected project
|
||||
The final model is regularized logistic regression with Platt calibration.
|
||||
Histogram gradient boosting is a benchmark only. The final feature set uses
|
||||
inspection history; DMV enrichment, multiclass modeling, cold-start prediction,
|
||||
station analysis, and vehicle-level prediction service are outside scope.
|
||||
|
||||
The selected direction is **Utah Vehicle Health**, an explainable model and
|
||||
dashboard for the first-attempt outcome of a vehicle's next inspection episode.
|
||||
Start with the [project charter](docs/project_charter.md), then use the
|
||||
[modeling protocol](docs/modeling_protocol.md) and
|
||||
[dashboard specification](docs/dashboard_spec.md) as the project contracts.
|
||||
The latest private-sample pipeline findings are summarized in
|
||||
[development results](docs/development_results.md), with intended use and
|
||||
limitations consolidated in the [model card](docs/model_card.md).
|
||||
All analytical results and dashboard values come from the private, page-sampled
|
||||
10,000-vehicle development cohort. They are **sample results, not population
|
||||
estimates**. They must not be used for statewide/county prevalence, county
|
||||
rankings, causal claims, individual decisions, diagnoses, or guarantees.
|
||||
|
||||
## Safe database access in VS Code
|
||||
Start with the [final report](docs/final_report.md). Supporting deliverables:
|
||||
|
||||
The local `.env` file contains the standard PostgreSQL connection variables and
|
||||
is intentionally excluded from Git. Never put those values in source code,
|
||||
screenshots, browser JavaScript, or a Bolt project.
|
||||
- [Project charter](docs/project_charter.md)
|
||||
- [Leakage-safe modeling protocol](docs/modeling_protocol.md)
|
||||
- [Detailed development results](docs/development_results.md)
|
||||
- [Model card](docs/model_card.md)
|
||||
- [Static dashboard specification](docs/dashboard_spec.md)
|
||||
- [Private-to-public architecture](docs/architecture.mmd)
|
||||
- [10-minute presentation outline](docs/presentation_outline.md)
|
||||
- [Dashboard demo script](docs/demo_script.md)
|
||||
- [Dashboard-only Bolt deployment](docs/bolt_deployment.md)
|
||||
|
||||
1. Install the workspace-recommended **PostgreSQL** extension by Microsoft
|
||||
(`ms-ossdata.vscode-pgsql`).
|
||||
2. In VS Code, run **Tasks: Run Task** and choose
|
||||
**Countydata: test read-only connection**. It loads `.env`, requires TLS, and
|
||||
forces the PostgreSQL session into read-only mode.
|
||||
3. To browse the server visually, open the PostgreSQL sidebar and add a
|
||||
connection using the values in `.env`. Set SSL mode to **Require**, save the
|
||||
profile at **User** scope, and store the password in macOS Keychain rather
|
||||
than workspace settings.
|
||||
4. Connect to `countydata` and run [sql/00_read_only_connection_check.sql](sql/00_read_only_connection_check.sql).
|
||||
Safe aggregate examples are in [sql/01_safe_data_overview.sql](sql/01_safe_data_overview.sql).
|
||||
5. To inspect every visible database, relation, column, and a bounded sample of
|
||||
raw JSON field paths without printing row values, run **Countydata: inventory
|
||||
metadata safely**. The task uses [scripts/inventory_metadata.py](scripts/inventory_metadata.py)
|
||||
and enforces the same TLS and read-only requirements.
|
||||
The [data inventory](docs/data_inventory.md) and archived
|
||||
[project options](docs/project_options.md) provide source-discovery history;
|
||||
they do not expand the finished prototype scope.
|
||||
|
||||
The selected project's first aggregate cohort check is
|
||||
[sql/10_episode_cohort_feasibility.sql](sql/10_episode_cohort_feasibility.sql).
|
||||
Private bounded extraction is handled by
|
||||
[scripts/export_inspection_batch.py](scripts/export_inspection_batch.py); it
|
||||
requires the project-specific `VIN_HASH_KEY` described in `.env.example` and
|
||||
never writes a raw VIN.
|
||||
## Privacy boundary
|
||||
|
||||
The current database login has write privileges even though this project only
|
||||
needs reads. Ask the database administrator for a dedicated read-only role
|
||||
before connecting any deployed service. Until then, always use explicit
|
||||
read-only transactions.
|
||||
Never publish or display VINs, plates, ZIPs, stations, technician identifiers,
|
||||
private vehicle tokens, raw JSON, credentials, operational records, or row-level
|
||||
predictions. Private extracts, marts, databases, keys, and model artifacts stay
|
||||
in Git-ignored local paths. The browser never connects to `countydata`.
|
||||
|
||||
## Reproducible development workflow
|
||||
```text
|
||||
countydata (read-only)
|
||||
-> private local extraction, feature, and model pipeline
|
||||
-> suppression-reviewed aggregate JSON
|
||||
-> dashboard-only static site
|
||||
```
|
||||
|
||||
## Local setup
|
||||
|
||||
Create the local environment once:
|
||||
|
||||
@ -65,71 +61,103 @@ python3 -m venv .venv
|
||||
.venv/bin/python scripts/create_vin_hash_key.py
|
||||
```
|
||||
|
||||
The recommended way to run the pipeline is **Tasks: Run Task** in VS Code. The
|
||||
tasks preserve the intended order:
|
||||
The local `.env` contains PostgreSQL connection values and the project-specific
|
||||
`VIN_HASH_KEY`; it is excluded from Git. Never put those values in source code,
|
||||
screenshots, browser JavaScript, presentation materials, or a Bolt project.
|
||||
|
||||
## Safe database access in VS Code
|
||||
|
||||
1. Install the workspace-recommended Microsoft PostgreSQL extension
|
||||
(`ms-ossdata.vscode-pgsql`).
|
||||
2. Run **Tasks: Run Task > Countydata: test read-only connection**. The task
|
||||
requires TLS and forces the PostgreSQL session into read-only mode.
|
||||
3. If using the PostgreSQL sidebar, store the profile at User scope and the
|
||||
password in macOS Keychain, not workspace settings.
|
||||
4. Run [sql/00_read_only_connection_check.sql](sql/00_read_only_connection_check.sql)
|
||||
after connecting. Safe aggregate examples are in
|
||||
[sql/01_safe_data_overview.sql](sql/01_safe_data_overview.sql).
|
||||
5. Run **Countydata: inventory metadata safely** to inspect relation/field
|
||||
metadata without printing row values. The task uses
|
||||
[scripts/inventory_metadata.py](scripts/inventory_metadata.py).
|
||||
|
||||
The current database login has broader privileges than this project needs.
|
||||
Continue to enforce read-only transactions and use a dedicated read-only role
|
||||
before any separately approved operational integration.
|
||||
|
||||
## Reproducible private workflow
|
||||
|
||||
The VS Code tasks preserve the intended order:
|
||||
|
||||
1. Test the read-only TLS connection.
|
||||
2. Run the source outcome-mapping and aggregate feasibility audits.
|
||||
2. Run label-mapping and aggregate feasibility audits.
|
||||
3. Export the private 10,000-vehicle development histories.
|
||||
4. Build the private DuckDB warehouse and Parquet feature mart.
|
||||
5. Run every test.
|
||||
6. Train the baselines while leaving the 2025 test partition locked.
|
||||
7. Train the nonlinear comparison while leaving the same partition locked.
|
||||
5. Run the test suite.
|
||||
6. Train the baselines and logistic model with the 2025 gate closed.
|
||||
7. Train the boosted-tree benchmark with the same gate closed.
|
||||
8. Export the sanitized dashboard preview from approved pre-2025 partitions.
|
||||
|
||||
The development history sample is deliberately not population-representative.
|
||||
It exists to exercise feature engineering and modeling before a complete,
|
||||
contiguous bounded extraction is approved. Generated extracts, marts, database
|
||||
files, keys, and model artifacts remain under Git-ignored private paths.
|
||||
The extraction is deliberately not population-representative. It page-samples
|
||||
inspection records and then retrieves complete histories for the selected
|
||||
vehicles, which over-represents vehicles with more inspection records.
|
||||
|
||||
The 2025 metrics are not calculated by the normal training task. Unlocking them
|
||||
requires the conspicuous `--evaluate-locked` flag after the feature set,
|
||||
hyperparameters, calibration method, and reporting plan are frozen.
|
||||
That gate was opened once on the page-sampled development extract during live
|
||||
verification; the normal artifact paths were then regenerated closed. The
|
||||
audit trail is in [development results](docs/development_results.md), and no
|
||||
further 2025-informed tuning is permitted.
|
||||
The explicit 2025 gate was opened once after specifications were frozen. No
|
||||
model was changed in response. The values in the final report are one-time
|
||||
development-sample holdout diagnostics—not pristine future-test or population
|
||||
performance. Normal training leaves the gate closed.
|
||||
|
||||
## Sanitized dashboard preview
|
||||
## Tests
|
||||
|
||||
After training both closed model bundles, run **Tasks: Run Task** and choose
|
||||
**Utah Vehicle Health: export sanitized dashboard preview**. The exporter
|
||||
publishes only suppression-reviewed JSON under `dashboard/public/data`; it
|
||||
refuses the sampled mart unless the explicit development-preview flag is used
|
||||
and refuses model artifacts containing holdout metrics.
|
||||
Run the complete repository test suite:
|
||||
|
||||
Preview the site without exposing the repository root:
|
||||
```bash
|
||||
.venv/bin/python -m unittest discover -s tests -v
|
||||
```
|
||||
|
||||
Run the dashboard contract checks directly:
|
||||
|
||||
```bash
|
||||
npm --prefix dashboard test
|
||||
```
|
||||
|
||||
## Static dashboard
|
||||
|
||||
Preview the site from the repository root:
|
||||
|
||||
```bash
|
||||
node dashboard/server.mjs
|
||||
```
|
||||
|
||||
Then open `http://localhost:4173`. Do not serve the repository root, because it
|
||||
contains the local connection profile and private ignored directories.
|
||||
Open `http://127.0.0.1:4173`. Do not open `index.html` with a `file:` URL and do
|
||||
not serve the repository root.
|
||||
|
||||
The checked-in bundle is visibly labeled as a non-population development
|
||||
preview. Before a real public-data release, rerun the frozen pipeline on an
|
||||
approved complete extraction and pass the publication review documented in
|
||||
[the dashboard specification](docs/dashboard_spec.md).
|
||||
The four routes are Overview, Sample cohorts, Model & benchmark, and Data &
|
||||
methods. The site identifies calibrated logistic regression as final, the tree
|
||||
as benchmark-only, and displayed model cards as 2024 calibration-fit checks.
|
||||
The opened 2025 comparison stays in the report and presentation, not the public
|
||||
JSON. There is no vehicle-level prediction service.
|
||||
|
||||
Local and Bolt handoff instructions are in
|
||||
[dashboard/README.md](dashboard/README.md). Bolt can import a GitHub repository,
|
||||
but the hosted site root must be the `dashboard` directory; never copy `.env`,
|
||||
private data, model artifacts, or database tooling into a web project.
|
||||
The browser validates schemas, publication flags, cross-file consistency, and
|
||||
checksums. It shows no estimates if the approved aggregate bundle fails its
|
||||
contract. Every checked-in asset is marked as a development preview and not a
|
||||
population estimate.
|
||||
|
||||
## Data safety and publishing
|
||||
## Publish through Bolt
|
||||
|
||||
The source contains direct identifiers and operational data, including VINs,
|
||||
plates, user records, sessions, and upload metadata. A public dashboard should
|
||||
contain only de-identified aggregates and model outputs with minimum group-size
|
||||
suppression.
|
||||
Do **not** import this full source repository into Bolt. Create a separate
|
||||
deployment project or repository containing only the contents of `dashboard/`,
|
||||
so `index.html` is at that project's root. Do not copy `.env`, private data,
|
||||
artifacts, models, SQL, pipeline scripts, credentials, or unrelated repository
|
||||
history.
|
||||
|
||||
Recommended deployment flow:
|
||||
Follow the complete preflight, import, Bolt Hosting, and post-publication checks
|
||||
in [docs/bolt_deployment.md](docs/bolt_deployment.md). No publishing action is
|
||||
performed by this repository.
|
||||
|
||||
```text
|
||||
countydata (read-only) -> local ETL/modeling -> sanitized aggregate tables/files
|
||||
-> Bolt-hosted dashboard
|
||||
```
|
||||
## Author
|
||||
|
||||
Do not connect a browser directly to `countydata`. If the dashboard must refresh
|
||||
automatically, use a scheduled backend job with a dedicated read-only database
|
||||
role and copy only approved aggregate results into the hosted application.
|
||||
Kevin Bell
|
||||
|
||||
<kevinBell@Linux.com>
|
||||
|
||||
<https://kevinbell.dev/>
|
||||
|
||||
@ -1,10 +1,18 @@
|
||||
# Utah Vehicle Health dashboard
|
||||
|
||||
This directory is a dependency-free static prototype. It uses semantic HTML,
|
||||
CSS, vanilla ES modules, and inline SVG generated from approved aggregate JSON.
|
||||
It has no database client, server credential, analytics SDK, or external CDN.
|
||||
This directory is a dependency-free static development prototype. It uses
|
||||
semantic HTML, CSS, vanilla ES modules, and inline SVG generated from approved,
|
||||
suppressed aggregate JSON. It has no database client, server credential,
|
||||
analytics SDK, or external CDN.
|
||||
|
||||
## Run locally from the repository root
|
||||
Every displayed result comes from the private 10,000-vehicle development
|
||||
sample. Results are sample-based, are not population estimates, and must not be
|
||||
used for individual decisions. The final prototype model is calibrated logistic
|
||||
regression; the boosted tree is a benchmark only.
|
||||
|
||||
## Run locally
|
||||
|
||||
From the Utah Vehicle Health source-repository root:
|
||||
|
||||
```bash
|
||||
node dashboard/server.mjs
|
||||
@ -23,6 +31,14 @@ Run the dependency-free contract checks with:
|
||||
node --test dashboard/tests/contract.test.mjs
|
||||
```
|
||||
|
||||
After the contents of `dashboard/` have been copied into a dashboard-only
|
||||
repository or project root, use the self-contained equivalents:
|
||||
|
||||
```bash
|
||||
npm start
|
||||
npm test
|
||||
```
|
||||
|
||||
## Public-data boundary
|
||||
|
||||
The browser requires and validates these files beneath `public/data/`:
|
||||
@ -41,27 +57,33 @@ contains a denied identifier-like field, the dashboard shows an unavailable
|
||||
state and no estimates. Regenerate assets with the repository's private local
|
||||
pipeline; never hand-edit public JSON to bypass suppression.
|
||||
|
||||
The next-test estimator is intentionally disabled. It must remain disabled
|
||||
until a separately reviewed and suppressed `prediction_lookup` contract is
|
||||
approved. Do not add a connection from this site to `countydata`.
|
||||
The site intentionally contains no vehicle lookup, individual estimator,
|
||||
operational connection, or row-level prediction output. VINs, plates, ZIPs,
|
||||
stations, technician identifiers, raw source JSON, credentials, and operational
|
||||
records must never enter this directory.
|
||||
|
||||
## Import into Bolt
|
||||
## Publish through Bolt
|
||||
|
||||
The recommended publication boundary is a separate static Bolt project (and,
|
||||
ideally, a separate deployment repository) containing only the contents of
|
||||
`dashboard/`. Upload or copy this directory's contents so `index.html` is at the
|
||||
new project root. There is no install or build step. If Bolt asks for a preview
|
||||
command, use `node server.mjs` with `HOST=0.0.0.0`; Bolt supplies `PORT`.
|
||||
Use a separate static Bolt project backed by a dashboard-only repository. Do
|
||||
not import this source repository into Bolt.
|
||||
|
||||
Before publishing, confirm that `/public/data/data_manifest.json` resolves,
|
||||
the development-preview banner remains visible, and the contract test passes.
|
||||
The generated JSON bundle must also complete its privacy review.
|
||||
1. Run `npm test` from this directory and confirm every contract check passes.
|
||||
2. Create a clean dashboard-only repository or Bolt project.
|
||||
3. Copy only the *contents* of `dashboard/` into that project, so `index.html`
|
||||
is at the project root.
|
||||
4. Confirm that private `data/`, `artifacts/`, `models/`, `.env` files, SQL,
|
||||
notebooks, and database tooling are absent.
|
||||
5. Preview the project. There is no install or build step; if Bolt requests a
|
||||
preview command, use `node server.mjs` with `HOST=0.0.0.0` and let Bolt
|
||||
provide `PORT`.
|
||||
6. Verify all four views, the development-sample messaging on every view, and
|
||||
that the in-app status reads **Validated sample aggregates**.
|
||||
7. Use **Publish** only after the aggregate bundle completes privacy review.
|
||||
Bolt hosting is the default; Netlify can instead be selected before the
|
||||
first publish if desired.
|
||||
|
||||
Importing the whole source repository is a discouraged fallback because Bolt
|
||||
would receive SQL, database tooling, and other files that are not needed by the
|
||||
public site. If that has already happened, set the project/working root to
|
||||
`dashboard`, never serve the repository root, and create a dashboard-only
|
||||
project before production publication. Do not copy private `data/`,
|
||||
`artifacts/`, `models/`, `.env`, or database tooling into the public project.
|
||||
See Bolt's official documentation for [Git
|
||||
integration](https://support.bolt.new/integrations/git) and [publishing with
|
||||
Netlify](https://support.bolt.new/integrations/netlify).
|
||||
|
||||
No publishing action is performed by this repository.
|
||||
|
||||
@ -3,8 +3,9 @@
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="description" content="Utah Vehicle Health — an accessible, aggregate view of emissions-inspection outcomes.">
|
||||
<meta name="description" content="Sample-based research on the first-attempt non-pass outcome of a returning vehicle's next emissions inspection.">
|
||||
<meta name="referrer" content="no-referrer">
|
||||
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'; font-src 'self'; object-src 'none'; base-uri 'none'; form-action 'none'">
|
||||
<title>Overview · Utah Vehicle Health</title>
|
||||
<link rel="stylesheet" href="./styles.css">
|
||||
<script type="module" src="./js/app.js"></script>
|
||||
@ -16,7 +17,7 @@
|
||||
<span class="preview-banner__icon" aria-hidden="true">◆</span>
|
||||
<div>
|
||||
<strong id="preview-title">Development preview</strong>
|
||||
<span id="preview-copy">Sample results are not population estimates and must not be used for individual decisions.</span>
|
||||
<span id="preview-copy">Private 10,000-vehicle development sample · suppressed aggregates only · not population estimates.</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@ -29,7 +30,7 @@
|
||||
</svg>
|
||||
<span>
|
||||
<span class="brand__name">Utah Vehicle Health</span>
|
||||
<span class="brand__tagline">Next-episode inspection insights</span>
|
||||
<span class="brand__tagline">Returning vehicles · next first attempt</span>
|
||||
</span>
|
||||
</a>
|
||||
|
||||
@ -42,8 +43,8 @@
|
||||
<nav class="primary-nav" aria-label="Dashboard views">
|
||||
<div class="primary-nav__inner">
|
||||
<a href="#overview" data-route="overview" aria-current="page">Overview</a>
|
||||
<a href="#reliability" data-route="reliability">Reliability explorer</a>
|
||||
<a href="#estimator" data-route="estimator">Next-test estimator</a>
|
||||
<a href="#cohorts" data-route="cohorts">Sample cohorts</a>
|
||||
<a href="#model" data-route="model">Model & benchmark</a>
|
||||
<a href="#methods" data-route="methods">Data & methods</a>
|
||||
</div>
|
||||
</nav>
|
||||
@ -54,9 +55,9 @@
|
||||
<div class="page-shell">
|
||||
<header class="page-heading page-heading--split">
|
||||
<div>
|
||||
<p class="eyebrow">Inspection outcomes over time</p>
|
||||
<h1 id="overview-heading">A clearer view of the next inspection</h1>
|
||||
<p class="lede">Explore aggregate first-attempt outcomes for returning vehicles in participating Utah county feeds. Non-pass combines fail, reject, and abort.</p>
|
||||
<p class="eyebrow">Private 10,000-vehicle development sample</p>
|
||||
<h1 id="overview-heading">The next inspection, in sample context</h1>
|
||||
<p class="lede">The research question is intentionally narrow: predict whether a returning vehicle's next inspection episode has a first-attempt non-pass. Non-pass combines fail, reject, and abort.</p>
|
||||
</div>
|
||||
<div class="data-stamp" aria-label="Dataset status">
|
||||
<span>Data through</span>
|
||||
@ -65,6 +66,11 @@
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<aside class="scope-strip" aria-label="Result scope">
|
||||
<strong>Sample-based only</strong>
|
||||
<span>Every value on this page describes the private development sample and covered feeds—not Utah's vehicle population.</span>
|
||||
</aside>
|
||||
|
||||
<div class="unavailable-state" data-unavailable hidden role="status">
|
||||
<div class="unavailable-state__icon" aria-hidden="true">!</div>
|
||||
<div>
|
||||
@ -75,22 +81,22 @@
|
||||
|
||||
<section class="kpi-grid" aria-label="Overview statistics" data-requires-data>
|
||||
<article class="kpi-card">
|
||||
<span class="kpi-card__label">Published support</span>
|
||||
<span class="kpi-card__label">Published episode support</span>
|
||||
<strong id="kpi-eligible">—</strong>
|
||||
<small>Rounded total; suppressed cells omitted</small>
|
||||
<small>Rounded total; sampled vehicles may contribute multiple eligible episodes</small>
|
||||
</article>
|
||||
<article class="kpi-card kpi-card--pass">
|
||||
<span class="kpi-card__label">Pass rate</span>
|
||||
<span class="kpi-card__label">Observed sample pass rate</span>
|
||||
<strong id="kpi-pass">—</strong>
|
||||
<small>Recognized first-attempt outcomes</small>
|
||||
</article>
|
||||
<article class="kpi-card kpi-card--nonpass">
|
||||
<span class="kpi-card__label">Non-pass rate</span>
|
||||
<span class="kpi-card__label">Observed sample non-pass rate</span>
|
||||
<strong id="kpi-nonpass">—</strong>
|
||||
<small>Fail + reject + abort</small>
|
||||
</article>
|
||||
<article class="kpi-card">
|
||||
<span class="kpi-card__label">Covered counties</span>
|
||||
<span class="kpi-card__label">Covered county feeds</span>
|
||||
<strong id="kpi-counties">—</strong>
|
||||
<small>Not statewide coverage</small>
|
||||
</article>
|
||||
@ -100,8 +106,8 @@
|
||||
<article class="panel panel--wide">
|
||||
<header class="panel__header">
|
||||
<div>
|
||||
<p class="eyebrow">Quarterly outcomes</p>
|
||||
<h2>First-attempt non-pass trend</h2>
|
||||
<p class="eyebrow">Sample association over time</p>
|
||||
<h2>Observed first-attempt non-pass trend</h2>
|
||||
</div>
|
||||
<div class="chart-legend" aria-label="Outcome legend">
|
||||
<span><i class="legend-swatch legend-swatch--fail"></i>Non-pass (fail + reject + abort)</span>
|
||||
@ -120,36 +126,41 @@
|
||||
</header>
|
||||
<div id="utah-coverage-map" class="utah-map" role="img" aria-label="Utah county-feed coverage"></div>
|
||||
<div id="county-coverage-list" class="county-list" aria-label="County availability list"></div>
|
||||
<p class="chart-note">Teal counties appear in the approved inspection feeds. Gray counties are unavailable, not zero.</p>
|
||||
<p class="chart-note">Availability only—this is not a county outcome ranking. Teal feeds are represented in the sample; gray counties are unavailable, not zero.</p>
|
||||
</article>
|
||||
|
||||
<article class="panel">
|
||||
<header class="panel__header">
|
||||
<div>
|
||||
<p class="eyebrow">Age pattern</p>
|
||||
<h2>Non-pass risk by vehicle age</h2>
|
||||
<p class="eyebrow">Sample association by age</p>
|
||||
<h2>Observed non-pass rate by vehicle age</h2>
|
||||
</div>
|
||||
</header>
|
||||
<div id="age-risk-chart" class="chart" role="img" aria-label="Published non-pass risk by vehicle-age band"></div>
|
||||
<p class="chart-note">The development bundle does not include confidence intervals; published aggregate rates are shown as points.</p>
|
||||
<p class="chart-note">Sample aggregates only, without confidence intervals. The pattern is descriptive and not a population estimate or causal effect.</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<aside class="callout" data-requires-data>
|
||||
<span class="callout__icon" aria-hidden="true">i</span>
|
||||
<p><strong>What this measures:</strong> emissions-inspection outcomes for covered feeds. It is not a diagnosis of mechanical condition, safety, roadworthiness, or legal compliance.</p>
|
||||
<p><strong>What this measures:</strong> sampled returning vehicles' next-episode first-attempt emissions-inspection outcomes in covered feeds. It is not a population estimate, mechanical diagnosis, safety assessment, or legal determination.</p>
|
||||
</aside>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="view" id="view-reliability" data-view="reliability" aria-labelledby="reliability-heading" hidden>
|
||||
<section class="view" id="view-cohorts" data-view="cohorts" aria-labelledby="cohorts-heading" hidden>
|
||||
<div class="page-shell">
|
||||
<header class="page-heading">
|
||||
<p class="eyebrow">Supported aggregate cohorts</p>
|
||||
<h1 id="reliability-heading">Reliability explorer</h1>
|
||||
<p class="lede">Compare published observed next-episode non-pass rates. Every result is a supported aggregate cohort—not an individual prediction.</p>
|
||||
<p class="eyebrow">Private 10,000-vehicle development sample</p>
|
||||
<h1 id="cohorts-heading">Sample cohort explorer</h1>
|
||||
<p class="lede">Explore supported prior make-and-model cohorts and their observed next-episode first-attempt non-pass rates. These are sample associations—not reliability grades or individual predictions.</p>
|
||||
</header>
|
||||
|
||||
<aside class="scope-strip" aria-label="Result scope">
|
||||
<strong>Not a ranking of population reliability</strong>
|
||||
<span>Rates use rounded, suppressed episode support from sampled vehicles. Cohorts can differ in age, history, period, and coverage.</span>
|
||||
</aside>
|
||||
|
||||
<div class="unavailable-state" data-unavailable hidden role="status">
|
||||
<div class="unavailable-state__icon" aria-hidden="true">!</div>
|
||||
<div><h2>Explorer unavailable</h2><p data-unavailable-message>Approved scorecards could not be validated.</p></div>
|
||||
@ -158,7 +169,7 @@
|
||||
<div class="explorer-layout" data-requires-data>
|
||||
<aside class="filter-panel" aria-labelledby="filter-heading">
|
||||
<div class="filter-panel__heading">
|
||||
<h2 id="filter-heading">Refine cohorts</h2>
|
||||
<h2 id="filter-heading">Find a cohort</h2>
|
||||
<button class="text-button" id="reset-filters" type="button">Reset</button>
|
||||
</div>
|
||||
<form id="explorer-filters">
|
||||
@ -166,31 +177,7 @@
|
||||
<span>Make or model</span>
|
||||
<input id="cohort-search" type="search" autocomplete="off" placeholder="Search supported cohorts">
|
||||
</label>
|
||||
<label class="field">
|
||||
<span>County</span>
|
||||
<select id="filter-county"><option value="">All covered counties</option></select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<span>Vehicle age</span>
|
||||
<select id="filter-age"><option value="">All age bands</option></select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<span>Fuel</span>
|
||||
<select id="filter-fuel"><option value="">All supported fuels</option></select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<span>Program</span>
|
||||
<select id="filter-program"><option value="">All approved programs</option></select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<span>Period</span>
|
||||
<select id="filter-period"><option value="">All published periods</option></select>
|
||||
</label>
|
||||
<fieldset class="segmented-control">
|
||||
<legend>Risk view</legend>
|
||||
<label><input type="radio" name="risk-view" value="observed" checked><span>Observed</span></label>
|
||||
<label><input type="radio" name="risk-view" value="adjusted"><span>Model-adjusted</span></label>
|
||||
</fieldset>
|
||||
<p class="filter-panel__note"><strong>Published grain:</strong> prior make and model only. County, age, fuel, program, and period slices are not available for these scorecards.</p>
|
||||
</form>
|
||||
</aside>
|
||||
|
||||
@ -201,8 +188,8 @@
|
||||
<span>Sort</span>
|
||||
<select id="result-sort">
|
||||
<option value="support">Largest support</option>
|
||||
<option value="risk-desc">Highest non-pass risk</option>
|
||||
<option value="risk-asc">Lowest non-pass risk</option>
|
||||
<option value="risk-desc">Highest observed sample rate</option>
|
||||
<option value="risk-asc">Lowest observed sample rate</option>
|
||||
<option value="name">Make and model</option>
|
||||
</select>
|
||||
</label>
|
||||
@ -210,9 +197,10 @@
|
||||
|
||||
<article class="panel">
|
||||
<header class="panel__header">
|
||||
<div><p class="eyebrow">Published aggregate comparison</p><h2>Supported cohort scorecard</h2></div>
|
||||
<div><p class="eyebrow">Descriptive sample comparison</p><h2>Observed cohort associations</h2></div>
|
||||
</header>
|
||||
<div id="cohort-dot-plot" class="chart chart--scorecard" role="img" aria-label="Ranked published cohort non-pass rates"></div>
|
||||
<div id="cohort-dot-plot" class="chart chart--scorecard" role="img" aria-label="Observed development-sample cohort non-pass rates"></div>
|
||||
<p class="chart-note">Ordering is a viewing aid within this sample, not a reliability ranking or population comparison.</p>
|
||||
</article>
|
||||
|
||||
<div id="cohort-cards" class="cohort-cards"></div>
|
||||
@ -225,51 +213,64 @@
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="view" id="view-estimator" data-view="estimator" aria-labelledby="estimator-heading" hidden>
|
||||
<section class="view" id="view-model" data-view="model" aria-labelledby="model-heading" hidden>
|
||||
<div class="page-shell page-shell--narrow">
|
||||
<header class="page-heading">
|
||||
<p class="eyebrow">Cohort estimate · not an individual diagnosis</p>
|
||||
<h1 id="estimator-heading">Next-test risk estimator</h1>
|
||||
<p class="lede">This planned tool will combine approved, coarsened attributes to return a calibrated next-episode non-pass probability.</p>
|
||||
<p class="eyebrow">Private 10,000-vehicle development sample</p>
|
||||
<h1 id="model-heading">Model & benchmark</h1>
|
||||
<p class="lede">Calibrated logistic regression is the final development-prototype model for the next-episode first-attempt non-pass target. The boosted tree is retained only as a benchmark.</p>
|
||||
</header>
|
||||
|
||||
<div class="locked-panel" role="status" aria-labelledby="estimator-status-title">
|
||||
<span class="locked-panel__icon" aria-hidden="true">◇</span>
|
||||
<div>
|
||||
<p class="eyebrow">Intentionally disabled</p>
|
||||
<h2 id="estimator-status-title">An approved prediction lookup is not available</h2>
|
||||
<p>The estimator will remain off until a privacy-reviewed, suppressed <code>prediction_lookup</code> is published. The current model diagnostics are not enough to serve individual or row-level estimates.</p>
|
||||
</div>
|
||||
<aside class="scope-strip" aria-label="Result scope">
|
||||
<strong>Development evidence only</strong>
|
||||
<span>These 2024 values come from the partition used to fit Platt calibration. They are sample-based calibration checks—not independent final-performance estimates, population estimates, or locked-test results.</span>
|
||||
</aside>
|
||||
|
||||
<div class="unavailable-state" data-unavailable hidden role="status">
|
||||
<div class="unavailable-state__icon" aria-hidden="true">!</div>
|
||||
<div><h2>Model diagnostics unavailable</h2><p data-unavailable-message>The approved diagnostic summary could not be validated.</p></div>
|
||||
</div>
|
||||
|
||||
<form class="estimator-form" aria-describedby="estimator-disabled-copy">
|
||||
<fieldset disabled>
|
||||
<legend>Coarsened cohort attributes</legend>
|
||||
<div class="form-grid">
|
||||
<label class="field"><span>County</span><select><option>Select a covered county</option></select></label>
|
||||
<label class="field"><span>Supported make and model</span><select><option>Select a cohort</option></select></label>
|
||||
<label class="field"><span>Vehicle-age band</span><select><option>Select an age band</option></select></label>
|
||||
<label class="field"><span>Fuel</span><select><option>Select a supported fuel</option></select></label>
|
||||
<label class="field"><span>Prior episode outcome</span><select><option>Pass / fail / reject / abort</option></select></label>
|
||||
<label class="field"><span>Time since prior episode</span><select><option>Select a coarsened interval</option></select></label>
|
||||
<label class="field"><span>Season</span><select><option>Select a season</option></select></label>
|
||||
<label class="field"><span>Program category</span><select><option>Select an approved program</option></select></label>
|
||||
</div>
|
||||
<button class="button" type="button">Estimate cohort risk</button>
|
||||
</fieldset>
|
||||
</form>
|
||||
<p id="estimator-disabled-copy" class="form-note">No VIN, plate, exact address, station, free text, or current-test diagnostic will ever be requested.</p>
|
||||
<section class="model-comparison" data-requires-data aria-label="Development model comparison">
|
||||
<article class="model-card model-card--final">
|
||||
<div class="model-card__topline"><span class="decision-badge decision-badge--final">Final prototype</span><span>2024 calibration-fit check</span></div>
|
||||
<h2>Calibrated logistic regression</h2>
|
||||
<p>The selected model for the research conclusion and any prototype-level discussion. Platt scaling calibrates its probabilities.</p>
|
||||
<dl class="metric-list">
|
||||
<div><dt>PR-AUC</dt><dd id="final-ap">—</dd></div>
|
||||
<div><dt>ROC-AUC</dt><dd id="final-roc">—</dd></div>
|
||||
<div><dt>Brier ↓</dt><dd id="final-brier">—</dd></div>
|
||||
</dl>
|
||||
</article>
|
||||
|
||||
<section class="method-preview" aria-labelledby="future-output-heading">
|
||||
<article class="model-card">
|
||||
<div class="model-card__topline"><span class="decision-badge">Benchmark only</span><span>2024 calibration-fit check</span></div>
|
||||
<h2>Calibrated boosted tree</h2>
|
||||
<p>Retained for development comparison only. It does not replace the logistic model or define dashboard outputs.</p>
|
||||
<dl class="metric-list">
|
||||
<div><dt>PR-AUC</dt><dd id="benchmark-ap">—</dd></div>
|
||||
<div><dt>ROC-AUC</dt><dd id="benchmark-roc">—</dd></div>
|
||||
<div><dt>Brier ↓</dt><dd id="benchmark-brier">—</dd></div>
|
||||
</dl>
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<aside class="callout" data-requires-data>
|
||||
<span class="callout__icon" aria-hidden="true">i</span>
|
||||
<p><strong>Calibration diagnostic—not a final performance estimate:</strong> Platt scaling was fit on this same 2024 partition. PR-AUC and ROC-AUC summarize discrimination; lower Brier is better for probability error. Opened 2025 holdout metrics are intentionally not displayed here.</p>
|
||||
</aside>
|
||||
|
||||
<section class="publication-boundary" aria-labelledby="publication-boundary-heading">
|
||||
<div>
|
||||
<p class="eyebrow">Future approved output</p>
|
||||
<h2 id="future-output-heading">Designed for calibrated context</h2>
|
||||
<p class="eyebrow">Finished publication boundary</p>
|
||||
<h2 id="publication-boundary-heading">No vehicle-level prediction service</h2>
|
||||
<p>This static dashboard serves supported aggregate sample rates and diagnostic summaries only. It contains no vehicle lookup, personal inputs, operational connection, or row-level prediction output.</p>
|
||||
</div>
|
||||
<ul class="check-list">
|
||||
<li>Calibrated non-pass probability and uncertainty</li>
|
||||
<li>Relevant aggregate baseline</li>
|
||||
<li>High-level factor contributions</li>
|
||||
<li>Support and coverage limitations</li>
|
||||
<li>Returning-vehicle next-episode target only</li>
|
||||
<li>First attempt; pass versus non-pass</li>
|
||||
<li>Final model: calibrated logistic regression</li>
|
||||
<li>Boosted tree: benchmark only</li>
|
||||
</ul>
|
||||
</section>
|
||||
</div>
|
||||
@ -278,11 +279,16 @@
|
||||
<section class="view" id="view-methods" data-view="methods" aria-labelledby="methods-heading" hidden>
|
||||
<div class="page-shell">
|
||||
<header class="page-heading">
|
||||
<p class="eyebrow">Transparent by design</p>
|
||||
<p class="eyebrow">Private 10,000-vehicle development sample</p>
|
||||
<h1 id="methods-heading">Data & methods</h1>
|
||||
<p class="lede">How episodes, labels, temporal evaluation, coverage limits, and privacy controls shape every published result.</p>
|
||||
</header>
|
||||
|
||||
<aside class="scope-strip" aria-label="Result scope">
|
||||
<strong>Sample-based, not population-based</strong>
|
||||
<span>All displayed rates and diagnostics come from the development sample. County displays communicate feed coverage only.</span>
|
||||
</aside>
|
||||
|
||||
<div class="unavailable-state" data-unavailable hidden role="status">
|
||||
<div class="unavailable-state__icon" aria-hidden="true">!</div>
|
||||
<div><h2>Live diagnostics unavailable</h2><p data-unavailable-message>Static methodology remains below; no performance values are being shown.</p></div>
|
||||
@ -330,13 +336,13 @@
|
||||
|
||||
<div class="dashboard-grid" data-requires-data>
|
||||
<article class="panel">
|
||||
<header class="panel__header"><div><p class="eyebrow" id="diagnostic-scope-label">Model diagnostics</p><h2>Published model diagnostics</h2></div></header>
|
||||
<div class="diagnostic-kpis">
|
||||
<div><span>PR-AUC</span><strong id="diagnostic-ap">—</strong></div>
|
||||
<div><span>Brier score</span><strong id="diagnostic-brier">—</strong></div>
|
||||
<div><span>Support</span><strong id="diagnostic-n">—</strong></div>
|
||||
</div>
|
||||
<div id="calibration-chart" class="chart" role="img" aria-label="Predicted versus observed non-pass risk calibration"></div>
|
||||
<header class="panel__header"><div><p class="eyebrow">Model governance</p><h2>Evidence status</h2></div></header>
|
||||
<ul class="limitation-list">
|
||||
<li>Calibrated logistic regression is the declared final prototype model.</li>
|
||||
<li>The boosted tree remains a benchmark only.</li>
|
||||
<li>Published diagnostics cover train, tune, and calibration partitions.</li>
|
||||
<li>No locked-test result or population performance claim is published.</li>
|
||||
</ul>
|
||||
</article>
|
||||
|
||||
<article class="panel">
|
||||
@ -344,7 +350,7 @@
|
||||
<ul class="limitation-list" id="manifest-limitations">
|
||||
<li>Participating inspection feeds do not cover all 29 counties.</li>
|
||||
<li>Feed and program changes can resemble real-world trends.</li>
|
||||
<li>DMV history has a 2021 gap and ends in March 2024.</li>
|
||||
<li>The 10,000-vehicle draw retained 9,996 vehicles; page sampling can over-represent vehicles with more inspection records.</li>
|
||||
<li>Reject and abort can reflect process or readiness issues.</li>
|
||||
<li>Association is not causation or a mechanical diagnosis.</li>
|
||||
</ul>
|
||||
@ -358,9 +364,9 @@
|
||||
<span aria-hidden="true">→</span>
|
||||
<span>Suppression<br><small>Minimum support + complements</small></span>
|
||||
<span aria-hidden="true">→</span>
|
||||
<span class="privacy-flow__public">Public assets<br><small>Aggregate JSON only</small></span>
|
||||
<span class="privacy-flow__public">Public assets<br><small>Suppressed summaries only</small></span>
|
||||
</div>
|
||||
<p>Direct identifiers, private vehicle tokens, plates, ZIPs, stations, raw records, and row-level predictions are outside the public data contract.</p>
|
||||
<p>VINs, plates, ZIPs, stations, technician identifiers, private vehicle tokens, raw source JSON, credentials, operational records, and row-level predictions are outside this site and its public data contract.</p>
|
||||
</section>
|
||||
</div>
|
||||
</section>
|
||||
@ -368,8 +374,8 @@
|
||||
|
||||
<footer class="site-footer">
|
||||
<div class="page-shell site-footer__inner">
|
||||
<div><strong>Utah Vehicle Health</strong><p>Aggregate emissions-inspection research.</p></div>
|
||||
<p>Not a diagnosis, certification, safety assessment, or guarantee of an inspection outcome.</p>
|
||||
<div><strong>Utah Vehicle Health</strong><p>Private 10,000-vehicle development sample.</p></div>
|
||||
<p>Sample-based research only—not a population estimate, diagnosis, certification, safety assessment, or guarantee of an inspection outcome.</p>
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
|
||||
@ -6,15 +6,14 @@ import {
|
||||
renderAgeRisk,
|
||||
renderCohortDotPlot,
|
||||
renderCoverageHeatmap,
|
||||
renderNoCalibration,
|
||||
renderOutcomeTrend,
|
||||
renderUtahCoverage,
|
||||
} from "./charts.js";
|
||||
|
||||
const ROUTES = Object.freeze({
|
||||
overview: "Overview",
|
||||
reliability: "Reliability explorer",
|
||||
estimator: "Next-test estimator",
|
||||
cohorts: "Sample cohorts",
|
||||
model: "Model & benchmark",
|
||||
methods: "Data & methods",
|
||||
});
|
||||
|
||||
@ -66,13 +65,6 @@ function normalizeCounty(value) {
|
||||
return String(value).trim().toLowerCase().replaceAll("_", " ").replace(/\s+/g, " ");
|
||||
}
|
||||
|
||||
function displayCategory(value) {
|
||||
return String(value)
|
||||
.replaceAll("_", " ")
|
||||
.toLowerCase()
|
||||
.replace(/\b\w/g, (letter) => letter.toUpperCase());
|
||||
}
|
||||
|
||||
function routeFromHash() {
|
||||
const candidate = window.location.hash.replace(/^#/, "").toLowerCase();
|
||||
return Object.hasOwn(ROUTES, candidate) ? candidate : "overview";
|
||||
@ -120,7 +112,7 @@ function setPreviewBanner(manifest) {
|
||||
setText("preview-title", "Development preview");
|
||||
setText(
|
||||
"preview-copy",
|
||||
"These suppressed sample aggregates are not population estimates and must not be used for individual decisions.",
|
||||
"Private 10,000-vehicle development sample · suppressed aggregates only · not population estimates.",
|
||||
);
|
||||
}
|
||||
|
||||
@ -144,25 +136,6 @@ function setUnavailable(error) {
|
||||
setHeaderStatus("error", "Approved data unavailable");
|
||||
}
|
||||
|
||||
function populateSelect(id, values, formatter = displayCategory) {
|
||||
const select = byId(id);
|
||||
if (!select) return;
|
||||
for (const value of values) {
|
||||
const option = document.createElement("option");
|
||||
option.value = String(value);
|
||||
option.textContent = formatter(value);
|
||||
select.append(option);
|
||||
}
|
||||
}
|
||||
|
||||
function disableUnsupportedFilter(id, explanation) {
|
||||
const select = byId(id);
|
||||
if (!select) return;
|
||||
select.disabled = true;
|
||||
select.title = explanation;
|
||||
select.options[0].textContent = explanation;
|
||||
}
|
||||
|
||||
function renderCountyList(coveredCounties) {
|
||||
const container = byId("county-coverage-list");
|
||||
container.replaceChildren();
|
||||
@ -195,12 +168,12 @@ function renderOverview(data) {
|
||||
setText("data-cutoff", latest ? `${latest.year} Q${latest.quarter}` : "Unavailable");
|
||||
setText(
|
||||
"model-version",
|
||||
`Release ${data.manifest.release_id.slice(0, 8)} · ${data.manifest.model_versions.join(" / ")}`,
|
||||
`Final: calibrated logistic · release ${data.manifest.release_id.slice(0, 8)}`,
|
||||
);
|
||||
renderOutcomeTrend(byId("outcome-trend-chart"), rows);
|
||||
setText(
|
||||
"outcome-trend-note",
|
||||
"The current approved bundle publishes binary non-pass rates only. Four-class outcome mix and blank rates are not inferred or displayed.",
|
||||
"Development-sample aggregates only, not a population trend. The bundle publishes binary non-pass rates; four-class outcome mix and blank rates are not inferred.",
|
||||
);
|
||||
renderAgeRisk(byId("age-risk-chart"), data.ageRisk.rows);
|
||||
renderUtahCoverage(byId("utah-coverage-map"), counties);
|
||||
@ -230,7 +203,7 @@ function renderScorecardCards(rows) {
|
||||
heading.textContent = scorecardLabel(row);
|
||||
const meta = document.createElement("div");
|
||||
meta.className = "cohort-card__meta";
|
||||
meta.textContent = `${compactNumber(row.support_rounded)} rounded support`;
|
||||
meta.textContent = `≈${compactNumber(row.support_rounded)} eligible sample episodes (rounded)`;
|
||||
const bar = document.createElement("div");
|
||||
bar.className = "risk-bar";
|
||||
bar.setAttribute("aria-hidden", "true");
|
||||
@ -242,7 +215,7 @@ function renderScorecardCards(rows) {
|
||||
const strong = document.createElement("strong");
|
||||
strong.textContent = percent(row.nonpass_rate);
|
||||
const small = document.createElement("small");
|
||||
small.textContent = "Observed non-pass";
|
||||
small.textContent = "Observed sample non-pass";
|
||||
value.append(strong, small);
|
||||
article.append(heading, meta, bar, value);
|
||||
container.append(article);
|
||||
@ -259,33 +232,14 @@ function updateExplorer() {
|
||||
visibleScorecards = sortedScorecards(matches, sortMode);
|
||||
setText(
|
||||
"result-summary",
|
||||
`${visibleScorecards.length} supported cohort${visibleScorecards.length === 1 ? "" : "s"}; showing up to 18 cards and 12 chart rows.`,
|
||||
`${visibleScorecards.length} supported sample cohort${visibleScorecards.length === 1 ? "" : "s"}; showing up to 18 cards and 12 chart rows.`,
|
||||
);
|
||||
byId("empty-results").hidden = visibleScorecards.length > 0;
|
||||
renderCohortDotPlot(byId("cohort-dot-plot"), visibleScorecards);
|
||||
renderScorecardCards(visibleScorecards);
|
||||
}
|
||||
|
||||
function initializeExplorer(data) {
|
||||
populateSelect("filter-county", data.filters.public_counties);
|
||||
populateSelect("filter-age", data.filters.age_bands, (value) => String(value));
|
||||
populateSelect(
|
||||
"filter-period",
|
||||
data.filters.periods.map((period) => period.year),
|
||||
(value) => String(value),
|
||||
);
|
||||
|
||||
// Current scorecards are make/model aggregates only. These planned controls
|
||||
// remain visible but disabled so the UI never implies unsupported slicing.
|
||||
disableUnsupportedFilter("filter-county", "Unavailable at current scorecard grain");
|
||||
disableUnsupportedFilter("filter-age", "Unavailable at current scorecard grain");
|
||||
disableUnsupportedFilter("filter-fuel", "Fuel not published in this bundle");
|
||||
disableUnsupportedFilter("filter-program", "Program not published in this bundle");
|
||||
disableUnsupportedFilter("filter-period", "Period not published in this scorecard");
|
||||
const adjusted = document.querySelector('input[name="risk-view"][value="adjusted"]');
|
||||
adjusted.disabled = true;
|
||||
adjusted.parentElement.title = "Model-adjusted cohort scorecards are not published.";
|
||||
|
||||
function initializeExplorer() {
|
||||
byId("cohort-search").addEventListener("input", updateExplorer);
|
||||
byId("result-sort").addEventListener("change", updateExplorer);
|
||||
byId("reset-filters").addEventListener("click", () => {
|
||||
@ -297,42 +251,37 @@ function initializeExplorer(data) {
|
||||
updateExplorer();
|
||||
}
|
||||
|
||||
function chooseDiagnostic(rows) {
|
||||
const partitionRank = { calibrate: 3, tune: 2, train: 1 };
|
||||
return [...rows].sort((left, right) => {
|
||||
const preferredLeft = left.model === "logistic_platt" ? 1 : 0;
|
||||
const preferredRight = right.model === "logistic_platt" ? 1 : 0;
|
||||
return (
|
||||
preferredRight - preferredLeft ||
|
||||
(partitionRank[right.partition] ?? 0) - (partitionRank[left.partition] ?? 0)
|
||||
);
|
||||
})[0];
|
||||
function findDiagnostic(rows, model) {
|
||||
return rows.find((row) => row.model === model && row.partition === "calibrate");
|
||||
}
|
||||
|
||||
function renderModel(data) {
|
||||
const finalModel = findDiagnostic(data.diagnostics.rows, "logistic_platt");
|
||||
const benchmark = findDiagnostic(data.diagnostics.rows, "hist_gradient_boosting_platt");
|
||||
const metric = (row, key) => (row ? row[key].toFixed(3) : "—");
|
||||
|
||||
setText("final-ap", metric(finalModel, "average_precision"));
|
||||
setText("final-roc", metric(finalModel, "roc_auc"));
|
||||
setText("final-brier", metric(finalModel, "brier"));
|
||||
setText("benchmark-ap", metric(benchmark, "average_precision"));
|
||||
setText("benchmark-roc", metric(benchmark, "roc_auc"));
|
||||
setText("benchmark-brier", metric(benchmark, "brier"));
|
||||
}
|
||||
|
||||
function renderMethods(data) {
|
||||
renderCoverageHeatmap(byId("coverage-heatmap"), data.coverage.rows);
|
||||
const diagnostic = chooseDiagnostic(data.diagnostics.rows);
|
||||
setText(
|
||||
"diagnostic-scope-label",
|
||||
diagnostic
|
||||
? `${diagnostic.partition === "calibrate" ? "Calibration cohort" : "Development diagnostics"} · ${diagnostic.model}`
|
||||
: "Development diagnostics · model unavailable",
|
||||
);
|
||||
setText("diagnostic-ap", diagnostic ? diagnostic.average_precision.toFixed(3) : "—");
|
||||
setText("diagnostic-brier", diagnostic ? diagnostic.brier.toFixed(3) : "—");
|
||||
setText("diagnostic-n", "Not published");
|
||||
renderNoCalibration(byId("calibration-chart"));
|
||||
}
|
||||
|
||||
function renderDashboard(data) {
|
||||
dashboardData = data;
|
||||
setPreviewBanner(data.manifest);
|
||||
renderOverview(data);
|
||||
initializeExplorer(data);
|
||||
initializeExplorer();
|
||||
renderModel(data);
|
||||
renderMethods(data);
|
||||
setHeaderStatus(
|
||||
"ready",
|
||||
data.manifest.development_preview ? "Validated development aggregates" : "Validated public aggregates",
|
||||
data.manifest.development_preview ? "Validated sample aggregates" : "Validated public aggregates",
|
||||
);
|
||||
}
|
||||
|
||||
@ -349,4 +298,4 @@ async function initialize() {
|
||||
|
||||
initialize();
|
||||
|
||||
export { displayCategory, normalizeCounty, routeFromHash };
|
||||
export { normalizeCounty, routeFromHash };
|
||||
|
||||
@ -109,7 +109,7 @@ export function renderOutcomeTrend(container, rows) {
|
||||
</svg>`;
|
||||
container.setAttribute(
|
||||
"aria-label",
|
||||
`Quarterly aggregate non-pass trend. ${description}`,
|
||||
`Development-sample quarterly aggregate non-pass rates, not population estimates. ${description}`,
|
||||
);
|
||||
}
|
||||
|
||||
@ -155,7 +155,7 @@ export function renderAgeRisk(container, rows) {
|
||||
</svg>`;
|
||||
container.setAttribute(
|
||||
"aria-label",
|
||||
`Non-pass risk by vehicle-age band. ${points
|
||||
`Observed development-sample non-pass rates by vehicle-age band, not population estimates. ${points
|
||||
.map((point) => `${point.age_band}: ${percent(point.nonpass_rate)}`)
|
||||
.join("; ")}`,
|
||||
);
|
||||
@ -199,7 +199,7 @@ export function renderCohortDotPlot(container, rows) {
|
||||
</svg>`;
|
||||
container.setAttribute(
|
||||
"aria-label",
|
||||
`Ranked supported cohort non-pass risk. ${points
|
||||
`Supported development-sample cohort associations, not reliability rankings or population estimates. ${points
|
||||
.map((point) => `${point.prior_make} ${point.prior_model}: ${percent(point.nonpass_rate)}`)
|
||||
.join("; ")}`,
|
||||
);
|
||||
@ -234,7 +234,7 @@ export function renderUtahCoverage(container, coveredCounties) {
|
||||
container.setAttribute(
|
||||
"aria-label",
|
||||
coveredCounties.length
|
||||
? `Utah feed coverage includes ${coveredCounties.join(", ")}. Other counties are unavailable.`
|
||||
? `Development-sample feed availability includes ${coveredCounties.join(", ")}. This is not an outcome ranking; other counties are unavailable.`
|
||||
: "No county feed coverage is available.",
|
||||
);
|
||||
}
|
||||
@ -267,14 +267,7 @@ export function renderCoverageHeatmap(container, rows) {
|
||||
</div>`;
|
||||
container.setAttribute(
|
||||
"aria-label",
|
||||
`Source-era coverage from ${years[0]} through ${years[years.length - 1]} for ${eras.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
|
||||
export function renderNoCalibration(container) {
|
||||
emptyChart(
|
||||
container,
|
||||
"Calibration-bin data is not published in this development bundle. No calibration curve is shown.",
|
||||
`Development-sample source-era coverage from ${years[0]} through ${years[years.length - 1]} for ${eras.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@ -1,5 +1,10 @@
|
||||
const SCHEMA_VERSION = "dashboard_data_v1";
|
||||
const APPROVED_PARTITIONS = Object.freeze(["train", "tune", "calibrate"]);
|
||||
const APPROVED_TARGET = "first-attempt next-episode binary non-pass rate";
|
||||
const REQUIRED_CALIBRATION_MODELS = Object.freeze([
|
||||
"hist_gradient_boosting_platt",
|
||||
"logistic_platt",
|
||||
]);
|
||||
const ENVELOPE_KEYS = Object.freeze([
|
||||
"development_preview",
|
||||
"population_estimate_allowed",
|
||||
@ -20,15 +25,22 @@ export const REQUIRED_ASSETS = Object.freeze({
|
||||
const SENSITIVE_KEY_PARTS = new Set([
|
||||
"address",
|
||||
"certificate",
|
||||
"credential",
|
||||
"email",
|
||||
"inspector",
|
||||
"internal",
|
||||
"ip",
|
||||
"owner",
|
||||
"password",
|
||||
"pid",
|
||||
"plate",
|
||||
"prediction",
|
||||
"probability",
|
||||
"raw",
|
||||
"secret",
|
||||
"session",
|
||||
"station",
|
||||
"technician",
|
||||
"token",
|
||||
"user",
|
||||
"vin",
|
||||
@ -337,6 +349,11 @@ export function validateAssetSet(rawAssets) {
|
||||
requireExactKeys(rawAssets, Object.keys(REQUIRED_ASSETS), "asset set");
|
||||
const manifest = rawAssets.manifest;
|
||||
validateEnvelope(manifest, "data_manifest");
|
||||
if (!manifest.development_preview || manifest.population_estimate_allowed) {
|
||||
throw new DataContractError(
|
||||
"The dashboard accepts development-sample, non-population assets only.",
|
||||
);
|
||||
}
|
||||
requireExactKeys(
|
||||
manifest,
|
||||
[
|
||||
@ -427,6 +444,9 @@ export function validateAssetSet(rawAssets) {
|
||||
requireInteger(manifest.definitions[key], `data_manifest.definitions.${key}`, { min: 1 });
|
||||
}
|
||||
requireString(manifest.definitions.target, "data_manifest.definitions.target");
|
||||
if (manifest.definitions.target !== APPROVED_TARGET) {
|
||||
throw new DataContractError("data_manifest.definitions.target is outside the approved scope.");
|
||||
}
|
||||
if (!Array.isArray(manifest.assets)) {
|
||||
throw new DataContractError("data_manifest.assets must be an array.");
|
||||
}
|
||||
@ -463,6 +483,16 @@ export function validateAssetSet(rawAssets) {
|
||||
throw new DataContractError(`model_diagnostics.rows[${index}].model is not cataloged.`);
|
||||
}
|
||||
}
|
||||
for (const model of REQUIRED_CALIBRATION_MODELS) {
|
||||
const matches = validated.diagnostics.rows.filter(
|
||||
(row) => row.model === model && row.partition === "calibrate",
|
||||
);
|
||||
if (matches.length !== 1) {
|
||||
throw new DataContractError(
|
||||
`model_diagnostics must contain exactly one approved calibration row for ${model}.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
const minimumSupport = manifest.definitions.suppression_min_support;
|
||||
const supportRounding = manifest.definitions.support_rounding;
|
||||
for (const name of ["overview", "ageRisk", "scorecards", "coverage"]) {
|
||||
|
||||
@ -37,12 +37,16 @@
|
||||
}
|
||||
|
||||
html {
|
||||
max-width: 100%;
|
||||
overflow-x: clip;
|
||||
scroll-behavior: smooth;
|
||||
}
|
||||
|
||||
body {
|
||||
min-width: 320px;
|
||||
max-width: 100%;
|
||||
margin: 0;
|
||||
overflow-x: clip;
|
||||
color: var(--ink-950);
|
||||
background:
|
||||
radial-gradient(circle at 7% 2%, rgb(231 216 185 / 35%), transparent 30rem),
|
||||
@ -165,6 +169,11 @@ h2 {
|
||||
margin-right: 0.35rem;
|
||||
}
|
||||
|
||||
.preview-banner > div {
|
||||
min-width: 0;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.preview-banner__icon {
|
||||
color: var(--amber-700);
|
||||
}
|
||||
@ -182,7 +191,8 @@ h2 {
|
||||
.site-header__inner,
|
||||
.primary-nav__inner,
|
||||
.page-shell {
|
||||
width: min(calc(100% - 2rem), var(--content));
|
||||
width: calc(100% - 2rem);
|
||||
max-width: var(--content);
|
||||
margin-inline: auto;
|
||||
}
|
||||
|
||||
@ -302,10 +312,15 @@ h2 {
|
||||
}
|
||||
|
||||
main {
|
||||
width: 100%;
|
||||
min-width: 0;
|
||||
max-width: 100%;
|
||||
min-height: 65vh;
|
||||
}
|
||||
|
||||
.view {
|
||||
min-width: 0;
|
||||
max-width: 100%;
|
||||
padding: clamp(2.5rem, 6vw, 5rem) 0 5rem;
|
||||
}
|
||||
|
||||
@ -318,9 +333,17 @@ main {
|
||||
}
|
||||
|
||||
.page-heading {
|
||||
min-width: 0;
|
||||
margin-bottom: clamp(2rem, 5vw, 3.4rem);
|
||||
}
|
||||
|
||||
.page-heading > *,
|
||||
.dashboard-grid > *,
|
||||
.kpi-grid > *,
|
||||
.model-comparison > * {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.page-heading--split {
|
||||
display: flex;
|
||||
align-items: end;
|
||||
@ -370,6 +393,26 @@ main {
|
||||
font-size: 1.04rem;
|
||||
}
|
||||
|
||||
.scope-strip {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
gap: 0.7rem 1rem;
|
||||
margin: -1.4rem 0 1rem;
|
||||
padding: 0.75rem 1rem;
|
||||
border: 1px solid #e9c77d;
|
||||
border-radius: var(--radius-sm);
|
||||
color: #503407;
|
||||
background: rgb(249 236 205 / 72%);
|
||||
font-size: 0.78rem;
|
||||
}
|
||||
|
||||
.scope-strip strong {
|
||||
flex: 0 0 auto;
|
||||
font-size: 0.72rem;
|
||||
letter-spacing: 0.06em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.unavailable-state {
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
@ -694,6 +737,14 @@ main {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.filter-panel__note {
|
||||
margin: 1rem 0 0;
|
||||
padding-top: 1rem;
|
||||
border-top: 1px solid var(--sand-200);
|
||||
color: var(--ink-650);
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
|
||||
.text-button {
|
||||
padding: 0.2rem;
|
||||
border: 0;
|
||||
@ -868,6 +919,124 @@ fieldset:disabled .field select {
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.model-comparison {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
gap: 1rem;
|
||||
}
|
||||
|
||||
.model-card {
|
||||
position: relative;
|
||||
padding: clamp(1.25rem, 3vw, 1.7rem);
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--sand-200);
|
||||
border-radius: var(--radius-lg);
|
||||
background: rgb(255 255 255 / 86%);
|
||||
box-shadow: var(--shadow-sm);
|
||||
}
|
||||
|
||||
.model-card::before {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
right: 0;
|
||||
left: 0;
|
||||
height: 4px;
|
||||
background: var(--sand-400);
|
||||
content: "";
|
||||
}
|
||||
|
||||
.model-card--final {
|
||||
border-color: #9bc9c6;
|
||||
background: linear-gradient(145deg, rgb(220 239 238 / 58%), rgb(255 255 255 / 92%));
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.model-card--final::before {
|
||||
background: var(--teal-700);
|
||||
}
|
||||
|
||||
.model-card__topline {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 0.75rem;
|
||||
margin-bottom: 1.2rem;
|
||||
color: var(--ink-500);
|
||||
font-size: 0.68rem;
|
||||
}
|
||||
|
||||
.decision-badge {
|
||||
padding: 0.25rem 0.55rem;
|
||||
border: 1px solid var(--sand-400);
|
||||
border-radius: 999px;
|
||||
color: #6a4c18;
|
||||
background: var(--amber-100);
|
||||
font-size: 0.64rem;
|
||||
font-weight: 800;
|
||||
letter-spacing: 0.06em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.decision-badge--final {
|
||||
border-color: #83bbb7;
|
||||
color: var(--teal-800);
|
||||
background: var(--teal-100);
|
||||
}
|
||||
|
||||
.model-card > p {
|
||||
min-height: 5.1em;
|
||||
margin-bottom: 1.2rem;
|
||||
color: var(--ink-650);
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
|
||||
.metric-list {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||
gap: 0.5rem;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.metric-list div {
|
||||
padding: 0.72rem;
|
||||
border-radius: var(--radius-sm);
|
||||
background: rgb(237 240 237 / 78%);
|
||||
}
|
||||
|
||||
.metric-list dt {
|
||||
color: var(--ink-500);
|
||||
font-size: 0.62rem;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.metric-list dd {
|
||||
margin: 0.18rem 0 0;
|
||||
font-family: var(--font-display);
|
||||
font-size: 1.35rem;
|
||||
}
|
||||
|
||||
.publication-boundary {
|
||||
display: grid;
|
||||
grid-template-columns: 1.15fr 0.85fr;
|
||||
gap: 2rem;
|
||||
align-items: start;
|
||||
margin-top: 1rem;
|
||||
padding: clamp(1.4rem, 3vw, 1.9rem);
|
||||
border-radius: var(--radius-lg);
|
||||
color: var(--white);
|
||||
background: var(--ink-950);
|
||||
}
|
||||
|
||||
.publication-boundary p:last-child {
|
||||
margin: 0;
|
||||
color: #cad7d5;
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
|
||||
.publication-boundary .eyebrow {
|
||||
color: #9bd4d1;
|
||||
}
|
||||
|
||||
.locked-panel {
|
||||
display: flex;
|
||||
gap: 1.2rem;
|
||||
@ -1242,6 +1411,8 @@ fieldset:disabled .field select {
|
||||
.dashboard-grid--wide,
|
||||
.explorer-layout,
|
||||
.split-section,
|
||||
.model-comparison,
|
||||
.publication-boundary,
|
||||
.privacy-panel {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
@ -1260,10 +1431,26 @@ fieldset:disabled .field select {
|
||||
}
|
||||
|
||||
@media (max-width: 660px) {
|
||||
.preview-banner {
|
||||
align-items: flex-start;
|
||||
justify-content: flex-start;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.preview-banner__icon {
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
|
||||
.site-header__inner {
|
||||
min-height: 68px;
|
||||
}
|
||||
|
||||
.page-shell {
|
||||
width: auto;
|
||||
max-width: calc(100% - 2rem);
|
||||
margin-inline: 1rem;
|
||||
}
|
||||
|
||||
.header-status,
|
||||
.brand__tagline {
|
||||
display: none;
|
||||
@ -1286,6 +1473,21 @@ fieldset:disabled .field select {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.scope-strip {
|
||||
align-items: flex-start;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.lede,
|
||||
.data-stamp,
|
||||
.scope-strip,
|
||||
.kpi-card,
|
||||
.model-card,
|
||||
.panel {
|
||||
max-width: 100%;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.data-stamp {
|
||||
width: 100%;
|
||||
margin-top: 1.3rem;
|
||||
@ -1296,10 +1498,15 @@ fieldset:disabled .field select {
|
||||
.method-grid,
|
||||
.cohort-cards,
|
||||
.form-grid,
|
||||
.method-preview {
|
||||
.method-preview,
|
||||
.metric-list {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.model-card > p {
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.kpi-card {
|
||||
min-height: 130px;
|
||||
}
|
||||
|
||||
@ -52,13 +52,25 @@ function filesRecursively(directory) {
|
||||
});
|
||||
}
|
||||
|
||||
function viewSource(html, route) {
|
||||
const marker = `<section class="view" id="view-${route}"`;
|
||||
const start = html.indexOf(marker);
|
||||
if (start < 0) return "";
|
||||
const remaining = html.slice(start + marker.length);
|
||||
const nextView = remaining.indexOf('<section class="view" id="view-');
|
||||
const mainEnd = remaining.indexOf("</main>");
|
||||
const candidates = [nextView, mainEnd].filter((index) => index >= 0);
|
||||
const end = candidates.length ? Math.min(...candidates) : remaining.length;
|
||||
return html.slice(start, start + marker.length + end);
|
||||
}
|
||||
|
||||
test("static shell exposes four semantic navigable views", () => {
|
||||
const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8");
|
||||
assert.match(html, /<header\b/);
|
||||
assert.match(html, /<nav\b[^>]*aria-label="Dashboard views"/);
|
||||
assert.match(html, /<main\b/);
|
||||
assert.match(html, /<footer\b/);
|
||||
for (const route of ["overview", "reliability", "estimator", "methods"]) {
|
||||
for (const route of ["overview", "cohorts", "model", "methods"]) {
|
||||
assert.match(html, new RegExp(`data-route="${route}"`));
|
||||
assert.match(html, new RegExp(`data-view="${route}"`));
|
||||
}
|
||||
@ -67,20 +79,45 @@ test("static shell exposes four semantic navigable views", () => {
|
||||
assert.doesNotMatch(html, /<strong>Locked test<\/strong>|<small>Final evaluation<\/small>/);
|
||||
});
|
||||
|
||||
test("estimator remains disabled and contains no identifying input", () => {
|
||||
test("narrow layouts constrain body content while preserving local nav scrolling", () => {
|
||||
const css = readFileSync(path.join(DASHBOARD_ROOT, "styles.css"), "utf8");
|
||||
assert.match(css, /html\s*{[^}]*max-width:\s*100%;[^}]*overflow-x:\s*clip;/s);
|
||||
assert.match(css, /body\s*{[^}]*max-width:\s*100%;[^}]*overflow-x:\s*clip;/s);
|
||||
assert.match(css, /\.primary-nav__inner\s*{[^}]*overflow-x:\s*auto;/s);
|
||||
const mobile = css.match(/@media \(max-width: 660px\)\s*{[\s\S]*?(?=@media|$)/)?.[0];
|
||||
assert.ok(mobile, "mobile breakpoint is present");
|
||||
assert.match(mobile, /\.page-shell\s*{[^}]*width:\s*auto;[^}]*max-width:\s*calc\(100% - 2rem\);/s);
|
||||
assert.match(mobile, /\.preview-banner\s*{[^}]*justify-content:\s*flex-start;/s);
|
||||
assert.match(mobile, /overflow-wrap:\s*anywhere;/);
|
||||
});
|
||||
|
||||
test("model view declares the final model, benchmark role, and publication boundary", () => {
|
||||
const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8");
|
||||
const estimator = html.match(/<section class="view" id="view-estimator"[\s\S]*?<\/section>/)?.[0];
|
||||
assert.ok(estimator, "estimator section is present");
|
||||
assert.match(estimator, /<fieldset disabled>/);
|
||||
const controlAttributes = [...estimator.matchAll(/<(?:input|select|textarea)\b([^>]*)>/g)].map(
|
||||
(match) => match[1],
|
||||
);
|
||||
for (const attributes of controlAttributes) {
|
||||
assert.doesNotMatch(
|
||||
attributes,
|
||||
/(?:name|id)\s*=\s*["'][^"']*(?:vin|plate|address|station|free.?text|zip)[^"']*["']/i,
|
||||
);
|
||||
const modelView = viewSource(html, "model");
|
||||
assert.ok(modelView, "model section is present");
|
||||
assert.match(modelView, /Calibrated logistic regression/);
|
||||
assert.match(modelView, /Final prototype/);
|
||||
assert.match(modelView, /Calibrated boosted tree/);
|
||||
assert.match(modelView, /Benchmark only/);
|
||||
assert.match(modelView, /partition used to fit Platt calibration/i);
|
||||
assert.match(modelView, /not independent final-performance estimates/i);
|
||||
assert.match(modelView, /No vehicle-level prediction service/);
|
||||
assert.doesNotMatch(modelView, /<(?:input|select|textarea|form)\b/i);
|
||||
assert.doesNotMatch(modelView, /prediction_lookup|planned tool|future approved output/i);
|
||||
});
|
||||
|
||||
test("every result-facing view visibly labels the sample and non-population scope", () => {
|
||||
const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8");
|
||||
assert.match(html, /Private 10,000-vehicle development sample/);
|
||||
for (const route of ["overview", "cohorts", "model", "methods"]) {
|
||||
const view = viewSource(html, route);
|
||||
assert.ok(view, `${route} view is present`);
|
||||
assert.match(view, /scope-strip/);
|
||||
assert.match(view, /sample/i);
|
||||
assert.match(view, /not[^.]{0,80}population|population estimate/i);
|
||||
}
|
||||
assert.match(html, /not a county outcome ranking/i);
|
||||
assert.match(html, /not a reliability ranking or population comparison/i);
|
||||
});
|
||||
|
||||
test("all required generated assets satisfy the browser contract", () => {
|
||||
@ -91,6 +128,12 @@ test("all required generated assets satisfy the browser contract", () => {
|
||||
assert.equal(validated.manifest.definitions.locked_test_metrics_published, false);
|
||||
assert.match(validated.manifest.release_id, /^[0-9a-f]{64}$/);
|
||||
assert.ok(validated.manifest.model_versions.length > 0);
|
||||
assert.ok(validated.diagnostics.rows.some(
|
||||
(row) => row.model === "logistic_platt" && row.partition === "calibrate",
|
||||
));
|
||||
assert.ok(validated.diagnostics.rows.some(
|
||||
(row) => row.model === "hist_gradient_boosting_platt" && row.partition === "calibrate",
|
||||
));
|
||||
assert.deepEqual(
|
||||
validated.filters.partitions,
|
||||
["train", "tune", "calibrate"],
|
||||
@ -151,10 +194,37 @@ test("contract fails closed for missing, inconsistent, or sensitive data", () =>
|
||||
inconsistent.ageRisk.population_estimate_allowed = true;
|
||||
assert.throws(() => validateAssetSet(inconsistent), DataContractError);
|
||||
|
||||
const populationClaim = assetSet();
|
||||
populationClaim.manifest.population_estimate_allowed = true;
|
||||
assert.throws(() => validateAssetSet(populationClaim), DataContractError);
|
||||
|
||||
const broadenedTarget = assetSet();
|
||||
broadenedTarget.manifest.definitions.target = "any inspection outcome";
|
||||
assert.throws(() => validateAssetSet(broadenedTarget), DataContractError);
|
||||
|
||||
const sensitive = assetSet();
|
||||
sensitive.scorecards.rows[0].vehicle_token = "not-public";
|
||||
assert.throws(() => validateAssetSet(sensitive), DataContractError);
|
||||
|
||||
for (const deniedField of [
|
||||
"vin",
|
||||
"plate",
|
||||
"zip",
|
||||
"station",
|
||||
"technician_id",
|
||||
"inspector_id",
|
||||
"row_prediction",
|
||||
"probability",
|
||||
"raw_json",
|
||||
"credential",
|
||||
"password",
|
||||
"secret",
|
||||
]) {
|
||||
const denied = assetSet();
|
||||
denied.overview.rows[0][deniedField] = "not-public";
|
||||
assert.throws(() => validateAssetSet(denied), DataContractError, deniedField);
|
||||
}
|
||||
|
||||
const extraRowField = assetSet();
|
||||
extraRowField.overview.rows[0].note = "unapproved";
|
||||
assert.throws(() => validateAssetSet(extraRowField), DataContractError);
|
||||
@ -167,6 +237,20 @@ test("contract fails closed for missing, inconsistent, or sensitive data", () =>
|
||||
extraPartition.filters.partitions.push("locked_test");
|
||||
assert.throws(() => validateAssetSet(extraPartition), DataContractError);
|
||||
|
||||
const missingFinalDiagnostic = assetSet();
|
||||
missingFinalDiagnostic.diagnostics.rows = missingFinalDiagnostic.diagnostics.rows.filter(
|
||||
(row) => !(row.model === "logistic_platt" && row.partition === "calibrate"),
|
||||
);
|
||||
assert.throws(() => validateAssetSet(missingFinalDiagnostic), DataContractError);
|
||||
|
||||
const duplicatedBenchmarkDiagnostic = assetSet();
|
||||
duplicatedBenchmarkDiagnostic.diagnostics.rows.push({
|
||||
...duplicatedBenchmarkDiagnostic.diagnostics.rows.find(
|
||||
(row) => row.model === "hist_gradient_boosting_platt" && row.partition === "calibrate",
|
||||
),
|
||||
});
|
||||
assert.throws(() => validateAssetSet(duplicatedBenchmarkDiagnostic), DataContractError);
|
||||
|
||||
const unsortedVersions = assetSet();
|
||||
unsortedVersions.manifest.model_versions = ["z_v1", "a_v1"];
|
||||
assert.throws(() => validateAssetSet(unsortedVersions), DataContractError);
|
||||
@ -194,3 +278,15 @@ test("dashboard source has no external runtime dependency or credential marker",
|
||||
assert.doesNotMatch(source, /<script[^>]+src=["']https?:|@import\s+url\(["']?https?:/i);
|
||||
}
|
||||
});
|
||||
|
||||
test("public data directory contains only the approved suppressed aggregate bundle", () => {
|
||||
const expected = Object.values(REQUIRED_ASSETS).sort();
|
||||
const observed = readdirSync(PUBLIC_DATA).sort();
|
||||
assert.deepEqual(observed, expected);
|
||||
|
||||
const serialized = observed
|
||||
.map((filename) => readFileSync(path.join(PUBLIC_DATA, filename), "utf8"))
|
||||
.join("\n");
|
||||
assert.doesNotMatch(serialized, /\b[A-HJ-NPR-Z0-9]{17}\b/);
|
||||
assert.doesNotMatch(serialized, /(?:postgres(?:ql)?:\/\/|PGPASSWORD|BEGIN [A-Z ]*PRIVATE KEY)/i);
|
||||
});
|
||||
|
||||
@ -1,115 +1,84 @@
|
||||
graph TD
|
||||
%% Data Sources
|
||||
PG[(PostgreSQL<br/>countydata)]
|
||||
DMV[("DMV Data<br/>~18M records<br/>2011-2024")]
|
||||
INSP[("Inspection Data<br/>~19.3M records<br/>2010-2026")]
|
||||
flowchart TD
|
||||
SOURCE[(Private inspection histories<br/>read-only source)]
|
||||
SAMPLE[Private page-sampled histories<br/>up to 10,000 vehicles<br/>keyed vehicle tokens]
|
||||
SOURCE -->|read-only bounded export| SAMPLE
|
||||
|
||||
PG --- DMV
|
||||
PG --- INSP
|
||||
SAMPLE --> STAGE[Normalize labels<br/>deduplicate and validate]
|
||||
STAGE --> EPISODES[Build 30-day episodes<br/>first attempt is the target]
|
||||
EPISODES --> MART[Point-in-time feature mart<br/>returning vehicles only]
|
||||
|
||||
%% Extraction
|
||||
PG -->|"export_history_sample.py<br/>(VIN hashing, page sampling)"| CSV["history_sample_10000.csv.gz<br/>+ manifest"]
|
||||
|
||||
%% Feature Mart Build
|
||||
CSV -->|"build_feature_mart.py"| DUCKDB
|
||||
|
||||
subgraph DuckDB["DuckDB Local Warehouse"]
|
||||
direction TB
|
||||
SQL20["20_stage_events.sql<br/>Normalize & validate"]
|
||||
SQL21["21_build_episodes.sql<br/>Group episodes (30-day gap)"]
|
||||
SQL22["22_build_features.sql<br/>Point-in-time feature windows"]
|
||||
SQL20 --> SQL21 --> SQL22
|
||||
end
|
||||
|
||||
DUCKDB --> MART["inspection_feature_mart.parquet<br/>(episode × features, temporal partitions)"]
|
||||
|
||||
%% Temporal Partitions
|
||||
subgraph Partitions["Temporal Splits"]
|
||||
TRAIN["Train: 2016–2022"]
|
||||
TUNE["Tune: 2023"]
|
||||
CAL["Calibrate: 2024"]
|
||||
TEST["Locked Test: 2025"]
|
||||
DRIFT["Shadow Drift: 2026"]
|
||||
end
|
||||
MART -.- Partitions
|
||||
|
||||
%% Model Training
|
||||
MART -->|"train_baselines.py"| BL["artifacts/baselines/baseline_v1/<br/>Logistic Regression +<br/>Prevalence & Prior-Outcome"]
|
||||
MART -->|"train_tree_model.py"| TREE["artifacts/tree/hist_gradient_boosting_v1/<br/>Histogram Gradient Boosting"]
|
||||
|
||||
%% Model Outputs
|
||||
BL --> ARTIFACTS["Model Artifacts<br/>manifest.json, metrics.json,<br/>model.joblib, calibration_bins.csv"]
|
||||
TREE --> ARTIFACTS
|
||||
|
||||
%% Dashboard Export
|
||||
MART -->|"export_dashboard_data.py<br/>(privacy suppression, aggregation)"| JSON
|
||||
|
||||
subgraph JSON["dashboard/public/data/"]
|
||||
direction TB
|
||||
DM["data_manifest.json"]
|
||||
OV["overview_period_county.json"]
|
||||
CS["cohort_scorecard.json"]
|
||||
ARC["age_risk_curve.json"]
|
||||
CQ["coverage_quality.json"]
|
||||
FC["filter_catalog.json"]
|
||||
MD["model_diagnostics.json"]
|
||||
end
|
||||
|
||||
ARTIFACTS -.->|metrics & manifest| JSON
|
||||
|
||||
%% Dashboard Frontend
|
||||
JSON --> SERVER["server.mjs<br/>Node.js HTTP server"]
|
||||
|
||||
subgraph Dashboard["Browser SPA"]
|
||||
direction TB
|
||||
APP["app.js<br/>Routing & pages"]
|
||||
CHARTS["charts.js<br/>D3 visualizations"]
|
||||
DATA["data.js<br/>Data loading & filters"]
|
||||
APP --- CHARTS
|
||||
APP --- DATA
|
||||
end
|
||||
|
||||
SERVER --> Dashboard
|
||||
|
||||
subgraph Pages["Dashboard Pages"]
|
||||
P1["Overview<br/>KPIs, trends, county map"]
|
||||
P2["Reliability Explorer<br/>Make/model comparison"]
|
||||
P3["Next-Test Estimator<br/>Risk prediction"]
|
||||
P4["Data & Methods<br/>Coverage, limitations"]
|
||||
end
|
||||
|
||||
Dashboard --> Pages
|
||||
|
||||
%% Privacy Controls
|
||||
subgraph Privacy["Privacy Controls"]
|
||||
subgraph SPLITS[Private development chronology]
|
||||
direction LR
|
||||
S1["≥100 inspections per cell"]
|
||||
S2["≥100 distinct vehicles"]
|
||||
S3["≥10 outcome records"]
|
||||
S4["No VINs/plates/ZIPs"]
|
||||
CONTEXT[2010-15<br/>history context]
|
||||
TRAIN[2016-22<br/>train]
|
||||
TUNE[2023<br/>tune]
|
||||
CAL[2024<br/>calibrate]
|
||||
HOLDOUT[2025<br/>one-time holdout]
|
||||
SHADOW[2026 partial<br/>shadow context]
|
||||
end
|
||||
MART -.-> SPLITS
|
||||
|
||||
JSON -.- Privacy
|
||||
MART --> LOGISTIC[Regularized logistic + Platt<br/><b>final prototype model</b>]
|
||||
MART --> TREE[Histogram gradient boosting + Platt<br/><b>benchmark only</b>]
|
||||
|
||||
%% Testing
|
||||
subgraph Tests["Test Suite"]
|
||||
LOGISTIC --> PRIVATE[Private model artifacts<br/>manifests, checksums, metrics]
|
||||
TREE --> PRIVATE
|
||||
HOLDOUT -.-> REPORT[Private/report-only<br/>one-time comparison]
|
||||
PRIVATE -.-> REPORT
|
||||
|
||||
MART --> EXPORT[Aggregate, suppress,<br/>round, validate]
|
||||
PRIVATE -.->|pre-2025 diagnostics only| EXPORT
|
||||
|
||||
subgraph PUBLIC[dashboard/public/data - public boundary]
|
||||
direction TB
|
||||
MANIFEST[data_manifest.json]
|
||||
OVERVIEW[overview_period_county.json]
|
||||
COHORTS[cohort_scorecard.json]
|
||||
AGE[age_risk_curve.json]
|
||||
COVERAGE[coverage_quality.json]
|
||||
FILTERS[filter_catalog.json]
|
||||
DIAGNOSTICS[model_diagnostics.json]
|
||||
HASHES[sha256_manifest.json]
|
||||
end
|
||||
EXPORT --> PUBLIC
|
||||
|
||||
PUBLIC --> CONTRACT[Fail-closed browser validation]
|
||||
CONTRACT --> SITE[Static dashboard<br/>semantic HTML + CSS<br/>vanilla ES modules + inline SVG]
|
||||
|
||||
subgraph VIEWS[Presentation-ready views]
|
||||
direction TB
|
||||
V1[Overview]
|
||||
V2[Sample cohorts]
|
||||
V3[Model & benchmark]
|
||||
V4[Data & methods]
|
||||
end
|
||||
SITE --> VIEWS
|
||||
|
||||
subgraph GUARDRAILS[Publication guardrails]
|
||||
direction TB
|
||||
G1[Private 10,000-vehicle sample<br/>not population estimates]
|
||||
G2[Minimum episode, vehicle,<br/>and binary-class support]
|
||||
G3[No identifiers, raw/operational rows,<br/>credentials, or row-level predictions]
|
||||
G4[No county population rankings<br/>or vehicle-level prediction service]
|
||||
end
|
||||
EXPORT -.-> GUARDRAILS
|
||||
|
||||
SITE --> BOLT[Separate dashboard-only<br/>Bolt project]
|
||||
|
||||
subgraph TESTS[Verification]
|
||||
direction LR
|
||||
PT["Python unittest<br/>(feature mart, models,<br/>export, baselines)"]
|
||||
CT["Node.js contract tests<br/>(dashboard JSON schemas)"]
|
||||
PY[Python pipeline/model/export tests]
|
||||
JS[Node dashboard contract tests]
|
||||
end
|
||||
MART -.-> PY
|
||||
PUBLIC -.-> JS
|
||||
|
||||
MART -.- PT
|
||||
JSON -.- CT
|
||||
classDef private fill:#fff3e0,stroke:#b45309,color:#3b2415
|
||||
classDef model fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e
|
||||
classDef public fill:#ecfdf5,stroke:#047857,color:#064e3b
|
||||
classDef warning fill:#fef2f2,stroke:#b91c1c,color:#7f1d1d
|
||||
|
||||
%% Styling
|
||||
classDef source fill:#e1f5fe,stroke:#0288d1
|
||||
classDef process fill:#fff3e0,stroke:#f57c00
|
||||
classDef storage fill:#e8f5e9,stroke:#388e3c
|
||||
classDef dashboard fill:#fce4ec,stroke:#c62828
|
||||
classDef test fill:#f3e5f5,stroke:#7b1fa2
|
||||
|
||||
class PG,DMV,INSP source
|
||||
class CSV,MART,ARTIFACTS,DUCKDB storage
|
||||
class BL,TREE process
|
||||
class SERVER,Dashboard,Pages,JSON dashboard
|
||||
class Tests test
|
||||
class SOURCE,SAMPLE,STAGE,EPISODES,MART,PRIVATE,REPORT private
|
||||
class LOGISTIC,TREE model
|
||||
class PUBLIC,CONTRACT,SITE,VIEWS,BOLT public
|
||||
class GUARDRAILS warning
|
||||
|
||||
187
docs/bolt_deployment.md
Normal file
187
docs/bolt_deployment.md
Normal file
@ -0,0 +1,187 @@
|
||||
# Dashboard-only Bolt deployment
|
||||
|
||||
Last reviewed: 2026-07-21
|
||||
|
||||
## Non-negotiable publication boundary
|
||||
|
||||
Publish a **separate Bolt project or GitHub repository containing only the
|
||||
contents of `dashboard/`**. In that project, `index.html` must be at the project
|
||||
root.
|
||||
|
||||
Do not import the full Utah Vehicle Health source repository into Bolt. Do not
|
||||
depend on Bolt supporting a `dashboard` working-root setting. The source
|
||||
repository contains private-pipeline structure that a static host does not need,
|
||||
even when local secrets and data are Git-ignored.
|
||||
|
||||
The deployment project must never contain `.env` files, credentials, private
|
||||
data or marts, database files, model artifacts, SQL, pipeline scripts, repository
|
||||
history from outside `dashboard/`, raw JSON, operational records, or row-level
|
||||
predictions.
|
||||
|
||||
## 1. Verify the source dashboard locally
|
||||
|
||||
From the Utah Vehicle Health repository root:
|
||||
|
||||
```bash
|
||||
npm --prefix dashboard test
|
||||
node dashboard/server.mjs
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:4173` and verify:
|
||||
|
||||
- the development-preview banner says the values come from the private
|
||||
10,000-vehicle sample and are not population estimates;
|
||||
- the routes are Overview, Sample cohorts, Model & benchmark, and Data & methods;
|
||||
- the final model is calibrated logistic regression and the boosted tree is
|
||||
benchmark-only;
|
||||
- model cards show 2024 calibration-fit diagnostics, not the opened 2025
|
||||
holdout comparison;
|
||||
- no vehicle-level prediction service or input form exists;
|
||||
- county displays communicate feed coverage rather than population rankings;
|
||||
and
|
||||
- a missing or invalid aggregate bundle produces an unavailable state, not
|
||||
fallback estimates.
|
||||
|
||||
Stop the local server with `Ctrl-C` after verification.
|
||||
|
||||
## 2. Create a fresh dashboard-only artifact
|
||||
|
||||
Use a new directory and, preferably, a new deployment repository. Copy the
|
||||
**contents** of `dashboard/`, not the directory's parent and not the main
|
||||
repository's `.git` history.
|
||||
|
||||
The reviewed project tree is:
|
||||
|
||||
```text
|
||||
index.html
|
||||
styles.css
|
||||
package.json
|
||||
server.mjs
|
||||
README.md
|
||||
js/
|
||||
app.js
|
||||
charts.js
|
||||
data.js
|
||||
public/data/
|
||||
age_risk_curve.json
|
||||
cohort_scorecard.json
|
||||
coverage_quality.json
|
||||
data_manifest.json
|
||||
filter_catalog.json
|
||||
model_diagnostics.json
|
||||
overview_period_county.json
|
||||
sha256_manifest.json
|
||||
tests/
|
||||
contract.test.mjs
|
||||
```
|
||||
|
||||
Do not add any other repository directory. In the fresh artifact root, run:
|
||||
|
||||
```bash
|
||||
npm test
|
||||
HOST=0.0.0.0 npm start
|
||||
```
|
||||
|
||||
No dependency install or build step is required. The `PORT` environment
|
||||
variable is honored automatically by `server.mjs` when a host supplies it.
|
||||
|
||||
Before creating the deployment repository, inspect its entire file tree. Apart
|
||||
from `package.json` project metadata, the only JSON allowed is the reviewed
|
||||
aggregate bundle under `public/data/`. Confirm that `data_manifest.json` still
|
||||
declares:
|
||||
|
||||
```text
|
||||
development_preview: true
|
||||
population_estimate_allowed: false
|
||||
locked_test_metrics_published: false
|
||||
```
|
||||
|
||||
Do not hand-edit an aggregate file to make validation pass. Regenerate it
|
||||
through the private local exporter and repeat the review.
|
||||
|
||||
## 3. Import only the dashboard repository into Bolt
|
||||
|
||||
The recommended handoff is a fresh GitHub repository containing the reviewed
|
||||
tree above. It may be private if the connected Bolt account has access.
|
||||
|
||||
1. On the Bolt homepage, choose the GitHub import control.
|
||||
2. Select the separate dashboard-only repository or use **Import from URL**.
|
||||
3. Confirm `index.html` appears at the imported project root.
|
||||
4. Inspect the Bolt file tree before previewing. If SQL, scripts, artifacts,
|
||||
private data, `.env`, or source-repository files appear, stop and delete that
|
||||
Bolt project; create a clean dashboard-only project instead.
|
||||
5. Use `HOST=0.0.0.0 npm start` if Bolt requests a preview command. There is no
|
||||
build command.
|
||||
|
||||
Bolt's current GitHub import workflow is documented in
|
||||
[GitHub for version control](https://support.bolt.new/integrations/git).
|
||||
|
||||
If a separate GitHub repository is not used, create a fresh Bolt/StackBlitz
|
||||
project and upload only the reviewed dashboard files so the same tree appears at
|
||||
the project root. Never upload the full source repository and then try to hide
|
||||
or ignore its private-pipeline directories.
|
||||
|
||||
## 4. Review in Bolt before publishing
|
||||
|
||||
Use the preview URL and repeat the local visual checks. Also verify directly:
|
||||
|
||||
- `/public/data/data_manifest.json` resolves;
|
||||
- an unknown file path returns `404`;
|
||||
- `/.env`, `/data/`, `/artifacts/`, `/models/`, `/sql/`, and `/scripts/` return
|
||||
`404`;
|
||||
- the browser makes requests only for the static HTML, CSS, JavaScript, and
|
||||
approved files under `/public/data/`; and
|
||||
- no analytics, database, authentication, or third-party data service was added
|
||||
by an automated Bolt edit.
|
||||
|
||||
Preview privately first. If Bolt proposes code changes, review the full diff and
|
||||
rerun `npm test`; do not accept changes that add a database, environment secret,
|
||||
tracking script, model API, prediction form, or remote data source.
|
||||
|
||||
## 5. Publish with Bolt Hosting
|
||||
|
||||
Bolt Hosting is the intended host for this prototype:
|
||||
|
||||
1. Open the dashboard-only project.
|
||||
2. Select **Publish** in the upper-right corner.
|
||||
3. Choose private visibility for presentation review when available.
|
||||
4. Select **Publish** and wait for the generated `bolt.host` URL.
|
||||
5. Run the post-publication checks below before changing visibility to public.
|
||||
|
||||
Bolt documents this flow in
|
||||
[Publish your project to a live website](https://support.bolt.new/cloud/hosting/publish).
|
||||
Bolt's Netlify integration is optional and is not required for this dashboard;
|
||||
see [Netlify integration](https://support.bolt.new/integrations/netlify) only if
|
||||
hosting requirements explicitly change.
|
||||
|
||||
## 6. Post-publication checks
|
||||
|
||||
At the exact published URL:
|
||||
|
||||
1. Hard-refresh and confirm the development-preview banner remains visible.
|
||||
2. Visit all four routes and verify labels, charts, and keyboard navigation.
|
||||
3. Confirm the Model & benchmark view identifies logistic as final, tree as
|
||||
benchmark, and its values as 2024 calibration-fit checks.
|
||||
4. Confirm no 2025 holdout metric appears in the site or public JSON.
|
||||
5. Confirm there is no vehicle-level lookup, input form, or row-level output.
|
||||
6. Confirm coverage language does not imply statewide representation or county
|
||||
ranking.
|
||||
7. Request the denied paths from step 4 again and require `404`.
|
||||
8. Save the published URL, release ID, publication date, and reviewer decision;
|
||||
do not save private paths, row values, or credentials in the release note.
|
||||
|
||||
If any check fails, use Bolt's Publish menu to unpublish or keep the site private
|
||||
until a corrected dashboard-only artifact passes local and hosted review.
|
||||
|
||||
## Updating a published prototype
|
||||
|
||||
Changes in a Bolt project are not automatically live. For each update:
|
||||
|
||||
1. regenerate/review aggregates locally;
|
||||
2. run the dashboard contract test;
|
||||
3. update the separate dashboard-only repository/project;
|
||||
4. inspect Bolt's diff and preview;
|
||||
5. use **Publish > Update**; and
|
||||
6. repeat every post-publication check.
|
||||
|
||||
Never synchronize private pipeline directories into the deployment repository.
|
||||
@ -1,97 +1,126 @@
|
||||
# Bolt dashboard specification
|
||||
# Static dashboard specification
|
||||
|
||||
## Product language
|
||||
## Release status and language
|
||||
|
||||
Use **Utah Vehicle Health** as the brand, but call the modeled quantity
|
||||
**next-episode non-pass risk**. The estimator is a cohort estimate, not a
|
||||
diagnosis, certification, or guarantee.
|
||||
The checked-in site is a **development preview** built from suppressed
|
||||
aggregates derived from the private, page-sampled 10,000-vehicle development
|
||||
cohort. Every value is a sample result, **not a population estimate**. The site
|
||||
must not present statewide prevalence, county rankings, causal comparisons, or
|
||||
individual predictions.
|
||||
|
||||
## Four-page MVP
|
||||
Use **Utah Vehicle Health** as the project name and **next-episode first-attempt
|
||||
non-pass** for the modeled outcome. Non-pass combines fail, reject, and abort.
|
||||
The dashboard is not a diagnosis, certification, safety assessment,
|
||||
roadworthiness assessment, or guarantee.
|
||||
|
||||
The sections below describe the full product target. The checked-in
|
||||
development preview intentionally implements a narrower safe subset: binary
|
||||
pass/non-pass aggregates, supported make/model scorecards, age bands, coverage
|
||||
quality, and pre-2025 model diagnostics. Four-class charts, uncertainty
|
||||
intervals, adjusted scorecards, and the prediction lookup remain disabled until
|
||||
their own reviewed aggregate assets exist.
|
||||
## Implemented four-view prototype
|
||||
|
||||
### Overview
|
||||
|
||||
- Eligible inspections, pass rate, non-pass rate, and covered-period KPIs
|
||||
- Quarterly pass/fail/reject/abort trend with blanks shown separately
|
||||
- Covered-county map; unavailable counties remain gray
|
||||
- Non-pass risk versus vehicle age with intervals
|
||||
- Clear notices for partial periods and limited feed coverage
|
||||
- Visibly persistent development-preview warning
|
||||
- Rounded published support and sample pass/non-pass rates
|
||||
- Sample first-attempt non-pass trend for approved, pre-2025 periods
|
||||
- County **feed-coverage** map, with unavailable counties shown as unavailable
|
||||
rather than zero
|
||||
- Sample non-pass pattern by coarsened vehicle-age band
|
||||
- Plain-language target and limitations callout
|
||||
|
||||
### Reliability explorer
|
||||
County values provide sample/feed context only. They must not be sorted,
|
||||
headlined, or described as population county performance.
|
||||
|
||||
- Search supported canonical make/model cohorts
|
||||
- Compare up to three cohorts across vehicle-age bands
|
||||
- Observed versus model-adjusted risk toggle
|
||||
- Outcome-mix bar and uncertainty-aware ranked dot plot
|
||||
- County, make/model, age band, fuel, program, and period filters
|
||||
- Support size and interval displayed for every estimate
|
||||
### Sample cohorts
|
||||
|
||||
### Next-test risk estimator
|
||||
- Search and compare only supported, suppression-cleared make/model cohorts
|
||||
- Show observed sample non-pass rates and rounded support
|
||||
- Default sorting by support rather than risk
|
||||
- Disable filters or adjusted views that the aggregate bundle cannot support
|
||||
- Explain that omitted cohorts may be suppressed or unavailable
|
||||
|
||||
Inputs are coarsened, non-identifying attributes: county, supported make/model,
|
||||
vehicle-age band, fuel, prior episode outcome, time-since-prior band, season,
|
||||
and approved program category.
|
||||
These are descriptive development-sample cohorts, not reliability ratings,
|
||||
causal make/model effects, or recommendations.
|
||||
|
||||
Output a calibrated non-pass probability, uncertainty interval, relevant
|
||||
baseline, and aggregate factor contributions. Never request VIN, plate, exact
|
||||
address, station, free text, or current-test diagnostics.
|
||||
### Model & benchmark
|
||||
|
||||
### Data and methods
|
||||
- Identify calibrated logistic regression as the sole final prototype model.
|
||||
- Identify calibrated histogram gradient boosting as benchmark-only.
|
||||
- Show only pre-2025, 2024 calibration-fit diagnostics from the private sample.
|
||||
- Explain that those same-partition values are calibration checks, not
|
||||
independent final-performance or population estimates.
|
||||
- Keep the opened 2025 holdout comparison in the private report/presentation,
|
||||
not in the browser bundle.
|
||||
- State plainly that the finished prototype has no vehicle-level prediction
|
||||
service, personal input form, or row-level output.
|
||||
|
||||
- Coverage timeline and source/year missingness heatmap
|
||||
- Episode and target definitions
|
||||
- `slc`/`slco` source-era explanation
|
||||
- Temporal split, PR-AUC, Brier score, and calibration plot
|
||||
- Subgroup/source-era performance
|
||||
- Leakage controls, DMV gaps, partial periods, and limitations
|
||||
### Data & methods
|
||||
|
||||
- First-attempt returning-vehicle episode and binary-target definitions
|
||||
- Point-in-time feature and leakage controls
|
||||
- Coverage timeline by source era
|
||||
- Chronological development-sample split and one-time 2025-gate disclosure
|
||||
- Evidence-governance summary for the logistic final model and tree benchmark
|
||||
- Known sampling, coverage, label, validation, and interpretation limits
|
||||
- Private-to-public publication flow
|
||||
|
||||
The boosted tree appears only in benchmark context, never as a second final
|
||||
model. The dashboard never requests a VIN, plate, ZIP, exact address, station,
|
||||
technician, free text, or current-test diagnostic.
|
||||
|
||||
## Public data contract
|
||||
|
||||
| Dataset | Safe grain |
|
||||
| Asset | Allowed content |
|
||||
| --- | --- |
|
||||
| `data_manifest` | Data cutoff, deterministic release ID, model versions, definitions and exclusions |
|
||||
| `overview_period_county` | Quarter/year × public county with rounded support and outcome rates |
|
||||
| `cohort_scorecard` | Approved make/model × age band, optionally coarsened county/fuel |
|
||||
| `age_risk_curve` | Approved cohort × age point/band with risk, interval and support |
|
||||
| `prediction_lookup` | Only supported coarsened input combinations and calibrated outputs |
|
||||
| `filter_catalog` | Publishable categories and valid combinations |
|
||||
| `model_diagnostics` | Approved partition-level metrics; locked metrics require a separate release gate |
|
||||
| `coverage_quality` | Source era × year volume, blank rate, linkage and availability |
|
||||
| `data_manifest.json` | Release ID, model versions, scope flags, definitions, and suppression settings |
|
||||
| `overview_period_county.json` | Suppressed quarter x covered-feed county sample aggregates |
|
||||
| `cohort_scorecard.json` | Suppressed make/model sample aggregates with rounded support |
|
||||
| `age_risk_curve.json` | Suppressed sample rates by coarsened vehicle-age band |
|
||||
| `filter_catalog.json` | Publishable categories and supported periods |
|
||||
| `model_diagnostics.json` | Train/tune/calibration development metrics only; no 2025 metrics |
|
||||
| `coverage_quality.json` | Source-era availability by year |
|
||||
| `sha256_manifest.json` | Checksums for the approved bundle |
|
||||
|
||||
Use purpose-built outputs rather than a single high-dimensional browser cube.
|
||||
Bolt receives only these sanitized, versioned assets—never countydata
|
||||
credentials or private analytical rows.
|
||||
Every JSON asset must declare `development_preview=true` and
|
||||
`population_estimate_allowed=false`. The public manifest must declare
|
||||
`locked_test_metrics_published=false`. The browser validates the complete bundle
|
||||
and shows no estimates if any required asset fails its contract.
|
||||
|
||||
## Publication controls
|
||||
The browser bundle must never contain raw analytical rows, prediction lookup
|
||||
rows, hidden unsuppressed chart layers, or values recoverable only through the
|
||||
developer console.
|
||||
|
||||
- Suppress cells below 100 eligible inspections.
|
||||
- Also require at least 100 distinct private vehicle tokens in every published
|
||||
cell; tokens and distinct counts never enter the public asset.
|
||||
- Suppress when an outcome or its complement has fewer than 10 records.
|
||||
- Require at least 10 distinct vehicles contributing each binary class.
|
||||
- Apply complementary suppression so totals cannot reconstruct hidden cells.
|
||||
- Combine rare categories, coarsen model years, and round displayed counts.
|
||||
- Recheck thresholds after every filter combination.
|
||||
- Do not include suppressed rows in browser bundles, API responses, downloads,
|
||||
analytics logs, or hidden chart layers.
|
||||
- Downloads contain only the sanitized summary currently displayed.
|
||||
## Suppression and privacy controls
|
||||
|
||||
## Visual direction
|
||||
- Require at least 100 eligible episodes and 100 distinct private vehicles per
|
||||
published cell.
|
||||
- Require at least 10 observations and 10 distinct vehicles in both binary
|
||||
outcome classes.
|
||||
- Apply complementary suppression when a visible total could reveal a hidden
|
||||
cell.
|
||||
- Combine rare categories, coarsen age/year fields, and round support.
|
||||
- Recheck thresholds for the exact grain of every released asset.
|
||||
- Omit suppressed rows entirely; do not send them to the browser.
|
||||
- Publish no VIN, plate, ZIP, exact address, station, technician identifier,
|
||||
private token, raw JSON, credential, operational record, free text, or
|
||||
row-level prediction.
|
||||
- Make no browser or hosted-service connection to `countydata`.
|
||||
|
||||
Use a restrained Utah/desert palette: teal pass, red fail, amber reject, purple
|
||||
abort, and gray missing. Use probability bars, calibrated dot plots, confidence
|
||||
bands, and cohort comparisons instead of gauges. Do not rely on color alone.
|
||||
## Accessibility and visual behavior
|
||||
|
||||
## Stretch pages
|
||||
- Use semantic headings, landmarks, labels, status messages, and keyboard-
|
||||
reachable navigation.
|
||||
- Provide text alternatives or accessible labels for charts.
|
||||
- Do not rely on color alone.
|
||||
- Use a restrained Utah/desert palette and direct probability/rate encodings;
|
||||
avoid gauges and decorative risk scores.
|
||||
- Preserve the warning banner and fail-closed unavailable state on desktop and
|
||||
mobile layouts.
|
||||
|
||||
- Failure-to-pass journeys with funnels, attempts-to-pass and survival curves
|
||||
- Four-class outcome probabilities
|
||||
- Source-scoped OBD early-warning analysis
|
||||
- Automated aggregate refresh with a dedicated read-only role
|
||||
- Model-drift monitoring
|
||||
## Deployment boundary
|
||||
|
||||
Bolt receives a separate dashboard-only project whose root is the contents of
|
||||
`dashboard/`. Do not import or upload the full private-pipeline repository and
|
||||
do not rely on a configurable subdirectory working root. The deployable project
|
||||
contains no `.env`, private `data/`, model artifacts, SQL, pipeline scripts, or
|
||||
repository history outside the dashboard directory.
|
||||
|
||||
See [bolt_deployment.md](bolt_deployment.md) for the reviewed handoff and
|
||||
post-publication checks.
|
||||
|
||||
@ -1,8 +1,15 @@
|
||||
# Countydata inventory
|
||||
|
||||
Inventory date: 2026-07-15; live schema revalidated 2026-07-21. All inspection
|
||||
was performed with read-only transactions, metadata queries, aggregate queries,
|
||||
and bounded JSON field-path sampling; no identifier values were exported.
|
||||
Inventory date: 2026-07-15; live schema revalidated 2026-07-21. All inventory
|
||||
work used read-only transactions, metadata queries, aggregate queries, and
|
||||
bounded JSON field-path sampling; no identifier values were exported.
|
||||
|
||||
> **Source-discovery context only:** Database-wide metadata and aggregate counts
|
||||
> below describe the accessible source, not Utah Vehicle Health research
|
||||
> results. The finished prototype uses a private, page-sampled 10,000-vehicle
|
||||
> development cohort; all model/dashboard findings are sample-based and not
|
||||
> population estimates. See the authoritative [project charter](project_charter.md)
|
||||
> and [final report](final_report.md).
|
||||
|
||||
## Live access and full-schema boundary
|
||||
|
||||
|
||||
113
docs/demo_script.md
Normal file
113
docs/demo_script.md
Normal file
@ -0,0 +1,113 @@
|
||||
# Utah Vehicle Health dashboard demo script
|
||||
|
||||
Target length: **2 minutes 45 seconds**, embedded in the
|
||||
[10-minute presentation](presentation_outline.md).
|
||||
|
||||
## Before the audience arrives
|
||||
|
||||
1. From the repository root, run `node dashboard/server.mjs`.
|
||||
2. Open `http://127.0.0.1:4173/#overview`.
|
||||
3. Confirm the header says the sample aggregates validated.
|
||||
4. Confirm the private-sample development-preview banner is visible.
|
||||
5. Visit all four routes: Overview, Sample cohorts, Model & benchmark, and Data
|
||||
& methods.
|
||||
6. Reset Sample cohorts and leave its sort on largest support.
|
||||
|
||||
Do not open developer tools, private files, model artifacts, database clients,
|
||||
or environment variables during the presentation.
|
||||
|
||||
## Live talk track
|
||||
|
||||
### 0:00-0:25 — Establish the boundary
|
||||
|
||||
**Action:** Start on **Overview**. Point to the development-preview banner before
|
||||
pointing to a chart.
|
||||
|
||||
**Say:**
|
||||
|
||||
> The most important element is this banner. Every value in the dashboard comes
|
||||
> from the private 10,000-vehicle development sample. These are suppression-
|
||||
> reviewed sample aggregates, not population estimates, and they cannot support
|
||||
> individual decisions or county rankings.
|
||||
|
||||
### 0:25-1:00 — Explain the overview
|
||||
|
||||
**Action:** Point to published support and the observed first-attempt non-pass
|
||||
trend, then the coverage map and vehicle-age chart. Do not rank counties by
|
||||
outcome.
|
||||
|
||||
**Say:**
|
||||
|
||||
> The overview reports rounded support and sample first-attempt outcomes for
|
||||
> returning vehicles. Non-pass combines fail, reject, and abort. The trend is
|
||||
> descriptive of sampled records only. This county graphic communicates feed
|
||||
> availability: gray means unavailable, not zero and not better. The age pattern
|
||||
> is also an observed sample association, not a causal claim or diagnosis.
|
||||
|
||||
### 1:00-1:35 — Use Sample cohorts safely
|
||||
|
||||
**Action:** Open **Sample cohorts**. Search for `Toyota` or another currently
|
||||
supported make. Leave sorting on largest support. Point to rounded support and
|
||||
the “observed sample non-pass” label.
|
||||
|
||||
**Say:**
|
||||
|
||||
> This view contains only prior make-and-model cohorts that cleared the
|
||||
> publication thresholds. I can search supported cohorts, but these are observed
|
||||
> development-sample associations—not reliability grades, rankings, or
|
||||
> recommendations. Unsupported slices are not inferred in the browser, and
|
||||
> suppressed rows are absent rather than hidden.
|
||||
|
||||
**Action:** Reset the search before leaving the view.
|
||||
|
||||
### 1:35-2:15 — Separate the final model from its benchmark
|
||||
|
||||
**Action:** Open **Model & benchmark**. Point first to the role badges, then the
|
||||
2024 calibration-fit scope strip, then the “No vehicle-level prediction service”
|
||||
boundary.
|
||||
|
||||
**Say:**
|
||||
|
||||
> The governance decision is explicit: calibrated logistic regression is the
|
||||
> final prototype model, while the boosted tree is benchmark-only. The numbers
|
||||
> shown here are pre-2025 checks on the same 2024 sample partition used to fit
|
||||
> calibration. They are not independent final-performance or population
|
||||
> estimates, so we do not select between the cards from these values. The
|
||||
> one-time 2025 comparison shown earlier favored logistic and stays in the report
|
||||
> and presentation, not the browser bundle. This finished dashboard contains no
|
||||
> vehicle lookup, personal inputs, or row-level prediction output.
|
||||
|
||||
### 2:15-2:40 — Close on method and privacy
|
||||
|
||||
**Action:** Open **Data & methods**. Briefly point to Prediction unit, the
|
||||
chronological timeline, evidence status, and the privacy flow.
|
||||
|
||||
**Say:**
|
||||
|
||||
> The method page makes the contract visible: the target is the first attempt of
|
||||
> a returning vehicle's next episode, and every feature ends before that episode.
|
||||
> Training, tuning, and calibration are chronological. It also records that 2025
|
||||
> was opened once after choices were frozen. Finally, the public path ends in
|
||||
> suppressed summaries. No private rows or database connection reach this site.
|
||||
|
||||
### 2:40-2:45 — Transition
|
||||
|
||||
**Say:**
|
||||
|
||||
> That is the prototype: useful sample evidence, with model and privacy
|
||||
> boundaries kept visible.
|
||||
|
||||
Return to the slide deck's privacy architecture.
|
||||
|
||||
## If something goes wrong
|
||||
|
||||
If the dashboard shows **data unavailable**, do not bypass validation or edit
|
||||
JSON. Say:
|
||||
|
||||
> The site rejected an aggregate-contract mismatch and is failing closed, so it
|
||||
> shows no estimates. That behavior is part of the privacy and integrity design.
|
||||
|
||||
Then continue with a backup screenshot or slides. If a cohort search returns no
|
||||
rows, explain that unsupported or suppressed cohorts are intentionally absent
|
||||
and reset the filter. Never substitute remembered values or improvise a vehicle-
|
||||
level example.
|
||||
@ -1,67 +1,66 @@
|
||||
# Development run results
|
||||
|
||||
Run date: 2026-07-15
|
||||
Run date: 2026-07-15; presentation wording reviewed 2026-07-21
|
||||
|
||||
These results validate the engineering and modeling pipeline. They are **not
|
||||
population estimates**: the development extract samples inspection-table pages
|
||||
and then retrieves complete histories for up to 10,000 vehicles, which
|
||||
over-represents vehicles with more inspection records.
|
||||
> **Scope of every value below:** the private, page-sampled 10,000-vehicle
|
||||
> development cohort. These are sample results, not population estimates. Page
|
||||
> sampling over-represents vehicles with more inspection records. Nothing in
|
||||
> this document supports statewide/county prevalence or county rankings.
|
||||
|
||||
## Label audit
|
||||
|
||||
The raw Utah County feed leaves `overall_result` blank for 3,747,862 records.
|
||||
An aggregate-only server audit found 3,469,102 Utah `obd`/`OBD` records with a
|
||||
controlled pass, fail, reject, or abort value in `obd_result`. Across non-Utah
|
||||
feeds where both fields were recognized, the two fields agreed on binary pass
|
||||
versus non-pass 99.30% of the time, but their fail-versus-reject categories were
|
||||
not interchangeable.
|
||||
The binary target is pass versus non-pass, where non-pass combines recognized
|
||||
fail, reject, and abort outcomes. Blanks and unrecognized values are unlabeled,
|
||||
not passes.
|
||||
|
||||
Label contract v3 therefore permits the Utah OBD value only as a provenance-
|
||||
tagged binary proxy. TSI, `other/C`, `B`, blank, and unknown values stay
|
||||
unlabeled. Four-class analysis must use `overall_result` exclusively. This also
|
||||
matches the official program distinction between a readiness rejection and a
|
||||
failed inspection; see the [Utah inspection requirements](https://dmv.utah.gov/register/inspections/)
|
||||
and [program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
|
||||
|
||||
In the development extract, the rule restored 11,334 labels and left 1,281
|
||||
events unlabeled. The label-source field remains private audit metadata and is
|
||||
not a model feature.
|
||||
For the older Utah County feed, the documented source/program/test rule permits
|
||||
a controlled OBD result to fill a blank overall result as a binary proxy. It
|
||||
does not support a four-class interpretation. In the private 10,000-vehicle
|
||||
sample, the rule restored 11,334 labels and left 1,281 events unlabeled. These
|
||||
sample counts are not population estimates. Label provenance stays in private
|
||||
audit metadata and is not a predictor.
|
||||
|
||||
## Pipeline reconciliation
|
||||
|
||||
| Stage | Rows |
|
||||
This table reports private 10,000-vehicle development-sample pipeline counts,
|
||||
not population totals.
|
||||
|
||||
| Sample pipeline stage | Rows |
|
||||
| --- | ---: |
|
||||
| Source events written after VIN validation | 83,552 |
|
||||
| Source events written after identifier validation | 83,552 |
|
||||
| Clean events after duplicate/conflict handling | 80,190 |
|
||||
| Inspection episodes | 69,588 |
|
||||
| Eligible returning targets | 44,659 |
|
||||
| Eligible returning-vehicle targets | 44,659 |
|
||||
|
||||
The sample contains 9,996 retained vehicle tokens. The extraction manifest,
|
||||
compressed-file SHA-256, mart manifest, Parquet SHA-256, and row counts all
|
||||
reconcile. Every mart invariant reports zero violations. Raw VIN and raw OBD
|
||||
result are absent from the staging output schema, feature mart, and model
|
||||
features.
|
||||
The sample retained 9,996 private vehicle tokens. Extraction and mart manifests,
|
||||
compressed-file and Parquet checksums, and stage counts reconcile. Every mart
|
||||
invariant reports zero violations. Raw identifiers and raw OBD values are absent
|
||||
from staging output, feature mart predictors, and model features.
|
||||
|
||||
| Temporal partition | Eligible targets | Never-fit audit targets |
|
||||
The following partition counts are also from the private 10,000-vehicle
|
||||
development sample and are not population estimates.
|
||||
|
||||
| Sample temporal partition | Eligible targets | Never-fit audit targets |
|
||||
| --- | ---: | ---: |
|
||||
| Train, 2016–2022 | 31,745 | 3,129 |
|
||||
| Train, 2016-2022 | 31,745 | 3,129 |
|
||||
| Tune, 2023 | 4,972 | 526 |
|
||||
| Calibrate, 2024 | 2,633 | 277 |
|
||||
| Locked test, 2025 | 4,451 | 480 |
|
||||
| One-time holdout, 2025 | 4,451 | 480 |
|
||||
| Shadow, 2026 partial | 858 | 106 |
|
||||
|
||||
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. The
|
||||
normal trainers do not calculate 2025 outcomes or metrics without the explicit
|
||||
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. Normal
|
||||
training does not calculate 2025 outcomes or metrics without the explicit
|
||||
locked-evaluation flag.
|
||||
|
||||
## Baseline results before the locked test
|
||||
## Pre-2025 development diagnostics
|
||||
|
||||
The table below uses only non-audit vehicles. Average precision is the project's
|
||||
PR-AUC summary. The calibrated logistic row is shown only on the partition used
|
||||
to fit the calibrator and is therefore a calibration diagnostic, not an
|
||||
independent final estimate.
|
||||
The table uses non-audit rows from the private 10,000-vehicle development
|
||||
sample. Average precision is the project's PR-AUC summary. The calibrated 2024
|
||||
row is evaluated on the same sample partition used to fit Platt scaling, so it
|
||||
is a calibration diagnostic rather than independent final performance. None of
|
||||
these values is a population estimate.
|
||||
|
||||
| Partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
||||
| Sample partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
|
||||
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
|
||||
@ -71,66 +70,62 @@ independent final estimate.
|
||||
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
|
||||
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |
|
||||
|
||||
The selected logistic regularization was `C=0.03`. It converged in 2,390 of the
|
||||
5,000 allowed iterations; Platt calibration converged in five iterations. All
|
||||
five candidates and all stored probabilities passed explicit convergence and
|
||||
finite-value checks.
|
||||
The selected logistic regularization is `C=0.03`. It converged in 2,390 of the
|
||||
5,000 allowed iterations; Platt calibration converged in five iterations. These
|
||||
are development-run diagnostics. All candidates and stored probabilities passed
|
||||
explicit convergence and finite-value checks.
|
||||
|
||||
The literal previous-outcome baseline is useful as a ranking sanity check but
|
||||
produces overconfident zero/one probabilities, explaining its poor Brier and log
|
||||
loss. The transparent logistic model is the current development leader: on the
|
||||
2023 tuning partition it more than doubles prevalence PR-AUC and raises top-10%
|
||||
precision from 12.5% to 31.5%.
|
||||
On the 2023 sample partition, logistic regression more than doubled prevalence
|
||||
PR-AUC and raised top-10% precision from 12.5% to 31.5%. This describes the
|
||||
private development sample only.
|
||||
|
||||
## One-time development holdout
|
||||
## One-time development-sample holdout
|
||||
|
||||
The nonlinear live-verification command invoked the explicit 2025 gate after
|
||||
its feature contract and four-candidate grid had already been fixed and its
|
||||
candidate had been selected only on 2023. No test-driven model change was made.
|
||||
At that point the development specification was frozen and the already-fixed
|
||||
logistic model was evaluated once for a direct comparison. These are
|
||||
development-sample holdout results, not final population claims.
|
||||
The nonlinear verification command invoked the explicit 2025 gate only after
|
||||
the feature contract and benchmark grid had been frozen and candidate choices
|
||||
had been made from pre-2025 sample data. No model was changed in response. These
|
||||
are one-time development-sample holdout results, not pristine future-test or
|
||||
population estimates.
|
||||
|
||||
| 2025 non-audit model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Training prevalence | 0.123 | 0.1076 | 0.500 | 0.123 |
|
||||
| Previous episode, literal | 0.158 | 0.1731 | 0.576 | 0.268 |
|
||||
| Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
|
||||
| Histogram gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
|
||||
| 2025 private-sample model | Role | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| Training prevalence | Baseline | 0.123 | 0.1076 | 0.500 | 0.123 |
|
||||
| Previous episode, literal | Baseline | 0.158 | 0.1731 | 0.576 | 0.268 |
|
||||
| Logistic + Platt | **Final model** | **0.261** | **0.1011** | **0.693** | **0.312** |
|
||||
| Histogram gradient boosting + Platt | Benchmark only | 0.238 | 0.1024 | 0.690 | 0.292 |
|
||||
|
||||
The holdout contains 3,971 non-audit episodes from 3,734 vehicles. The
|
||||
never-fit audit contains another 480 episodes from 450 vehicles; calibrated
|
||||
logistic PR-AUC is 0.253 and Brier is 0.0859 there. The logistic model beats
|
||||
both simple baselines on the project's headline metrics and remains the chosen
|
||||
development model. The tree did not provide a decisive pre-test improvement
|
||||
that justified its added complexity, and no further 2025-informed tuning is
|
||||
permitted.
|
||||
The sample holdout contains 3,971 non-audit episodes from 3,734 vehicles. Its
|
||||
never-fit audit adds 480 episodes from 450 vehicles; calibrated logistic PR-AUC
|
||||
is 0.253 and Brier score is 0.0859 there. All are private-sample diagnostics,
|
||||
not population estimates or external validation.
|
||||
|
||||
## Remaining gates
|
||||
|
||||
1. Run a complete, contiguous bounded extraction for publishable population
|
||||
aggregates; the page-sampled development cohort cannot support dashboard
|
||||
prevalence or county rankings.
|
||||
2. Treat any later full-data 2025 result as confirmatory rather than a pristine
|
||||
unseen test, because the development sample's holdout has now been opened.
|
||||
3. Run the frozen subgroup/source-era report and clustered uncertainty
|
||||
analysis on the complete extraction.
|
||||
4. Replace the checked-in development-preview bundle with complete-data
|
||||
aggregates only after the population-publication review passes.
|
||||
The calibrated logistic regression remains the final development model. It
|
||||
beats both simple baselines on the headline metrics and beats the boosted-tree
|
||||
benchmark in the one-time comparison. The benchmark did not justify its added
|
||||
complexity. No further 2025-informed tuning is permitted.
|
||||
|
||||
## Sanitized dashboard preview
|
||||
|
||||
The repository now includes a fail-closed exporter and a static four-view
|
||||
dashboard shell. The preview exporter reads only train, tune, and calibration
|
||||
rows dated before 2025. It rejects any model manifest that says the holdout was
|
||||
evaluated, rejects any metric row outside the three approved partitions, and
|
||||
marks every asset `development_preview=true` and
|
||||
`population_estimate_allowed=false`.
|
||||
The static preview uses only suppression-cleared sample aggregates and
|
||||
development diagnostics dated before 2025. It intentionally excludes the
|
||||
one-time holdout metrics. Every asset declares `development_preview=true` and
|
||||
`population_estimate_allowed=false`; the manifest declares
|
||||
`locked_test_metrics_published=false`.
|
||||
|
||||
Published cells must clear minimum episode and distinct-vehicle thresholds,
|
||||
including both binary-class complements. Supports are rounded, direct and
|
||||
pseudonymous identifiers stay private, and a checksum manifest covers the
|
||||
approved JSON bundle. Private model manifests bind the exact metrics files, and
|
||||
the public manifest exposes a deterministic release ID plus model versions for
|
||||
provenance. The site disables the estimator because no privacy-reviewed
|
||||
prediction lookup exists.
|
||||
Published cells must clear minimum episode, distinct-vehicle, and binary-class
|
||||
thresholds. Supports are rounded, suppressed rows are omitted, and a checksum
|
||||
manifest covers the approved JSON bundle. The site fails closed if a contract
|
||||
check fails and contains no vehicle-level prediction service.
|
||||
|
||||
No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
|
||||
credential, operational record, or row-level prediction is in the public
|
||||
bundle.
|
||||
|
||||
## Boundary of the finished prototype
|
||||
|
||||
The repository completes a presentation-ready development prototype, not a
|
||||
population study or production service. A complete population-frame extraction,
|
||||
fresh external/future validation, individualized prediction lookup, DMV
|
||||
enrichment, multiclass modeling, and operational monitoring are outside this
|
||||
prototype's deliverables. They would require separate approval, analysis, and
|
||||
privacy review; they are not implied by the current results.
|
||||
|
||||
223
docs/final_report.md
Normal file
223
docs/final_report.md
Normal file
@ -0,0 +1,223 @@
|
||||
# Utah Vehicle Health: final development-prototype report
|
||||
|
||||
Prepared July 21, 2026
|
||||
|
||||
> **Result scope:** Every analytical count, rate, chart, and model metric in this
|
||||
> report comes from the private, page-sampled 10,000-vehicle development cohort.
|
||||
> These are sample results, not population estimates. The cohort cannot support
|
||||
> statewide or county prevalence, county rankings, causal conclusions, or
|
||||
> decisions about an individual vehicle.
|
||||
|
||||
## Executive summary
|
||||
|
||||
Utah Vehicle Health asks one focused question:
|
||||
|
||||
> Using only information available before an inspection episode begins, how
|
||||
> well can a calibrated logistic regression estimate whether a returning
|
||||
> vehicle's next episode will have a first-attempt non-pass outcome?
|
||||
|
||||
The finished prototype builds leakage-safe inspection episodes, creates
|
||||
point-in-time history features, evaluates models chronologically, and publishes
|
||||
only suppression-reviewed sample aggregates to a static dashboard. Non-pass
|
||||
combines recognized fail, reject, and abort outcomes; blank or unrecognized
|
||||
results are not treated as passes.
|
||||
|
||||
The final model is regularized logistic regression with Platt calibration. On
|
||||
the private sample's one-time 2025 holdout, it achieved PR-AUC 0.261 and Brier
|
||||
score 0.1011. The calibrated histogram gradient-boosted benchmark achieved
|
||||
PR-AUC 0.238 and Brier score 0.1024 on the same sample holdout. Logistic
|
||||
regression remains the final model because it was both more transparent and
|
||||
better on the declared headline metrics in this development comparison.
|
||||
|
||||
Those values demonstrate that the pipeline and model comparison work on this
|
||||
development sample. They do not establish population performance, production
|
||||
readiness, or external validity.
|
||||
|
||||
## Research scope
|
||||
|
||||
The prediction unit is the first attempt of the next inspection episode for a
|
||||
returning vehicle. Consecutive attempts no more than 30 days apart form one
|
||||
episode. This prevents a fail followed quickly by a retest from becoming two
|
||||
nominally independent prediction targets. A vehicle is eligible only after a
|
||||
prior episode has been observed.
|
||||
|
||||
The binary outcome is:
|
||||
|
||||
- pass; or
|
||||
- non-pass, combining fail, reject, and abort.
|
||||
|
||||
Reject and abort can reflect process or readiness conditions, so this report
|
||||
does not equate non-pass with mechanical failure. Cold-start prediction,
|
||||
four-class modeling, fail-only sensitivity analysis, DMV enrichment,
|
||||
station-level analysis, and individual prediction are outside the completed
|
||||
prototype.
|
||||
|
||||
## Development data
|
||||
|
||||
The private extraction samples inspection-table pages, selects up to 10,000
|
||||
vehicles, and then retrieves their complete available inspection histories. It
|
||||
retained 9,996 private vehicle tokens after validation. The resulting sample
|
||||
contains 83,552 source events, 69,588 episodes, and 44,659 eligible returning-
|
||||
vehicle targets. These figures are pipeline-reconciliation counts for the
|
||||
private sample, not estimates of Utah inspection volume.
|
||||
|
||||
The sample design intentionally makes end-to-end development manageable, but it
|
||||
over-represents vehicles with more inspection records. Participating feeds also
|
||||
change over time and do not cover all 29 Utah counties. The dashboard therefore
|
||||
labels every value as a development-sample aggregate and uses county only to
|
||||
communicate covered-feed context—not to publish population rankings.
|
||||
|
||||
The extraction and mart manifests bind hashes, configurations, and row counts.
|
||||
Private vehicle linkage uses a one-way keyed token. Direct identifiers and raw
|
||||
source payloads remain outside the analytical exports and public site.
|
||||
|
||||
## Leakage-safe method
|
||||
|
||||
Every feature exists before the target episode begins. The final model uses
|
||||
vehicle age, prior episode and attempt counts, time since prior history, prior
|
||||
non-pass rate, prior first/final outcomes, previously observed make/model,
|
||||
public county context, and season.
|
||||
|
||||
The model excludes the target attempt's outcome and diagnostics, later attempts
|
||||
from the target episode, station or technician information, future records,
|
||||
full-history leakage, direct or pseudonymous identifiers, and preprocessing
|
||||
learned from later partitions. DMV enrichment and rich current-test OBD fields
|
||||
are not part of the final model.
|
||||
|
||||
The chronology is fixed:
|
||||
|
||||
| Period | Role in the private 10,000-vehicle development sample |
|
||||
| --- | --- |
|
||||
| 2010-2015 | Historical context only |
|
||||
| 2016-2022 | Fit preprocessing and model parameters |
|
||||
| 2023 | Select logistic regularization and compare fixed candidates |
|
||||
| 2024 | Fit Platt calibration and inspect calibration behavior |
|
||||
| 2025 | One-time development-sample holdout |
|
||||
| 2026 partial | Shadow context only |
|
||||
|
||||
Random row splitting is prohibited. A separate set of private vehicle buckets
|
||||
is never used for fitting and serves as a stress test.
|
||||
|
||||
## Model comparison and decision
|
||||
|
||||
The declared comparison includes training prevalence, the literal previous-
|
||||
episode outcome, regularized logistic regression, and histogram gradient
|
||||
boosting. Logistic regression uses `C=0.03`; Platt scaling is fit on the 2024
|
||||
sample partition. The tree exists only to test whether nonlinearity provides a
|
||||
meaningful development gain.
|
||||
|
||||
The table reports non-audit rows from the private, page-sampled 10,000-vehicle
|
||||
development cohort. It is not population performance.
|
||||
|
||||
| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| 2023 tune | Prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
|
||||
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
|
||||
| 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
|
||||
| 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
|
||||
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
|
||||
|
||||
On the 2023 sample partition, logistic regression more than doubled the
|
||||
prevalence baseline's PR-AUC and raised precision among the highest-scored 10%
|
||||
from 12.5% to 31.5%. On the one-time 2025 sample holdout, calibrated logistic
|
||||
also exceeded the simple baselines and the boosted-tree benchmark on PR-AUC and
|
||||
Brier score.
|
||||
|
||||
The explicit 2025 gate was opened once after model specifications were frozen
|
||||
from pre-2025 data. No model was changed in response. That evaluation is a
|
||||
development holdout, not a pristine future test, and no further 2025-informed
|
||||
tuning is allowed.
|
||||
|
||||
The final decision is therefore straightforward: keep calibrated logistic
|
||||
regression. It satisfies the development objective with a simpler, more
|
||||
explainable form, while the nonlinear benchmark did not show a gain that
|
||||
justified extra complexity.
|
||||
|
||||
## Static dashboard
|
||||
|
||||
The presentation-ready dashboard is a dependency-free static site with four
|
||||
views:
|
||||
|
||||
1. **Overview** shows rounded support, sample outcome patterns, vehicle-age
|
||||
patterns, and county feed coverage.
|
||||
2. **Sample cohorts** shows only supported make/model sample aggregates; it does
|
||||
not issue reliability ratings or recommendations.
|
||||
3. **Model & benchmark** identifies calibrated logistic regression as final and
|
||||
the boosted tree as benchmark-only. It shows pre-2025, 2024 calibration-fit
|
||||
checks—not the opened 2025 holdout comparison.
|
||||
4. **Data & methods** explains the target, chronology, evidence status,
|
||||
limitations, and privacy boundary.
|
||||
|
||||
The finished dashboard contains no vehicle lookup, personal input form, or
|
||||
row-level prediction service.
|
||||
|
||||
The dashboard loads purpose-built JSON rather than private analytical rows. It
|
||||
checks schemas, cross-file consistency, publication flags, and checksums. If the
|
||||
bundle fails validation, the interface shows an unavailable state instead of
|
||||
fallback estimates.
|
||||
|
||||
Every published cell requires minimum support for episodes, distinct vehicles,
|
||||
and both binary outcomes. Supports are rounded and suppressed rows are omitted
|
||||
entirely. The checked-in assets use only pre-2025 sample aggregates and
|
||||
diagnostics; the one-time 2025 values in this report are not in the browser
|
||||
bundle.
|
||||
|
||||
## Privacy and release boundary
|
||||
|
||||
The public boundary is intentionally narrow:
|
||||
|
||||
```text
|
||||
private read-only source
|
||||
-> private local extraction and model pipeline
|
||||
-> suppression-reviewed aggregate JSON
|
||||
-> dashboard-only static Bolt project
|
||||
```
|
||||
|
||||
The public dashboard and presentation materials contain no VIN, plate, ZIP,
|
||||
station, technician identifier, private vehicle token, raw JSON, credential,
|
||||
operational record, or row-level prediction. The browser never connects to the
|
||||
source database.
|
||||
|
||||
Bolt publication must use a separate project or repository containing only the
|
||||
contents of `dashboard/`, with `index.html` at its root. The full source
|
||||
repository must not be imported into Bolt. The reviewed procedure is in
|
||||
[bolt_deployment.md](bolt_deployment.md).
|
||||
|
||||
## Limitations
|
||||
|
||||
- The private page sample is not population-representative.
|
||||
- Results cover returning vehicles in participating feeds, not all Utah
|
||||
vehicles or counties.
|
||||
- Geography, source era, program mix, and time are entangled.
|
||||
- Reject and abort are heterogeneous non-pass outcomes.
|
||||
- The 2025 comparison is a one-time development holdout, not external
|
||||
validation.
|
||||
- Sample calibration and ranking do not establish operational usefulness.
|
||||
- Observed cohort differences are descriptive associations, not causal effects
|
||||
or recommendations.
|
||||
- The absence of protected attributes does not establish fairness.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The prototype answers its development question: a leakage-safe, calibrated
|
||||
logistic regression produces materially better discrimination and probability-error metrics
|
||||
than simple baselines in the private 10,000-vehicle sample, and it performs
|
||||
better than the boosted-tree benchmark in the one-time holdout comparison. The
|
||||
more important product result is the disciplined boundary around that finding:
|
||||
one target, chronological evaluation, transparent model selection, persistent
|
||||
sample labeling, fail-closed aggregate publication, and no row-level exposure.
|
||||
|
||||
This is a completed development prototype—not a population estimate, county
|
||||
ranking, production model, diagnosis, or individual decision tool.
|
||||
|
||||
## Supporting documentation
|
||||
|
||||
- [Project charter](project_charter.md)
|
||||
- [Modeling protocol](modeling_protocol.md)
|
||||
- [Detailed development results](development_results.md)
|
||||
- [Model card](model_card.md)
|
||||
- [Dashboard specification](dashboard_spec.md)
|
||||
- [Private-to-public architecture](architecture.mmd)
|
||||
- [10-minute presentation outline](presentation_outline.md)
|
||||
- [Dashboard demo script](demo_script.md)
|
||||
- [Dashboard-only Bolt deployment](bolt_deployment.md)
|
||||
@ -1,197 +1,161 @@
|
||||
# Utah Vehicle Health model card
|
||||
|
||||
Last updated: 2026-07-15
|
||||
Last updated: 2026-07-21
|
||||
|
||||
Status: **development prototype; not approved for production or population claims**
|
||||
Status: **final development-prototype model; not approved for production or
|
||||
population claims**
|
||||
|
||||
> Every count, rate, and performance value in this model card comes from the
|
||||
> private, page-sampled 10,000-vehicle development cohort. These are sample
|
||||
> results, not population estimates.
|
||||
|
||||
## Model summary
|
||||
|
||||
Utah Vehicle Health estimates the probability that the first attempt of a
|
||||
returning vehicle's next emissions-inspection episode will be a non-pass. The
|
||||
current selected model is a regularized logistic regression followed by Platt
|
||||
probability calibration. It is an inspection-history model, not a mechanical
|
||||
health, safety, roadworthiness, or legal-compliance model.
|
||||
Utah Vehicle Health estimates whether the first attempt of a returning
|
||||
vehicle's next emissions-inspection episode will be a non-pass. The final model
|
||||
is regularized logistic regression (`C=0.03`) followed by Platt probability
|
||||
calibration. Histogram gradient boosting is retained only as a benchmark.
|
||||
|
||||
This model card summarizes the current page-sampled development run. The full
|
||||
research and product contracts are in the [project charter](project_charter.md),
|
||||
[modeling protocol](modeling_protocol.md), and
|
||||
[dashboard specification](dashboard_spec.md).
|
||||
This is an inspection-history model. It is not a model of overall mechanical
|
||||
health, safety, roadworthiness, or legal compliance.
|
||||
|
||||
## Intended use and prohibited use
|
||||
## Intended use
|
||||
|
||||
Intended uses are to validate the data-engineering and modeling pipeline,
|
||||
compare leakage-safe model candidates, study aggregate patterns, and support a
|
||||
clearly marked development dashboard preview.
|
||||
The development model supports:
|
||||
|
||||
Do not use the model to:
|
||||
- validating leakage-safe longitudinal feature engineering;
|
||||
- comparing a transparent final model with simple and nonlinear benchmarks;
|
||||
- explaining calibration and chronological evaluation; and
|
||||
- powering aggregate, suppression-reviewed development-preview diagnostics.
|
||||
|
||||
- make decisions about an individual vehicle, owner, registration, inspection,
|
||||
or station;
|
||||
- diagnose a vehicle, guarantee an inspection result, or infer safety or
|
||||
roadworthiness;
|
||||
- rank or penalize people, counties, programs, or inspection stations;
|
||||
- make causal claims from observed associations; or
|
||||
- report statewide prevalence, county rankings, or production performance from
|
||||
the current development sample.
|
||||
Do not use it to:
|
||||
|
||||
The public estimator remains disabled because no privacy-reviewed prediction
|
||||
lookup has been approved.
|
||||
- decide anything about an individual vehicle, owner, registration, inspection,
|
||||
county, program, station, or technician;
|
||||
- diagnose a vehicle or guarantee an inspection outcome;
|
||||
- rank counties, programs, stations, vehicle cohorts, or people;
|
||||
- make causal claims from observed sample associations;
|
||||
- report statewide/county prevalence or production performance; or
|
||||
- serve a prediction lookup or row-level score.
|
||||
|
||||
## Prediction unit and target
|
||||
|
||||
The prediction is made immediately before a new inspection episode begins. An
|
||||
episode groups consecutive attempts no more than 30 days apart, and only its
|
||||
first attempt is the supervised target. A vehicle must have at least one prior
|
||||
completed episode to enter the returning-vehicle cohort.
|
||||
The prediction is made immediately before a new inspection episode begins.
|
||||
Attempts no more than 30 days apart form one episode. The supervised target is
|
||||
the episode's first attempt, and a vehicle must have at least one prior completed
|
||||
episode.
|
||||
|
||||
The binary target is:
|
||||
The binary target is recognized pass (`0`) versus recognized fail, reject, or
|
||||
abort (`1`, collectively **non-pass**). Blank, null, and unrecognized results
|
||||
remain unlabeled. Reject and abort may reflect process/readiness conditions, so
|
||||
non-pass must not be paraphrased as mechanical failure.
|
||||
|
||||
- `0`: recognized pass;
|
||||
- `1`: recognized fail, reject, or abort; and
|
||||
- unlabeled: blank, null, or unrecognized results.
|
||||
A recognized overall result is preferred. The documented older Utah County OBD
|
||||
exception may fill a blank overall result under a narrow source/program/test
|
||||
contract. That value is a binary proxy only. Its provenance remains private
|
||||
audit metadata and is not a predictor. See the
|
||||
[modeling protocol](modeling_protocol.md#binary-label-contract).
|
||||
|
||||
A recognized overall result is preferred. For the older Utah County OBD feed,
|
||||
a controlled OBD result may fill a blank overall result only under the narrow
|
||||
source/program/test contract documented in the
|
||||
[modeling protocol](modeling_protocol.md#source-specific-label-contract). That
|
||||
proxy is approved only for binary pass versus non-pass; it must not support a
|
||||
four-class interpretation. Label provenance is private audit metadata and is
|
||||
not a predictor.
|
||||
## Development data
|
||||
|
||||
## Development data and representativeness
|
||||
The extraction page-samples inspection records and retrieves complete histories
|
||||
for up to 10,000 vehicles. After validation, the private run retained 9,996
|
||||
vehicle tokens, 83,552 source events, 69,588 episodes, and 44,659 eligible
|
||||
returning targets. These sample counts describe pipeline reconciliation only;
|
||||
they do not imply population coverage.
|
||||
|
||||
The current extract samples inspection-table pages and then retrieves complete
|
||||
histories for up to 10,000 vehicles. This over-represents vehicles with more
|
||||
inspection records. The run contains 83,552 retained source events, 69,588
|
||||
episodes, and 44,659 eligible returning targets, but those counts do not make
|
||||
the sample population-representative.
|
||||
The sampling design over-represents vehicles with more inspection records. Feed
|
||||
coverage changes by source and time and does not include all 29 Utah counties.
|
||||
Accordingly, the model card makes no statewide prevalence claim, no county
|
||||
ranking, and no claim that sample subgroup differences generalize.
|
||||
|
||||
Coverage is limited to participating county/source feeds and changes over time.
|
||||
It does not represent all 29 Utah counties. Source-system transitions, missing
|
||||
years, and partial periods can resemble real changes in risk. See the
|
||||
[data inventory](data_inventory.md) and
|
||||
[development results](development_results.md) for the audited scope.
|
||||
## Features and exclusions
|
||||
|
||||
## Features and leakage exclusions
|
||||
The final model uses only pre-episode inspection-history information: vehicle
|
||||
age, prior episode/attempt counts, time since prior history, prior non-pass rate,
|
||||
prior first/final outcomes, previously observed make/model, public county
|
||||
context, and season.
|
||||
|
||||
The selected model uses information available before the target episode:
|
||||
The model excludes current-test results and diagnostics, later attempts, station
|
||||
and technician information, direct/pseudonymous identifiers, future records,
|
||||
full-history leakage, DMV enrichment, and preprocessing learned from evaluation
|
||||
periods. It does not use VIN, plate, ZIP, private token, raw JSON, credentials,
|
||||
or operational records. The complete contract is in
|
||||
[modeling_protocol.md](modeling_protocol.md#leakage-and-privacy-exclusions).
|
||||
|
||||
- vehicle age;
|
||||
- prior episode and attempt counts;
|
||||
- days since the prior episode and prior adverse outcome;
|
||||
- prior non-pass rate;
|
||||
- prior first and final outcomes;
|
||||
- previously observed make and model;
|
||||
- public county; and
|
||||
- target season.
|
||||
## Chronological evaluation
|
||||
|
||||
Categorical mappings and all preprocessing are fit on training data only. The
|
||||
current model does not yet include DMV enrichment or rich same-test OBD fields.
|
||||
|
||||
Excluded inputs include direct or pseudonymous identifiers, station
|
||||
information, the target attempt's result or diagnostics, later attempts in the
|
||||
target episode, future records, full-history aggregates, label provenance, and
|
||||
preprocessing learned from evaluation periods. The complete exclusion contract
|
||||
is in [modeling_protocol.md](modeling_protocol.md#leakage-exclusions).
|
||||
|
||||
## Chronology and holdout status
|
||||
|
||||
The fixed temporal design is:
|
||||
|
||||
| Period | Role |
|
||||
| Period | Development-sample role |
|
||||
| --- | --- |
|
||||
| 2010-2015 | Historical context only |
|
||||
| 2016-2022 | Fit preprocessing and models |
|
||||
| 2023 | Select hyperparameters and compare candidates |
|
||||
| 2024 | Fit probability calibration and inspect calibration behavior |
|
||||
| 2025 | One-time development holdout |
|
||||
| 2026 partial | Shadow monitoring only |
|
||||
| 2016-2022 | Fit preprocessing and logistic parameters |
|
||||
| 2023 | Select regularization and compare fixed candidates |
|
||||
| 2024 | Fit Platt scaling and inspect calibration behavior |
|
||||
| 2025 | One-time development-sample holdout |
|
||||
| 2026 partial | Shadow context only |
|
||||
|
||||
During live verification, the explicit 2025 gate was opened once after both
|
||||
candidate specifications and the tree search grid had already been fixed from
|
||||
pre-2025 data. No model was changed in response. The already-fixed logistic
|
||||
model was then evaluated once for a direct comparison. Consequently, this
|
||||
page-sampled 2025 cohort is no longer a pristine unseen test, and no further
|
||||
2025-informed tuning is permitted. Any complete-data 2025 analysis must be
|
||||
described as confirmatory. The detailed audit trail is in
|
||||
[development_results.md](development_results.md#one-time-development-holdout).
|
||||
The explicit 2025 gate was opened once after specifications were frozen from
|
||||
pre-2025 data. No model was changed in response. This makes 2025 a one-time
|
||||
development holdout, not a pristine future test. No further 2025-informed tuning
|
||||
is permitted.
|
||||
|
||||
## Candidate comparison and selected model
|
||||
## Development-sample performance
|
||||
|
||||
The candidates were training prevalence, the literal previous-episode outcome,
|
||||
regularized logistic regression, and histogram gradient boosting. Results below
|
||||
use non-audit development rows.
|
||||
The following values use non-audit rows from the **private 10,000-vehicle
|
||||
development sample**. They are **not population performance estimates**.
|
||||
|
||||
| Evaluation | Model | PR-AUC | Brier | ROC-AUC | Top-10% precision |
|
||||
| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| 2023 tuning | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
|
||||
| 2023 tuning | Previous episode | 0.166 | 0.1723 | 0.582 | 0.285 |
|
||||
| 2023 tuning | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** |
|
||||
| 2025 one-time holdout | Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** |
|
||||
| 2025 one-time holdout | Gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 |
|
||||
| 2023 tune | Training-prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
|
||||
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
|
||||
| 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
|
||||
| 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
|
||||
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
|
||||
|
||||
The selected logistic model uses `C=0.03`. It satisfied explicit convergence
|
||||
and finite-value checks. Platt scaling was fit on the 2024 calibration cohort;
|
||||
on that same cohort, PR-AUC was 0.257 and Brier score was 0.0936. Those 2024
|
||||
values are calibration diagnostics, not independent final performance.
|
||||
Platt scaling was fit on the 2024 development-sample calibration partition. Its
|
||||
same-partition PR-AUC of 0.257 and Brier score of 0.0936 are calibration
|
||||
diagnostics, not independent final performance.
|
||||
|
||||
The logistic model remains selected because it beat both simple baselines and
|
||||
the more complex tree on the one-time development holdout while remaining more
|
||||
transparent. The separate never-fit vehicle audit also supported the logistic
|
||||
model, but the development sample is too limited for final generalization or
|
||||
fairness claims.
|
||||
The final logistic model was retained because it beat the two simple baselines
|
||||
on headline development metrics and outperformed the more complex tree in the
|
||||
one-time sample holdout. The benchmark did not justify added complexity.
|
||||
|
||||
## Coverage, fairness, and privacy limitations
|
||||
The never-fit private-vehicle stress-test subset contained 480 eligible 2025
|
||||
sample episodes from 450 vehicles; calibrated logistic PR-AUC was 0.253 and
|
||||
Brier score was 0.0859. These are also private-sample diagnostics, not
|
||||
population or external-validation results.
|
||||
|
||||
- The model applies only to returning vehicles with recognizable labels and
|
||||
sufficient prior history; cold-start behavior is not established.
|
||||
- Geography and source era are entangled. Performance may shift when a feed,
|
||||
county program, vehicle mix, or label process changes.
|
||||
- Reject and abort outcomes can reflect readiness or process issues rather than
|
||||
mechanical failure.
|
||||
- No protected attributes are modeled, but their absence does not establish
|
||||
fairness. Subgroup sample sizes, errors, and calibration still require review.
|
||||
- Make/model normalization and complete-data subgroup analysis are unfinished.
|
||||
- The current probability calibration has not yet received independent,
|
||||
vehicle-clustered uncertainty analysis or external validation.
|
||||
## Limitations
|
||||
|
||||
Private analytical artifacts remain local. Public assets contain only reviewed,
|
||||
rounded aggregates that satisfy episode, distinct-vehicle, and binary-class
|
||||
suppression thresholds. Direct identifiers, private linkage values, raw rows,
|
||||
and row-level predictions are outside the public contract. See
|
||||
[dashboard_spec.md](dashboard_spec.md#publication-controls).
|
||||
- Results apply only to the sampled returning-vehicle cohort with recognizable
|
||||
labels and sufficient prior history.
|
||||
- Page sampling is not population-representative.
|
||||
- Source, program, time, and geography are entangled.
|
||||
- Reject and abort are heterogeneous non-pass outcomes.
|
||||
- Make/model aliases and incomplete feed coverage can distort cohorts.
|
||||
- The one-time 2025 holdout is not an external validation dataset.
|
||||
- No protected attributes are modeled, but that does not establish fairness.
|
||||
- Sample calibration does not establish production calibration.
|
||||
|
||||
## Monitoring
|
||||
## Privacy and public release
|
||||
|
||||
Before any deployment, monitoring must cover source-level volume, label
|
||||
recognition, outcome prevalence, source/program mix, missingness, unseen
|
||||
categories, score distributions, PR-AUC, Brier score, and calibration. Reviews
|
||||
must explicitly separate the Salt Lake source transition, newer rich feeds, DMV
|
||||
coverage gaps, and the partial 2026 period. Alert thresholds and a response plan
|
||||
remain to be defined.
|
||||
Private analytical artifacts remain local and Git-ignored. Dashboard files
|
||||
contain only rounded aggregate cells that pass episode, distinct-vehicle, and
|
||||
binary-class suppression. The browser validates publication flags and checksums
|
||||
and fails closed.
|
||||
|
||||
No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
|
||||
credential, operational record, or row-level prediction belongs in a public
|
||||
asset. The finished dashboard has no vehicle-level prediction service. See the
|
||||
[dashboard specification](dashboard_spec.md#suppression-and-privacy-controls).
|
||||
|
||||
## Reproducibility
|
||||
|
||||
The pipeline uses fixed temporal boundaries and a fixed random seed. Python
|
||||
dependencies are pinned in [requirements.txt](../requirements.txt). Private
|
||||
manifests record input lineage, software/model configuration, convergence, row
|
||||
reconciliation, and checksums without publishing private paths or values. SQL
|
||||
transforms and validation queries are versioned under [sql](../sql), model
|
||||
runners are under [scripts](../scripts), and automated checks are under
|
||||
[tests](../tests). Normal training commands leave the 2025 evaluation gate
|
||||
closed unless an explicit flag is supplied.
|
||||
The pipeline uses fixed chronological boundaries and a fixed seed. Private
|
||||
manifests bind input lineage, configuration, convergence checks, row
|
||||
reconciliation, and checksums. Normal training commands keep the 2025 gate
|
||||
closed unless the explicit evaluation flag is supplied.
|
||||
|
||||
## Remaining approval gates
|
||||
|
||||
1. Run a complete, contiguous, bounded extraction and rebuild the frozen
|
||||
pipeline without page-sampling bias.
|
||||
2. Treat complete-data 2025 results as confirmation and reserve a genuinely new
|
||||
period or external dataset for future unseen evaluation.
|
||||
3. Complete subgroup/source-era reporting, vehicle-clustered uncertainty,
|
||||
calibration diagnostics, and fairness review.
|
||||
4. Finish make/model normalization, episode-gap sensitivity checks, and the
|
||||
preregistered inspection-only versus DMV-enhanced ablation.
|
||||
5. Pass privacy and publication review before replacing the development-preview
|
||||
aggregates or enabling any prediction lookup.
|
||||
6. Define monitoring thresholds, ownership, rollback criteria, and a model
|
||||
update policy before operational use.
|
||||
The detailed sample audit is in
|
||||
[development_results.md](development_results.md), and the narrative conclusion
|
||||
is in [final_report.md](final_report.md).
|
||||
|
||||
@ -1,141 +1,133 @@
|
||||
# Leakage-safe modeling protocol
|
||||
|
||||
## Reporting boundary
|
||||
|
||||
This protocol governs the private, page-sampled 10,000-vehicle development
|
||||
cohort. All resulting counts, rates, comparisons, and performance metrics are
|
||||
**sample results, not population estimates**. County/source reporting is for
|
||||
coverage and drift context only; it must not be turned into population county
|
||||
rankings.
|
||||
|
||||
## Prediction unit
|
||||
|
||||
Score the **first attempt of the next inspection episode** immediately before
|
||||
check-in. Consecutive attempts for the same private vehicle token belong to the
|
||||
same episode when they are no more than 30 days apart. Re-run the analysis with
|
||||
14- and 45-day gaps as sensitivity checks.
|
||||
Estimate the probability that the **first attempt of a returning vehicle's next
|
||||
inspection episode is a non-pass**, immediately before the episode begins.
|
||||
Consecutive attempts for the same private vehicle token form one episode when
|
||||
they are no more than 30 days apart. Rapid retests are not separate targets.
|
||||
|
||||
This prevents rapid fail/retest sequences from dominating the target. Those
|
||||
within-episode attempts belong in the separate failure-to-pass journey analysis.
|
||||
This is the sole research target for the completed prototype. Cold-start,
|
||||
four-class, fail-only, failure-to-pass, and individualized prediction tasks are
|
||||
out of scope.
|
||||
|
||||
## Eligibility
|
||||
|
||||
- Use 2010-2015 events only as historical context; supervised targets begin in
|
||||
2016.
|
||||
- Exclude the four 1990 date outliers.
|
||||
- Require a recognized first-attempt outcome and at least one prior completed
|
||||
episode for the returning-vehicle model.
|
||||
- Keep first-observed vehicles as a separate cold-start cohort.
|
||||
- Exclude the four known 1990 date outliers.
|
||||
- Require a recognized first-attempt binary outcome and at least one prior
|
||||
completed episode.
|
||||
- Keep first-observed vehicles outside the modeled returning-vehicle cohort.
|
||||
- Normalize `slc` and `slco` to Salt Lake County for geography while preserving
|
||||
source era for drift reporting.
|
||||
- Deduplicate exact uploads and quarantine shared/test/placeholder identifiers
|
||||
using preregistered rules for impossible conflicts or extreme activity.
|
||||
- Do not require a DMV match; retain explicit match and staleness indicators.
|
||||
source era for diagnostics.
|
||||
- Deduplicate exact uploads and quarantine shared, test, or placeholder
|
||||
identifiers under preregistered conflict/activity rules.
|
||||
|
||||
Every exclusion must appear in a cohort-flow report.
|
||||
Every exclusion appears in the private cohort-flow audit.
|
||||
|
||||
## Source-specific label contract
|
||||
## Binary label contract
|
||||
|
||||
The normalized target prefers a recognized `overall_result` for every source.
|
||||
The older Utah County feed is the sole exception: when its overall result is
|
||||
blank, `obd_result` may supply the binary target only when the source is
|
||||
`utah`, the program is `obd`, the test type is `OBD`, and the controlled value
|
||||
is pass, fail, reject, or abort. `B`, blank, TSI, `other/C`, and unknown values
|
||||
remain unlabeled. Every target retains `target_outcome_label_source` as private
|
||||
audit metadata, and that field is never a predictor.
|
||||
The target is `0` for a recognized pass and `1` for recognized fail, reject, or
|
||||
abort. Blank, null, and unrecognized values remain unlabeled rather than being
|
||||
treated as passes.
|
||||
|
||||
This is a binary pass-versus-non-pass proxy, not a four-class substitution.
|
||||
Across non-Utah feeds where both fields are recognized, an aggregate audit found
|
||||
99.30% agreement on pass versus non-pass, while the fail/reject distinction was
|
||||
not interchangeable. Utah program rules also distinguish a readiness rejection
|
||||
from a failed inspection. Future four-class analysis must therefore require
|
||||
`target_outcome_label_source = 'overall_result'`. See the official
|
||||
[Utah inspection requirements](https://dmv.utah.gov/register/inspections/) and
|
||||
[program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
|
||||
The normalized target prefers a recognized `overall_result`. The older Utah
|
||||
County feed is the sole exception: when its overall result is blank,
|
||||
`obd_result` may supply the binary label only when source is `utah`, program is
|
||||
`obd`, test type is `OBD`, and the controlled value is pass, fail, reject, or
|
||||
abort. `B`, blank, TSI, `other/C`, and unknown values remain unlabeled. Private
|
||||
label provenance is retained for audit and is never a predictor or public
|
||||
field.
|
||||
|
||||
This exception is a binary pass/non-pass proxy. It does not justify interpreting
|
||||
fail, reject, and abort as interchangeable mechanical conditions.
|
||||
|
||||
## Point-in-time features
|
||||
|
||||
Every historical window ends strictly before the target episode:
|
||||
Every feature window ends strictly before the target episode. The final
|
||||
inspection-history feature set includes:
|
||||
|
||||
- vehicle age;
|
||||
- prior episode and attempt counts;
|
||||
- previous episode first/final outcome;
|
||||
- expanding and trailing prior pass/fail/reject/abort counts and rates;
|
||||
- days since the previous episode and prior adverse result;
|
||||
- attempts required in prior episodes;
|
||||
- vehicle age and canonical make/model from prior information;
|
||||
- physical county and target month/season;
|
||||
- latest DMV record dated before the target, registration count/recency, fuel,
|
||||
and DMV match/staleness indicators; and
|
||||
- missingness indicators.
|
||||
- days since the prior episode and prior adverse outcome;
|
||||
- prior non-pass rate;
|
||||
- prior episode first and final outcomes;
|
||||
- previously observed make and model;
|
||||
- public county context; and
|
||||
- target season.
|
||||
|
||||
The stable core model is inspection-only. The DMV-enhanced model is an explicit
|
||||
ablation because DMV history ends in March 2024 and is nearly absent in 2021.
|
||||
The final prototype does not use DMV enrichment or rich current-test OBD data.
|
||||
|
||||
## Leakage exclusions
|
||||
## Leakage and privacy exclusions
|
||||
|
||||
The MVP must not use:
|
||||
The model and public product exclude:
|
||||
|
||||
- the target attempt's overall/OBD result, result reason, DTC count, MIL or
|
||||
readiness values, PIDs, visual checks, measurements, certificate, or
|
||||
calibration fields;
|
||||
- the target attempt's result, reason, DTC count, MIL/readiness state, PIDs,
|
||||
visual checks, measurements, certificate, or calibration fields;
|
||||
- later attempts or eventual outcome from the target episode;
|
||||
- station or station-level outcome statistics;
|
||||
- raw VIN, private token, plate, ZIP, or other identifiers as features;
|
||||
- target-row vehicle attributes when a prior/static source is available;
|
||||
- future DMV records;
|
||||
- full-history aggregates; or
|
||||
- preprocessing, category mappings, target encodings, or imputation learned
|
||||
from validation/test data.
|
||||
- station, technician, or station-level outcome information;
|
||||
- raw VIN, private vehicle token, plate, ZIP, exact address, or another direct
|
||||
or pseudonymous identifier as a feature;
|
||||
- future records, full-history aggregates, or target-row information that was
|
||||
not available before the episode;
|
||||
- preprocessing, category mappings, imputation, or calibration learned from a
|
||||
later partition; and
|
||||
- raw JSON, credentials, operational records, and row-level predictions in any
|
||||
public artifact.
|
||||
|
||||
Timestamp ties must be resolved before lag/window calculations. Cumulative
|
||||
windows end at the preceding event or episode.
|
||||
Timestamp ties are resolved before lag/window calculations. Cumulative windows
|
||||
end at the preceding event or episode.
|
||||
|
||||
## Fixed evaluation timeline
|
||||
## Frozen chronology
|
||||
|
||||
| Partition | Target dates | Purpose |
|
||||
| Partition | Target dates | Role in the private 10,000-vehicle development sample |
|
||||
| --- | --- | --- |
|
||||
| Historical context | 2010-2015 | Lag features only |
|
||||
| Train | 2016-2022 | Fit preprocessing and models |
|
||||
| Tune | 2023 | Hyperparameters and selection |
|
||||
| Calibrate | 2024 | Probability calibration and thresholds |
|
||||
| Locked test | 2025 | Final reported performance |
|
||||
| Shadow drift | 2026-01-01 to 2026-06-22 | Monitoring only |
|
||||
| Train | 2016-2022 | Fit preprocessing and model parameters |
|
||||
| Tune | 2023 | Select logistic regularization; compare fixed candidates |
|
||||
| Calibrate | 2024 | Fit Platt scaling and inspect calibration behavior |
|
||||
| One-time holdout | 2025 | Development comparison after specifications were frozen |
|
||||
| Shadow | 2026-01-01 to 2026-06-22 | Partial-period monitoring context only |
|
||||
|
||||
The page-sampled development extract's 2025 gate was opened once during live
|
||||
verification after both candidate specifications were fixed. It is therefore
|
||||
a one-time development holdout, not a pristine future test. No model change was
|
||||
made from it; a later complete-data 2025 run is confirmatory. See
|
||||
[development_results.md](development_results.md) for the audit trail. The table
|
||||
continues to define the frozen chronology for a complete extraction.
|
||||
The 2025 gate was opened once during live verification. It is no longer a
|
||||
pristine unseen test, no 2025-informed tuning is permitted, and every reported
|
||||
2025 value must say “one-time development-sample holdout.” A later complete-data
|
||||
2025 analysis would be confirmatory, not a new final test.
|
||||
|
||||
Repeated vehicles may cross ordinary time partitions because returning-vehicle
|
||||
prediction is the deployment scenario. Separately reserve 10% of keyed vehicle
|
||||
buckets as a never-fit VIN audit and report its 2025 performance as an unseen-
|
||||
vehicle stress test. Never use random row splitting.
|
||||
Repeated vehicles may cross chronological partitions because returning-vehicle
|
||||
prediction is the intended scenario. Separately held-out private vehicle buckets
|
||||
form a never-fit stress test. Random row splitting is prohibited.
|
||||
|
||||
## Baselines and candidate model
|
||||
## Models and decision rule
|
||||
|
||||
1. Training prevalence
|
||||
2. Repeat the previous episode's first outcome
|
||||
3. Regularized logistic regression with age splines and one-hot categoricals
|
||||
4. A boosted-tree model
|
||||
5. Inspection-only versus inspection-plus-DMV ablation
|
||||
1. Training prevalence: probability baseline.
|
||||
2. Previous episode's first outcome: literal history baseline.
|
||||
3. Regularized logistic regression: selected using the 2023 sample partition,
|
||||
then Platt-calibrated on the 2024 sample partition; **the final model**.
|
||||
4. Histogram gradient boosting with Platt calibration: fixed nonlinear
|
||||
**benchmark only**.
|
||||
|
||||
Keep validation and test sets at natural prevalence. If pass rows are sampled
|
||||
for training, preserve sampling probabilities and recalibrate on the untouched
|
||||
2024 partition.
|
||||
The benchmark does not power a separate dashboard experience. Added complexity
|
||||
would require a clear, predeclared improvement to displace the logistic model;
|
||||
the one-time 2025 sample comparison did not show that improvement.
|
||||
|
||||
## Metrics
|
||||
## Metrics and interpretation
|
||||
|
||||
Headline binary metrics:
|
||||
Headline metrics are non-pass PR-AUC and Brier score. Log loss, ROC-AUC, and
|
||||
precision at the highest-scored 10% are supporting diagnostics. Calibration
|
||||
values computed on the 2024 partition used to fit Platt scaling are calibration
|
||||
diagnostics, not independent final performance.
|
||||
|
||||
- non-pass PR-AUC;
|
||||
- Brier score and log loss;
|
||||
- calibration intercept, slope, and reliability curve;
|
||||
- precision, recall, and lift at fixed review capacities; and
|
||||
- ROC-AUC as secondary context.
|
||||
|
||||
Use vehicle-clustered bootstrap confidence intervals. Report results by county,
|
||||
source era, vehicle-age band, prior outcome, fuel, history depth, and DMV
|
||||
match/staleness. Four-class analysis adds class-specific and macro PR-AUC,
|
||||
multiclass log loss/Brier score, calibration, and a confusion matrix.
|
||||
|
||||
## Drift contract
|
||||
|
||||
Monitor monthly/source-level volume, label recognition, outcome prevalence,
|
||||
blank rate, source/program mix, missingness, DMV staleness, unseen categories,
|
||||
prediction distribution, PR-AUC, Brier score, and calibration. Treat the
|
||||
`slc`→`slco` transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and
|
||||
partial 2026 period as explicit stress cases rather than ordinary random drift.
|
||||
All tables and charts must state that they use the private 10,000-vehicle
|
||||
development sample and are not population estimates. Report sample support with
|
||||
every subgroup result. Do not describe observed differences as causal, do not
|
||||
claim generalization to all Utah vehicles, and do not publish county rankings.
|
||||
|
||||
170
docs/presentation_outline.md
Normal file
170
docs/presentation_outline.md
Normal file
@ -0,0 +1,170 @@
|
||||
# Utah Vehicle Health: 10-minute presentation outline
|
||||
|
||||
## Presentation rule
|
||||
|
||||
Keep this sentence visible on every results or dashboard slide:
|
||||
|
||||
> Private 10,000-vehicle development sample; sample results, not population
|
||||
> estimates.
|
||||
|
||||
Do not show county rankings, imply statewide coverage, or describe non-pass as
|
||||
mechanical failure. Refer to calibrated logistic regression as the **final
|
||||
model** and histogram gradient boosting as the **benchmark**.
|
||||
|
||||
## Timed outline
|
||||
|
||||
| Time | Slide / action | Core message |
|
||||
| --- | --- | --- |
|
||||
| 0:00-0:40 | 1. Title and boundary | One focused prediction question; development prototype, not a population study or decision tool |
|
||||
| 0:40-1:25 | 2. Research question | Returning vehicle, next episode, first attempt, binary non-pass |
|
||||
| 1:25-2:15 | 3. Development data | Private page-sampled 10,000-vehicle histories; complete sampled histories but nonrepresentative selection |
|
||||
| 2:15-3:10 | 4. Leakage-safe design | 30-day episodes, only pre-episode history, chronological split, never random rows |
|
||||
| 3:10-4:20 | 5. Model comparison | Logistic final versus prevalence/previous-outcome baselines and boosted-tree benchmark |
|
||||
| 4:20-4:50 | 6. Model decision | Calibrated logistic wins the declared sample comparison and is easier to explain |
|
||||
| 4:50-7:35 | Live dashboard demo | Overview, Sample cohorts, Model & benchmark, Data & methods |
|
||||
| 7:35-8:30 | 7. Privacy architecture | Private pipeline to suppressed aggregate JSON to dashboard-only Bolt project |
|
||||
| 8:30-9:20 | 8. Limitations | Sampling, feed coverage, source/time confounding, heterogeneous non-pass, one-time holdout |
|
||||
| 9:20-10:00 | 9. Close | Prototype is complete and honest about what it can—and cannot—claim |
|
||||
|
||||
Total: **10:00**
|
||||
|
||||
## Slide notes
|
||||
|
||||
### 1. Title and boundary — 40 seconds
|
||||
|
||||
**Title:** Utah Vehicle Health
|
||||
|
||||
**Subtitle:** Predicting a returning vehicle's next-episode first-attempt
|
||||
non-pass outcome
|
||||
|
||||
Say:
|
||||
|
||||
> This is a presentation-ready development prototype built from a private,
|
||||
> page-sampled 10,000-vehicle cohort. Every result I show is sample-based, not a
|
||||
> population estimate. The project does not rank counties or diagnose vehicles.
|
||||
|
||||
### 2. Research question — 45 seconds
|
||||
|
||||
Show the question:
|
||||
|
||||
> Using only information available before an inspection episode begins, how
|
||||
> well can a calibrated logistic regression estimate whether a returning
|
||||
> vehicle's next episode will have a first-attempt non-pass outcome?
|
||||
|
||||
Explain that attempts separated by no more than 30 days form one episode. This
|
||||
keeps rapid fail/retest sequences from becoming repeated target rows. Non-pass
|
||||
combines fail, reject, and abort; blanks remain unlabeled.
|
||||
|
||||
### 3. Development data — 50 seconds
|
||||
|
||||
Show a compact flow:
|
||||
|
||||
```text
|
||||
sample up to 10,000 vehicles -> retrieve their histories -> build episodes
|
||||
-> create 44,659 eligible returning targets
|
||||
```
|
||||
|
||||
Say that 9,996 private tokens remained after validation, producing 83,552 source
|
||||
events and 69,588 episodes in the private sample. Immediately repeat that these
|
||||
are reconciliation counts, not Utah totals. Page sampling over-represents
|
||||
vehicles with more inspection records, and participating feeds do not cover all
|
||||
29 counties.
|
||||
|
||||
### 4. Leakage-safe design — 55 seconds
|
||||
|
||||
Show the chronology:
|
||||
|
||||
```text
|
||||
2010-15 context | 2016-22 train | 2023 tune | 2024 calibrate
|
||||
| 2025 one-time holdout | 2026 partial shadow
|
||||
```
|
||||
|
||||
Explain that every feature window ends before the target episode. The model uses
|
||||
prior outcomes, history depth, timing, age, previously observed make/model,
|
||||
public county context, and season. It excludes current-test diagnostics, future
|
||||
records, identifiers, stations, technicians, and later attempts.
|
||||
|
||||
Note that the 2025 sample gate was opened once after specifications were frozen;
|
||||
it is a development holdout, not a pristine future test.
|
||||
|
||||
### 5. Model comparison — 70 seconds
|
||||
|
||||
Use a two-metric table. Label it “Private 10,000-vehicle development sample; not
|
||||
population performance.”
|
||||
|
||||
| 2025 one-time sample holdout | Role | PR-AUC | Brier |
|
||||
| --- | --- | ---: | ---: |
|
||||
| Training prevalence | Baseline | 0.123 | 0.1076 |
|
||||
| Previous episode | Baseline | 0.158 | 0.1731 |
|
||||
| Logistic + Platt | **Final** | **0.261** | **0.1011** |
|
||||
| Boosted tree + Platt | Benchmark | 0.238 | 0.1024 |
|
||||
|
||||
Say that higher PR-AUC and lower Brier are better. Logistic beat both baselines
|
||||
and the boosted-tree benchmark in this one-time sample comparison. Avoid the
|
||||
word “accuracy,” which obscures class imbalance and calibration.
|
||||
|
||||
### 6. Model decision — 30 seconds
|
||||
|
||||
Say:
|
||||
|
||||
> The final model is calibrated logistic regression. The boosted tree is a
|
||||
> benchmark only. It did not improve the declared sample metrics enough to
|
||||
> justify greater complexity, while logistic regression is easier to audit and
|
||||
> explain.
|
||||
|
||||
### Live dashboard demo — 2 minutes 45 seconds
|
||||
|
||||
Follow [demo_script.md](demo_script.md). Keep the development-preview banner in
|
||||
view at the start. Explicitly show calibrated logistic as final, the tree as
|
||||
benchmark-only, and the absence of a vehicle-level prediction service.
|
||||
|
||||
### 7. Privacy architecture — 55 seconds
|
||||
|
||||
Show:
|
||||
|
||||
```text
|
||||
private read-only data -> local modeling -> suppressed aggregate JSON
|
||||
-> dashboard-only Bolt project
|
||||
```
|
||||
|
||||
State that the public bundle has no VINs, plates, ZIPs, stations, technician
|
||||
identifiers, private tokens, raw JSON, credentials, operational records, or
|
||||
row-level predictions. The browser has no database connection and fails closed
|
||||
if its aggregate contract does not validate.
|
||||
|
||||
### 8. Limitations — 50 seconds
|
||||
|
||||
Name the limitations directly:
|
||||
|
||||
- page-sampled cohort is not population-representative;
|
||||
- feeds are partial and change over time;
|
||||
- geography and source era are confounded;
|
||||
- reject/abort do not necessarily mean mechanical failure;
|
||||
- 2025 is a one-time development holdout, not external validation; and
|
||||
- sample associations do not support county rankings or causal conclusions.
|
||||
|
||||
### 9. Close — 40 seconds
|
||||
|
||||
End with:
|
||||
|
||||
> The prototype is complete at the development stage: one precise question, a
|
||||
> leakage-safe timeline, calibrated logistic regression as the final model, a
|
||||
> boosted-tree benchmark, and a privacy-reviewed static dashboard. Its value is
|
||||
> not just the sample performance; it is the discipline of making only the
|
||||
> claims the data and release boundary can support.
|
||||
|
||||
Pause, then invite questions about episode construction, calibration, privacy,
|
||||
or the dashboard—not about county “winners” and “losers,” which this sample
|
||||
cannot establish.
|
||||
|
||||
## Presentation checklist
|
||||
|
||||
- Start the local dashboard before presenting: `node dashboard/server.mjs`.
|
||||
- Open `http://127.0.0.1:4173/#overview` and close unrelated browser tabs.
|
||||
- Use a fresh page load to confirm aggregate validation succeeds.
|
||||
- Keep a screenshot or short recording as a backup, with the sample warning
|
||||
visible.
|
||||
- Never open private data, artifacts, terminal environment variables, database
|
||||
tools, or browser developer tools during the talk.
|
||||
- Do not type or display a VIN, plate, ZIP, station, technician identifier, raw
|
||||
record, or row-level prediction.
|
||||
@ -1,122 +1,110 @@
|
||||
# Utah Vehicle Health project charter
|
||||
|
||||
## Working title
|
||||
## Prototype status
|
||||
|
||||
**Utah Vehicle Health**
|
||||
|
||||
*What millions of emissions inspections reveal about the next test across the
|
||||
state of Utah county feeds available in this dataset.*
|
||||
**Presentation-ready development prototype.** Every analytical finding and every
|
||||
dashboard value comes from the private, page-sampled 10,000-vehicle development
|
||||
cohort. These values are **sample results, not population estimates**. They must
|
||||
not be used to estimate statewide or county prevalence, create county rankings,
|
||||
or make decisions about an individual vehicle.
|
||||
|
||||
## Product promise
|
||||
|
||||
Explain how vehicle age, type, location, and prior inspection history relate to
|
||||
the chance of passing the first attempt of the next emissions-inspection
|
||||
episode.
|
||||
Utah Vehicle Health explains how information available before an inspection
|
||||
relates to the first-attempt non-pass outcome of a returning vehicle's next
|
||||
inspection episode.
|
||||
|
||||
The product measures **emissions-inspection outcomes**. It must not describe its
|
||||
score as a diagnosis of overall mechanical reliability, roadworthiness, safety,
|
||||
or legal compliance.
|
||||
The product measures an emissions-inspection outcome. It does not measure
|
||||
overall vehicle health, mechanical reliability, safety, roadworthiness, or
|
||||
legal compliance.
|
||||
|
||||
## Primary research question
|
||||
|
||||
> Using only information available before an inspection episode begins, how
|
||||
> accurately and reliably can we estimate whether its first attempt will pass?
|
||||
> well can a calibrated logistic regression estimate whether a returning
|
||||
> vehicle's next episode will have a first-attempt non-pass outcome?
|
||||
|
||||
An episode groups attempts for the same internal vehicle token when the gap
|
||||
between consecutive attempts is 30 days or less. The target is the first attempt
|
||||
of a new episode, not every rapid retest.
|
||||
An episode groups consecutive attempts for the same private vehicle token when
|
||||
the gap is 30 days or less. The target is the first attempt of a new episode,
|
||||
not a rapid retest. A vehicle is eligible only after at least one completed
|
||||
prior episode.
|
||||
|
||||
## Primary target
|
||||
## Binary target
|
||||
|
||||
- `0 — pass`: normalized `PASS` or `P`
|
||||
- `1 — non-pass`: normalized `FAIL`, `F`, `REJECT`, or `ABORT`
|
||||
- Blank, null, and unrecognized results are unlabeled. They remain in chronology
|
||||
and coverage reporting but are excluded as supervised targets.
|
||||
- `0 - pass`: normalized `PASS` or `P`
|
||||
- `1 - non-pass`: normalized `FAIL`, `F`, `REJECT`, or `ABORT`
|
||||
- unlabeled: blank, null, and unrecognized results
|
||||
|
||||
Because reject and abort can reflect process/readiness problems rather than
|
||||
mechanical failure, the project will also report:
|
||||
Reject and abort can reflect process or readiness conditions rather than a
|
||||
mechanical failure. The prototype therefore says **non-pass**, never “vehicle
|
||||
failure,” when referring to the combined target. Multiclass outcome modeling
|
||||
and fail-only sensitivity analysis are outside the finished prototype scope.
|
||||
|
||||
- a four-class pass/fail/reject/abort analysis; and
|
||||
- a fail-versus-pass sensitivity analysis that excludes reject and abort.
|
||||
## Data and product boundary
|
||||
|
||||
## Scope
|
||||
The development cohort contains complete inspection histories for up to 10,000
|
||||
sampled vehicles from participating Utah county/source feeds. Page sampling
|
||||
over-represents vehicles with more inspection records, so neither the cohort nor
|
||||
its dashboard aggregates are population-representative. The feeds do not cover
|
||||
all 29 Utah counties.
|
||||
|
||||
The returning-vehicle MVP uses universal inspection-history fields and prior
|
||||
DMV information where it is point-in-time valid. It covers participating
|
||||
inspection county/source feeds, not all 29 Utah counties.
|
||||
The final model is inspection-history only. DMV enrichment, rich current-test
|
||||
OBD fields, station effects, cold-start prediction, and individualized lookup
|
||||
are outside scope. The static dashboard publishes only rounded, suppressed
|
||||
sample aggregates and pre-2025 development diagnostics.
|
||||
|
||||
The rich OBD/odometer fields are a later, source-specific extension for newer
|
||||
`slco`, `davis`, and `cache` records. They are not part of the statewide-style
|
||||
historical baseline.
|
||||
## Final model and benchmark
|
||||
|
||||
## Deliverables
|
||||
The final model is regularized logistic regression (`C=0.03`) with Platt
|
||||
probability calibration fit on the 2024 development-sample partition. A
|
||||
histogram gradient-boosted tree is retained only as a nonlinear benchmark. It
|
||||
is not a second final model and is not used to drive the product.
|
||||
|
||||
1. A reproducible, read-only extraction and private pseudonymization pipeline.
|
||||
2. A point-in-time episode/feature mart with an auditable exclusion report.
|
||||
3. Prevalence, previous-outcome, and logistic-regression baselines.
|
||||
4. One calibrated boosted-tree model and an inspection-only versus DMV-enhanced
|
||||
ablation.
|
||||
5. A locked temporal evaluation with subgroup and source-era diagnostics.
|
||||
6. Versioned, suppressed public aggregates for a four-page Bolt dashboard.
|
||||
7. A model card and data/methodology page documenting limitations.
|
||||
## Completed deliverables
|
||||
|
||||
## Headline success criteria
|
||||
1. Read-only extraction with private vehicle pseudonymization.
|
||||
2. Leakage-safe episode and point-in-time feature mart.
|
||||
3. Training-prevalence and previous-episode baselines.
|
||||
4. Calibrated logistic regression as the final model.
|
||||
5. Histogram gradient boosting as a benchmark only.
|
||||
6. Chronological development evaluation with an explicit one-time 2025 gate.
|
||||
7. Suppression-reviewed static dashboard assets with fail-closed validation.
|
||||
8. Model card, final report, presentation materials, and dashboard-only Bolt
|
||||
deployment instructions.
|
||||
|
||||
- Beat both the training-prevalence and previous-episode-outcome baselines on
|
||||
2025 PR-AUC and Brier score.
|
||||
- Produce calibrated probabilities, not just class labels.
|
||||
- Report performance by county/source era, vehicle-age band, history depth, and
|
||||
DMV match/staleness.
|
||||
- Reproduce all published charts from versioned sanitized outputs.
|
||||
- Export no VIN, plate, ZIP, station, raw JSON, operational record, or
|
||||
row-level prediction to Bolt.
|
||||
- Suppress public cells with fewer than 100 eligible inspections or fewer than
|
||||
10 observations in an outcome or its complement, with complementary
|
||||
suppression where totals could reveal a hidden cell.
|
||||
## Acceptance criteria
|
||||
|
||||
## Initial feasibility result
|
||||
|
||||
Under label contract v3, the fixed-seed aggregate-only query in
|
||||
`sql/10_episode_cohort_feasibility.sql` produced 8,916 eligible
|
||||
returning-vehicle episodes from 1,730 of 2,000 sampled vehicle histories. The
|
||||
first-attempt non-pass rate was 11.80%, and the median gap from the prior
|
||||
episode was about 372 days. Of those targets, 1,732 use the documented Utah OBD
|
||||
binary proxy because that source leaves its overall-result field blank. This
|
||||
supports both the episode definition and a calibrated binary model. The sample
|
||||
is for pipeline feasibility, not a population estimate.
|
||||
|
||||
The 2025 sample showed a longer median gap and different outcome mix, reinforcing
|
||||
the need for source-era drift reporting and a locked chronological test.
|
||||
|
||||
## Development holdout status
|
||||
|
||||
During live verification, the explicit 2025 gate was opened once on the
|
||||
page-sampled development extract after both candidate specifications had been
|
||||
fixed from pre-2025 data. No test-informed model change was made. The frozen
|
||||
logistic model remains selected, and a later complete-data 2025 analysis will
|
||||
be treated as confirmatory rather than described as a pristine unseen test.
|
||||
The audit trail and results are recorded in
|
||||
[development_results.md](development_results.md).
|
||||
- The prediction unit remains a returning vehicle's next-episode first attempt.
|
||||
- Model features exist before that episode begins.
|
||||
- The calibrated logistic model beats both simple baselines on development-
|
||||
sample PR-AUC and Brier score.
|
||||
- The boosted tree is presented only as a benchmark.
|
||||
- Every displayed value is labeled as a private-sample result and not a
|
||||
population estimate.
|
||||
- County views communicate feed coverage and sample context; they do not claim
|
||||
population rankings or causal county differences.
|
||||
- No VIN, plate, ZIP, station, technician identifier, raw JSON, credential,
|
||||
operational record, private token, or row-level prediction reaches the
|
||||
dashboard or presentation materials.
|
||||
- The dashboard contains no vehicle-level estimator, input form, or prediction
|
||||
service.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Diagnosing an individual vehicle
|
||||
- Certifying that a vehicle will pass
|
||||
- Ranking or accusing inspection stations
|
||||
- Identifying owners or accepting VIN/plate input
|
||||
- Making causal claims about county programs
|
||||
- Treating missing outcomes as passes
|
||||
- Predicting the result of a first-observed vehicle
|
||||
- Diagnosing, certifying, or guaranteeing an individual vehicle outcome
|
||||
- Ranking counties, programs, stations, technicians, owners, or vehicles
|
||||
- Making causal claims about geography, vehicle makes, or inspection programs
|
||||
- Accepting VIN, plate, address, station, or diagnostic inputs
|
||||
- Publishing population prevalence from the development sample
|
||||
- Shipping a production decision service or live database connection
|
||||
- Completing DMV enrichment, four-class modeling, or failure-to-pass journeys
|
||||
|
||||
## Staged build
|
||||
## Evaluation status
|
||||
|
||||
1. **Foundation:** episode definition, label normalization, exclusions, secure
|
||||
extraction, and coverage checks.
|
||||
2. **Baseline:** inspection-history feature mart and transparent baselines.
|
||||
3. **Enrichment:** canonical vehicle dimension and point-in-time DMV features.
|
||||
4. **Modeling:** tree model, calibration, locked test, drift and subgroup audits.
|
||||
5. **Product:** sanitized aggregate export and Bolt MVP.
|
||||
6. **Stretch:** failure-to-pass journeys, multiclass probabilities, and a
|
||||
clearly scoped rich-OBD model.
|
||||
|
||||
See [modeling_protocol.md](modeling_protocol.md) and
|
||||
[dashboard_spec.md](dashboard_spec.md) for the detailed contracts.
|
||||
The explicit 2025 gate was opened once after model specifications were frozen
|
||||
from pre-2025 data. No 2025-informed model change was made. The 2025 values are
|
||||
therefore a one-time **development-sample holdout**, not a pristine future test
|
||||
and not population performance. See [development_results.md](development_results.md)
|
||||
for the audit trail and [modeling_protocol.md](modeling_protocol.md) for the
|
||||
frozen evaluation contract.
|
||||
|
||||
@ -2,6 +2,13 @@
|
||||
|
||||
Inventory date: 2026-07-15
|
||||
|
||||
> **Archived planning context:** This shortlist records ideas considered during
|
||||
> discovery; it is not the scope of the finished prototype. DMV enrichment,
|
||||
> multiclass outcomes, station analysis, forecasting, and population/county
|
||||
> rankings below are not current deliverables. The authoritative scope is the
|
||||
> [project charter](project_charter.md), and the completed development story is
|
||||
> in the [final report](final_report.md).
|
||||
|
||||
> **Selected:** Option 1, Utah Vehicle Health. The implementation contract is in
|
||||
> [project_charter.md](project_charter.md).
|
||||
|
||||
|
||||
@ -97,6 +97,14 @@ REQUIRED_MART_COLUMNS = {
|
||||
"last_observed_model",
|
||||
}
|
||||
DENIED_EXACT_KEYS = {
|
||||
"credential",
|
||||
"credentials",
|
||||
"inspector",
|
||||
"inspector_id",
|
||||
"password",
|
||||
"secret",
|
||||
"technician",
|
||||
"technician_id",
|
||||
"vehicle_token",
|
||||
"vehicle_bucket",
|
||||
"is_vin_audit",
|
||||
@ -122,6 +130,11 @@ DENIED_EXACT_KEYS = {
|
||||
"row_count",
|
||||
}
|
||||
DENIED_KEY_FRAGMENTS = (
|
||||
"credential",
|
||||
"inspector",
|
||||
"password",
|
||||
"secret",
|
||||
"technician",
|
||||
"vehicle_token",
|
||||
"internal_event",
|
||||
"license_plate",
|
||||
|
||||
@ -89,6 +89,17 @@ PLATT_CALIBRATION_CONFIG = {
|
||||
# These fields are forbidden even when they are not selected as model features.
|
||||
# Their presence indicates that the input is not the approved episode mart.
|
||||
FORBIDDEN_EXACT_COLUMNS = {
|
||||
"credential",
|
||||
"credentials",
|
||||
"inspector",
|
||||
"inspector_id",
|
||||
"password",
|
||||
"prediction",
|
||||
"probability",
|
||||
"row_prediction",
|
||||
"secret",
|
||||
"technician",
|
||||
"technician_id",
|
||||
"vin",
|
||||
"plate",
|
||||
"zip",
|
||||
@ -109,7 +120,14 @@ FORBIDDEN_EXACT_COLUMNS = {
|
||||
"target_final_outcome",
|
||||
}
|
||||
FORBIDDEN_COLUMN_FRAGMENTS = (
|
||||
"credential",
|
||||
"inspector",
|
||||
"license_plate",
|
||||
"password",
|
||||
"raw_json",
|
||||
"row_prediction",
|
||||
"secret",
|
||||
"technician",
|
||||
"certificate",
|
||||
"calibration_id",
|
||||
"visual_check",
|
||||
@ -122,17 +140,27 @@ FORBIDDEN_COLUMN_FRAGMENTS = (
|
||||
FORBIDDEN_UNKNOWN_COLUMN_TOKENS = {
|
||||
"address",
|
||||
"certificate",
|
||||
"credential",
|
||||
"dtc",
|
||||
"email",
|
||||
"inspector",
|
||||
"ip",
|
||||
"json",
|
||||
"mil",
|
||||
"obd",
|
||||
"owner",
|
||||
"password",
|
||||
"payload",
|
||||
"pid",
|
||||
"plate",
|
||||
"prediction",
|
||||
"probability",
|
||||
"raw",
|
||||
"readiness",
|
||||
"secret",
|
||||
"session",
|
||||
"station",
|
||||
"technician",
|
||||
"user",
|
||||
"vin",
|
||||
"zip",
|
||||
|
||||
@ -104,6 +104,20 @@ class SchemaTests(unittest.TestCase):
|
||||
with self.assertRaisesRegex(baselines.SchemaError, "owner_zip_code"):
|
||||
baselines.validate_schema(columns)
|
||||
|
||||
def test_prohibited_sensitive_columns_fail_closed(self) -> None:
|
||||
for column in (
|
||||
"technician_id",
|
||||
"inspection_technician_code",
|
||||
"inspector_identifier",
|
||||
"database_password",
|
||||
"api_secret_value",
|
||||
"raw_inspection_json",
|
||||
"row_prediction_probability",
|
||||
):
|
||||
with self.subTest(column=column):
|
||||
with self.assertRaisesRegex(baselines.SchemaError, column):
|
||||
baselines.validate_schema(list(valid_record().keys()) + [column])
|
||||
|
||||
def test_duplicate_columns_fail_closed(self) -> None:
|
||||
columns = list(valid_record().keys()) + ["vehicle_token"]
|
||||
with self.assertRaisesRegex(baselines.SchemaError, "duplicate"):
|
||||
|
||||
@ -193,6 +193,19 @@ class DashboardExportTests(unittest.TestCase):
|
||||
"development_preview": True,
|
||||
"population_estimate_allowed": False,
|
||||
}
|
||||
for unsafe_field in (
|
||||
"technician_id",
|
||||
"inspection_technician_code",
|
||||
"inspector_identifier",
|
||||
"database_password",
|
||||
"api_secret_value",
|
||||
):
|
||||
with self.subTest(unsafe_field=unsafe_field):
|
||||
with self.assertRaises(dashboard_export.PublicSchemaError):
|
||||
dashboard_export._scan_public_value(
|
||||
{unsafe_field: "not-public"}
|
||||
)
|
||||
|
||||
unsafe_key = {
|
||||
**base,
|
||||
"rows": [
|
||||
|
||||
Loading…
Reference in New Issue
Block a user