Centralized_I-M_Database/SummerProject2026
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

repository is now a presentation-ready development prototype

Browse Source
This commit is contained in:
Kevin Bell 2026-07-21 15:40:44 -06:00
parent 640165649d
commit 88161a6f16
24 changed files with 1989 additions and 946 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

214
README.md
View File

@ -1,61 +1,57 @@
# Summer Project 2026 # Utah Vehicle Health
kevinBell@Linux.com Kevin Bell's summer 2026 development prototype for the first-attempt outcome of
a returning vehicle's next emissions-inspection episode.
https://kevinbell.dev/ ## Prototype status
Kevin Bell's summer 2026 data project using state of Utah county vehicle-registration The repository implements one research question:
and emissions-inspection data.
See the [data inventory](docs/data_inventory.md) and the data-backed > Using only information available before an inspection episode begins, how
[project shortlist](docs/project_options.md). > well can a calibrated logistic regression estimate whether a returning
> vehicle's next episode will have a first-attempt non-pass outcome?
## Selected project The final model is regularized logistic regression with Platt calibration.
Histogram gradient boosting is a benchmark only. The final feature set uses
inspection history; DMV enrichment, multiclass modeling, cold-start prediction,
station analysis, and vehicle-level prediction service are outside scope.
The selected direction is **Utah Vehicle Health**, an explainable model and All analytical results and dashboard values come from the private, page-sampled
dashboard for the first-attempt outcome of a vehicle's next inspection episode. 10,000-vehicle development cohort. They are **sample results, not population
Start with the [project charter](docs/project_charter.md), then use the estimates**. They must not be used for statewide/county prevalence, county
[modeling protocol](docs/modeling_protocol.md) and rankings, causal claims, individual decisions, diagnoses, or guarantees.
[dashboard specification](docs/dashboard_spec.md) as the project contracts.
The latest private-sample pipeline findings are summarized in
[development results](docs/development_results.md), with intended use and
limitations consolidated in the [model card](docs/model_card.md).
## Safe database access in VS Code Start with the [final report](docs/final_report.md). Supporting deliverables:
The local `.env` file contains the standard PostgreSQL connection variables and - [Project charter](docs/project_charter.md)
is intentionally excluded from Git. Never put those values in source code, - [Leakage-safe modeling protocol](docs/modeling_protocol.md)
screenshots, browser JavaScript, or a Bolt project. - [Detailed development results](docs/development_results.md)
- [Model card](docs/model_card.md)
- [Static dashboard specification](docs/dashboard_spec.md)
- [Private-to-public architecture](docs/architecture.mmd)
- [10-minute presentation outline](docs/presentation_outline.md)
- [Dashboard demo script](docs/demo_script.md)
- [Dashboard-only Bolt deployment](docs/bolt_deployment.md)
1. Install the workspace-recommended **PostgreSQL** extension by Microsoft The [data inventory](docs/data_inventory.md) and archived
(`ms-ossdata.vscode-pgsql`). [project options](docs/project_options.md) provide source-discovery history;
2. In VS Code, run **Tasks: Run Task** and choose they do not expand the finished prototype scope.
**Countydata: test read-only connection**. It loads `.env`, requires TLS, and
forces the PostgreSQL session into read-only mode.
3. To browse the server visually, open the PostgreSQL sidebar and add a
connection using the values in `.env`. Set SSL mode to **Require**, save the
profile at **User** scope, and store the password in macOS Keychain rather
than workspace settings.
4. Connect to `countydata` and run [sql/00_read_only_connection_check.sql](sql/00_read_only_connection_check.sql).
Safe aggregate examples are in [sql/01_safe_data_overview.sql](sql/01_safe_data_overview.sql).
5. To inspect every visible database, relation, column, and a bounded sample of
raw JSON field paths without printing row values, run **Countydata: inventory
metadata safely**. The task uses [scripts/inventory_metadata.py](scripts/inventory_metadata.py)
and enforces the same TLS and read-only requirements.
The selected project's first aggregate cohort check is ## Privacy boundary
[sql/10_episode_cohort_feasibility.sql](sql/10_episode_cohort_feasibility.sql).
Private bounded extraction is handled by
[scripts/export_inspection_batch.py](scripts/export_inspection_batch.py); it
requires the project-specific `VIN_HASH_KEY` described in `.env.example` and
never writes a raw VIN.
The current database login has write privileges even though this project only Never publish or display VINs, plates, ZIPs, stations, technician identifiers,
needs reads. Ask the database administrator for a dedicated read-only role private vehicle tokens, raw JSON, credentials, operational records, or row-level
before connecting any deployed service. Until then, always use explicit predictions. Private extracts, marts, databases, keys, and model artifacts stay
read-only transactions. in Git-ignored local paths. The browser never connects to `countydata`.
## Reproducible development workflow ```text
countydata (read-only)
-> private local extraction, feature, and model pipeline
-> suppression-reviewed aggregate JSON
-> dashboard-only static site
```
## Local setup
Create the local environment once: Create the local environment once:
@ -65,71 +61,103 @@ python3 -m venv .venv
.venv/bin/python scripts/create_vin_hash_key.py .venv/bin/python scripts/create_vin_hash_key.py
``` ```
The recommended way to run the pipeline is **Tasks: Run Task** in VS Code. The The local `.env` contains PostgreSQL connection values and the project-specific
tasks preserve the intended order: `VIN_HASH_KEY`; it is excluded from Git. Never put those values in source code,
screenshots, browser JavaScript, presentation materials, or a Bolt project.
## Safe database access in VS Code
1. Install the workspace-recommended Microsoft PostgreSQL extension
(`ms-ossdata.vscode-pgsql`).
2. Run **Tasks: Run Task > Countydata: test read-only connection**. The task
requires TLS and forces the PostgreSQL session into read-only mode.
3. If using the PostgreSQL sidebar, store the profile at User scope and the
password in macOS Keychain, not workspace settings.
4. Run [sql/00_read_only_connection_check.sql](sql/00_read_only_connection_check.sql)
after connecting. Safe aggregate examples are in
[sql/01_safe_data_overview.sql](sql/01_safe_data_overview.sql).
5. Run **Countydata: inventory metadata safely** to inspect relation/field
metadata without printing row values. The task uses
[scripts/inventory_metadata.py](scripts/inventory_metadata.py).
The current database login has broader privileges than this project needs.
Continue to enforce read-only transactions and use a dedicated read-only role
before any separately approved operational integration.
## Reproducible private workflow
The VS Code tasks preserve the intended order:
1. Test the read-only TLS connection. 1. Test the read-only TLS connection.
2. Run the source outcome-mapping and aggregate feasibility audits. 2. Run label-mapping and aggregate feasibility audits.
3. Export the private 10,000-vehicle development histories. 3. Export the private 10,000-vehicle development histories.
4. Build the private DuckDB warehouse and Parquet feature mart. 4. Build the private DuckDB warehouse and Parquet feature mart.
5. Run every test. 5. Run the test suite.
6. Train the baselines while leaving the 2025 test partition locked. 6. Train the baselines and logistic model with the 2025 gate closed.
7. Train the nonlinear comparison while leaving the same partition locked. 7. Train the boosted-tree benchmark with the same gate closed.
8. Export the sanitized dashboard preview from approved pre-2025 partitions.
The development history sample is deliberately not population-representative. The extraction is deliberately not population-representative. It page-samples
It exists to exercise feature engineering and modeling before a complete, inspection records and then retrieves complete histories for the selected
contiguous bounded extraction is approved. Generated extracts, marts, database vehicles, which over-represents vehicles with more inspection records.
files, keys, and model artifacts remain under Git-ignored private paths.
The 2025 metrics are not calculated by the normal training task. Unlocking them The explicit 2025 gate was opened once after specifications were frozen. No
requires the conspicuous `--evaluate-locked` flag after the feature set, model was changed in response. The values in the final report are one-time
hyperparameters, calibration method, and reporting plan are frozen. development-sample holdout diagnostics—not pristine future-test or population
That gate was opened once on the page-sampled development extract during live performance. Normal training leaves the gate closed.
verification; the normal artifact paths were then regenerated closed. The
audit trail is in [development results](docs/development_results.md), and no
further 2025-informed tuning is permitted.
## Sanitized dashboard preview ## Tests
After training both closed model bundles, run **Tasks: Run Task** and choose Run the complete repository test suite:
**Utah Vehicle Health: export sanitized dashboard preview**. The exporter
publishes only suppression-reviewed JSON under `dashboard/public/data`; it
refuses the sampled mart unless the explicit development-preview flag is used
and refuses model artifacts containing holdout metrics.
Preview the site without exposing the repository root: ```bash
.venv/bin/python -m unittest discover -s tests -v
```
Run the dashboard contract checks directly:
```bash
npm --prefix dashboard test
```
## Static dashboard
Preview the site from the repository root:
```bash ```bash
node dashboard/server.mjs node dashboard/server.mjs
``` ```
Then open `http://localhost:4173`. Do not serve the repository root, because it Open `http://127.0.0.1:4173`. Do not open `index.html` with a `file:` URL and do
contains the local connection profile and private ignored directories. not serve the repository root.
The checked-in bundle is visibly labeled as a non-population development The four routes are Overview, Sample cohorts, Model & benchmark, and Data &
preview. Before a real public-data release, rerun the frozen pipeline on an methods. The site identifies calibrated logistic regression as final, the tree
approved complete extraction and pass the publication review documented in as benchmark-only, and displayed model cards as 2024 calibration-fit checks.
[the dashboard specification](docs/dashboard_spec.md). The opened 2025 comparison stays in the report and presentation, not the public
JSON. There is no vehicle-level prediction service.
Local and Bolt handoff instructions are in The browser validates schemas, publication flags, cross-file consistency, and
[dashboard/README.md](dashboard/README.md). Bolt can import a GitHub repository, checksums. It shows no estimates if the approved aggregate bundle fails its
but the hosted site root must be the `dashboard` directory; never copy `.env`, contract. Every checked-in asset is marked as a development preview and not a
private data, model artifacts, or database tooling into a web project. population estimate.
## Data safety and publishing ## Publish through Bolt
The source contains direct identifiers and operational data, including VINs, Do **not** import this full source repository into Bolt. Create a separate
plates, user records, sessions, and upload metadata. A public dashboard should deployment project or repository containing only the contents of `dashboard/`,
contain only de-identified aggregates and model outputs with minimum group-size so `index.html` is at that project's root. Do not copy `.env`, private data,
suppression. artifacts, models, SQL, pipeline scripts, credentials, or unrelated repository
history.
Recommended deployment flow: Follow the complete preflight, import, Bolt Hosting, and post-publication checks
in [docs/bolt_deployment.md](docs/bolt_deployment.md). No publishing action is
performed by this repository.
```text ## Author
countydata (read-only) -> local ETL/modeling -> sanitized aggregate tables/files
-> Bolt-hosted dashboard
```
Do not connect a browser directly to `countydata`. If the dashboard must refresh Kevin Bell
automatically, use a scheduled backend job with a dedicated read-only database
role and copy only approved aggregate results into the hosted application. <kevinBell@Linux.com>
<https://kevinbell.dev/>

66
dashboard/README.md
View File

@ -1,10 +1,18 @@
# Utah Vehicle Health dashboard # Utah Vehicle Health dashboard
This directory is a dependency-free static prototype. It uses semantic HTML, This directory is a dependency-free static development prototype. It uses
CSS, vanilla ES modules, and inline SVG generated from approved aggregate JSON. semantic HTML, CSS, vanilla ES modules, and inline SVG generated from approved,
It has no database client, server credential, analytics SDK, or external CDN. suppressed aggregate JSON. It has no database client, server credential,
analytics SDK, or external CDN.
## Run locally from the repository root Every displayed result comes from the private 10,000-vehicle development
sample. Results are sample-based, are not population estimates, and must not be
used for individual decisions. The final prototype model is calibrated logistic
regression; the boosted tree is a benchmark only.
## Run locally
From the Utah Vehicle Health source-repository root:
```bash ```bash
node dashboard/server.mjs node dashboard/server.mjs
@ -23,6 +31,14 @@ Run the dependency-free contract checks with:
node --test dashboard/tests/contract.test.mjs node --test dashboard/tests/contract.test.mjs
``` ```
After the contents of `dashboard/` have been copied into a dashboard-only
repository or project root, use the self-contained equivalents:
```bash
npm start
npm test
```
## Public-data boundary ## Public-data boundary
The browser requires and validates these files beneath `public/data/`: The browser requires and validates these files beneath `public/data/`:
@ -41,27 +57,33 @@ contains a denied identifier-like field, the dashboard shows an unavailable
state and no estimates. Regenerate assets with the repository's private local state and no estimates. Regenerate assets with the repository's private local
pipeline; never hand-edit public JSON to bypass suppression. pipeline; never hand-edit public JSON to bypass suppression.
The next-test estimator is intentionally disabled. It must remain disabled The site intentionally contains no vehicle lookup, individual estimator,
until a separately reviewed and suppressed `prediction_lookup` contract is operational connection, or row-level prediction output. VINs, plates, ZIPs,
approved. Do not add a connection from this site to `countydata`. stations, technician identifiers, raw source JSON, credentials, and operational
records must never enter this directory.
## Import into Bolt ## Publish through Bolt
The recommended publication boundary is a separate static Bolt project (and, Use a separate static Bolt project backed by a dashboard-only repository. Do
ideally, a separate deployment repository) containing only the contents of not import this source repository into Bolt.
`dashboard/`. Upload or copy this directory's contents so `index.html` is at the
new project root. There is no install or build step. If Bolt asks for a preview
command, use `node server.mjs` with `HOST=0.0.0.0`; Bolt supplies `PORT`.
Before publishing, confirm that `/public/data/data_manifest.json` resolves, 1. Run `npm test` from this directory and confirm every contract check passes.
the development-preview banner remains visible, and the contract test passes. 2. Create a clean dashboard-only repository or Bolt project.
The generated JSON bundle must also complete its privacy review. 3. Copy only the *contents* of `dashboard/` into that project, so `index.html`
is at the project root.
4. Confirm that private `data/`, `artifacts/`, `models/`, `.env` files, SQL,
notebooks, and database tooling are absent.
5. Preview the project. There is no install or build step; if Bolt requests a
preview command, use `node server.mjs` with `HOST=0.0.0.0` and let Bolt
provide `PORT`.
6. Verify all four views, the development-sample messaging on every view, and
that the in-app status reads **Validated sample aggregates**.
7. Use **Publish** only after the aggregate bundle completes privacy review.
Bolt hosting is the default; Netlify can instead be selected before the
first publish if desired.
Importing the whole source repository is a discouraged fallback because Bolt See Bolt's official documentation for [Git
would receive SQL, database tooling, and other files that are not needed by the integration](https://support.bolt.new/integrations/git) and [publishing with
public site. If that has already happened, set the project/working root to Netlify](https://support.bolt.new/integrations/netlify).
`dashboard`, never serve the repository root, and create a dashboard-only
project before production publication. Do not copy private `data/`,
`artifacts/`, `models/`, `.env`, or database tooling into the public project.
No publishing action is performed by this repository. No publishing action is performed by this repository.

210
dashboard/index.html
View File

@ -3,8 +3,9 @@
<head> <head>
<meta charset="utf-8"> <meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Utah Vehicle Health — an accessible, aggregate view of emissions-inspection outcomes."> <meta name="description" content="Sample-based research on the first-attempt non-pass outcome of a returning vehicle's next emissions inspection.">
<meta name="referrer" content="no-referrer"> <meta name="referrer" content="no-referrer">
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'; font-src 'self'; object-src 'none'; base-uri 'none'; form-action 'none'">
<title>Overview · Utah Vehicle Health</title> <title>Overview · Utah Vehicle Health</title>
<link rel="stylesheet" href="./styles.css"> <link rel="stylesheet" href="./styles.css">
<script type="module" src="./js/app.js"></script> <script type="module" src="./js/app.js"></script>
@ -16,7 +17,7 @@
<span class="preview-banner__icon" aria-hidden="true">◆</span> <span class="preview-banner__icon" aria-hidden="true">◆</span>
<div> <div>
<strong id="preview-title">Development preview</strong> <strong id="preview-title">Development preview</strong>
<span id="preview-copy">Sample results are not population estimates and must not be used for individual decisions.</span> <span id="preview-copy">Private 10,000-vehicle development sample · suppressed aggregates only · not population estimates.</span>
</div> </div>
</div> </div>
@ -29,7 +30,7 @@
</svg> </svg>
<span> <span>
<span class="brand__name">Utah Vehicle Health</span> <span class="brand__name">Utah Vehicle Health</span>
<span class="brand__tagline">Next-episode inspection insights</span> <span class="brand__tagline">Returning vehicles · next first attempt</span>
</span> </span>
</a> </a>
@ -42,8 +43,8 @@
<nav class="primary-nav" aria-label="Dashboard views"> <nav class="primary-nav" aria-label="Dashboard views">
<div class="primary-nav__inner"> <div class="primary-nav__inner">
<a href="#overview" data-route="overview" aria-current="page">Overview</a> <a href="#overview" data-route="overview" aria-current="page">Overview</a>
<a href="#reliability" data-route="reliability">Reliability explorer</a> <a href="#cohorts" data-route="cohorts">Sample cohorts</a>
<a href="#estimator" data-route="estimator">Next-test estimator</a> <a href="#model" data-route="model">Model &amp; benchmark</a>
<a href="#methods" data-route="methods">Data &amp; methods</a> <a href="#methods" data-route="methods">Data &amp; methods</a>
</div> </div>
</nav> </nav>
@ -54,9 +55,9 @@
<div class="page-shell"> <div class="page-shell">
<header class="page-heading page-heading--split"> <header class="page-heading page-heading--split">
<div> <div>
<p class="eyebrow">Inspection outcomes over time</p> <p class="eyebrow">Private 10,000-vehicle development sample</p>
<h1 id="overview-heading">A clearer view of the next inspection</h1> <h1 id="overview-heading">The next inspection, in sample context</h1>
<p class="lede">Explore aggregate first-attempt outcomes for returning vehicles in participating Utah county feeds. Non-pass combines fail, reject, and abort.</p> <p class="lede">The research question is intentionally narrow: predict whether a returning vehicle's next inspection episode has a first-attempt non-pass. Non-pass combines fail, reject, and abort.</p>
</div> </div>
<div class="data-stamp" aria-label="Dataset status"> <div class="data-stamp" aria-label="Dataset status">
<span>Data through</span> <span>Data through</span>
@ -65,6 +66,11 @@
</div> </div>
</header> </header>
<aside class="scope-strip" aria-label="Result scope">
<strong>Sample-based only</strong>
<span>Every value on this page describes the private development sample and covered feeds—not Utah's vehicle population.</span>
</aside>
<div class="unavailable-state" data-unavailable hidden role="status"> <div class="unavailable-state" data-unavailable hidden role="status">
<div class="unavailable-state__icon" aria-hidden="true">!</div> <div class="unavailable-state__icon" aria-hidden="true">!</div>
<div> <div>
@ -75,22 +81,22 @@
<section class="kpi-grid" aria-label="Overview statistics" data-requires-data> <section class="kpi-grid" aria-label="Overview statistics" data-requires-data>
<article class="kpi-card"> <article class="kpi-card">
<span class="kpi-card__label">Published support</span> <span class="kpi-card__label">Published episode support</span>
<strong id="kpi-eligible">—</strong> <strong id="kpi-eligible">—</strong>
<small>Rounded total; suppressed cells omitted</small> <small>Rounded total; sampled vehicles may contribute multiple eligible episodes</small>
</article> </article>
<article class="kpi-card kpi-card--pass"> <article class="kpi-card kpi-card--pass">
<span class="kpi-card__label">Pass rate</span> <span class="kpi-card__label">Observed sample pass rate</span>
<strong id="kpi-pass">—</strong> <strong id="kpi-pass">—</strong>
<small>Recognized first-attempt outcomes</small> <small>Recognized first-attempt outcomes</small>
</article> </article>
<article class="kpi-card kpi-card--nonpass"> <article class="kpi-card kpi-card--nonpass">
<span class="kpi-card__label">Non-pass rate</span> <span class="kpi-card__label">Observed sample non-pass rate</span>
<strong id="kpi-nonpass">—</strong> <strong id="kpi-nonpass">—</strong>
<small>Fail + reject + abort</small> <small>Fail + reject + abort</small>
</article> </article>
<article class="kpi-card"> <article class="kpi-card">
<span class="kpi-card__label">Covered counties</span> <span class="kpi-card__label">Covered county feeds</span>
<strong id="kpi-counties">—</strong> <strong id="kpi-counties">—</strong>
<small>Not statewide coverage</small> <small>Not statewide coverage</small>
</article> </article>
@ -100,8 +106,8 @@
<article class="panel panel--wide"> <article class="panel panel--wide">
<header class="panel__header"> <header class="panel__header">
<div> <div>
<p class="eyebrow">Quarterly outcomes</p> <p class="eyebrow">Sample association over time</p>
<h2>First-attempt non-pass trend</h2> <h2>Observed first-attempt non-pass trend</h2>
</div> </div>
<div class="chart-legend" aria-label="Outcome legend"> <div class="chart-legend" aria-label="Outcome legend">
<span><i class="legend-swatch legend-swatch--fail"></i>Non-pass (fail + reject + abort)</span> <span><i class="legend-swatch legend-swatch--fail"></i>Non-pass (fail + reject + abort)</span>
@ -120,36 +126,41 @@
</header> </header>
<div id="utah-coverage-map" class="utah-map" role="img" aria-label="Utah county-feed coverage"></div> <div id="utah-coverage-map" class="utah-map" role="img" aria-label="Utah county-feed coverage"></div>
<div id="county-coverage-list" class="county-list" aria-label="County availability list"></div> <div id="county-coverage-list" class="county-list" aria-label="County availability list"></div>
<p class="chart-note">Teal counties appear in the approved inspection feeds. Gray counties are unavailable, not zero.</p> <p class="chart-note">Availability only—this is not a county outcome ranking. Teal feeds are represented in the sample; gray counties are unavailable, not zero.</p>
</article> </article>
<article class="panel"> <article class="panel">
<header class="panel__header"> <header class="panel__header">
<div> <div>
<p class="eyebrow">Age pattern</p> <p class="eyebrow">Sample association by age</p>
<h2>Non-pass risk by vehicle age</h2> <h2>Observed non-pass rate by vehicle age</h2>
</div> </div>
</header> </header>
<div id="age-risk-chart" class="chart" role="img" aria-label="Published non-pass risk by vehicle-age band"></div> <div id="age-risk-chart" class="chart" role="img" aria-label="Published non-pass risk by vehicle-age band"></div>
<p class="chart-note">The development bundle does not include confidence intervals; published aggregate rates are shown as points.</p> <p class="chart-note">Sample aggregates only, without confidence intervals. The pattern is descriptive and not a population estimate or causal effect.</p>
</article> </article>
</div> </div>
<aside class="callout" data-requires-data> <aside class="callout" data-requires-data>
<span class="callout__icon" aria-hidden="true">i</span> <span class="callout__icon" aria-hidden="true">i</span>
<p><strong>What this measures:</strong> emissions-inspection outcomes for covered feeds. It is not a diagnosis of mechanical condition, safety, roadworthiness, or legal compliance.</p> <p><strong>What this measures:</strong> sampled returning vehicles' next-episode first-attempt emissions-inspection outcomes in covered feeds. It is not a population estimate, mechanical diagnosis, safety assessment, or legal determination.</p>
</aside> </aside>
</div> </div>
</section> </section>
<section class="view" id="view-reliability" data-view="reliability" aria-labelledby="reliability-heading" hidden> <section class="view" id="view-cohorts" data-view="cohorts" aria-labelledby="cohorts-heading" hidden>
<div class="page-shell"> <div class="page-shell">
<header class="page-heading"> <header class="page-heading">
<p class="eyebrow">Supported aggregate cohorts</p> <p class="eyebrow">Private 10,000-vehicle development sample</p>
<h1 id="reliability-heading">Reliability explorer</h1> <h1 id="cohorts-heading">Sample cohort explorer</h1>
<p class="lede">Compare published observed next-episode non-pass rates. Every result is a supported aggregate cohort—not an individual prediction.</p> <p class="lede">Explore supported prior make-and-model cohorts and their observed next-episode first-attempt non-pass rates. These are sample associations—not reliability grades or individual predictions.</p>
</header> </header>
<aside class="scope-strip" aria-label="Result scope">
<strong>Not a ranking of population reliability</strong>
<span>Rates use rounded, suppressed episode support from sampled vehicles. Cohorts can differ in age, history, period, and coverage.</span>
</aside>
<div class="unavailable-state" data-unavailable hidden role="status"> <div class="unavailable-state" data-unavailable hidden role="status">
<div class="unavailable-state__icon" aria-hidden="true">!</div> <div class="unavailable-state__icon" aria-hidden="true">!</div>
<div><h2>Explorer unavailable</h2><p data-unavailable-message>Approved scorecards could not be validated.</p></div> <div><h2>Explorer unavailable</h2><p data-unavailable-message>Approved scorecards could not be validated.</p></div>
@ -158,7 +169,7 @@
<div class="explorer-layout" data-requires-data> <div class="explorer-layout" data-requires-data>
<aside class="filter-panel" aria-labelledby="filter-heading"> <aside class="filter-panel" aria-labelledby="filter-heading">
<div class="filter-panel__heading"> <div class="filter-panel__heading">
<h2 id="filter-heading">Refine cohorts</h2> <h2 id="filter-heading">Find a cohort</h2>
<button class="text-button" id="reset-filters" type="button">Reset</button> <button class="text-button" id="reset-filters" type="button">Reset</button>
</div> </div>
<form id="explorer-filters"> <form id="explorer-filters">
@ -166,31 +177,7 @@
<span>Make or model</span> <span>Make or model</span>
<input id="cohort-search" type="search" autocomplete="off" placeholder="Search supported cohorts"> <input id="cohort-search" type="search" autocomplete="off" placeholder="Search supported cohorts">
</label> </label>
<label class="field"> <p class="filter-panel__note"><strong>Published grain:</strong> prior make and model only. County, age, fuel, program, and period slices are not available for these scorecards.</p>
<span>County</span>
<select id="filter-county"><option value="">All covered counties</option></select>
</label>
<label class="field">
<span>Vehicle age</span>
<select id="filter-age"><option value="">All age bands</option></select>
</label>
<label class="field">
<span>Fuel</span>
<select id="filter-fuel"><option value="">All supported fuels</option></select>
</label>
<label class="field">
<span>Program</span>
<select id="filter-program"><option value="">All approved programs</option></select>
</label>
<label class="field">
<span>Period</span>
<select id="filter-period"><option value="">All published periods</option></select>
</label>
<fieldset class="segmented-control">
<legend>Risk view</legend>
<label><input type="radio" name="risk-view" value="observed" checked><span>Observed</span></label>
<label><input type="radio" name="risk-view" value="adjusted"><span>Model-adjusted</span></label>
</fieldset>
</form> </form>
</aside> </aside>
@ -201,8 +188,8 @@
<span>Sort</span> <span>Sort</span>
<select id="result-sort"> <select id="result-sort">
<option value="support">Largest support</option> <option value="support">Largest support</option>
<option value="risk-desc">Highest non-pass risk</option> <option value="risk-desc">Highest observed sample rate</option>
<option value="risk-asc">Lowest non-pass risk</option> <option value="risk-asc">Lowest observed sample rate</option>
<option value="name">Make and model</option> <option value="name">Make and model</option>
</select> </select>
</label> </label>
@ -210,9 +197,10 @@
<article class="panel"> <article class="panel">
<header class="panel__header"> <header class="panel__header">
<div><p class="eyebrow">Published aggregate comparison</p><h2>Supported cohort scorecard</h2></div> <div><p class="eyebrow">Descriptive sample comparison</p><h2>Observed cohort associations</h2></div>
</header> </header>
<div id="cohort-dot-plot" class="chart chart--scorecard" role="img" aria-label="Ranked published cohort non-pass rates"></div> <div id="cohort-dot-plot" class="chart chart--scorecard" role="img" aria-label="Observed development-sample cohort non-pass rates"></div>
<p class="chart-note">Ordering is a viewing aid within this sample, not a reliability ranking or population comparison.</p>
</article> </article>
<div id="cohort-cards" class="cohort-cards"></div> <div id="cohort-cards" class="cohort-cards"></div>
@ -225,51 +213,64 @@
</div> </div>
</section> </section>
<section class="view" id="view-estimator" data-view="estimator" aria-labelledby="estimator-heading" hidden> <section class="view" id="view-model" data-view="model" aria-labelledby="model-heading" hidden>
<div class="page-shell page-shell--narrow"> <div class="page-shell page-shell--narrow">
<header class="page-heading"> <header class="page-heading">
<p class="eyebrow">Cohort estimate · not an individual diagnosis</p> <p class="eyebrow">Private 10,000-vehicle development sample</p>
<h1 id="estimator-heading">Next-test risk estimator</h1> <h1 id="model-heading">Model &amp; benchmark</h1>
<p class="lede">This planned tool will combine approved, coarsened attributes to return a calibrated next-episode non-pass probability.</p> <p class="lede">Calibrated logistic regression is the final development-prototype model for the next-episode first-attempt non-pass target. The boosted tree is retained only as a benchmark.</p>
</header> </header>
<div class="locked-panel" role="status" aria-labelledby="estimator-status-title"> <aside class="scope-strip" aria-label="Result scope">
<span class="locked-panel__icon" aria-hidden="true">◇</span> <strong>Development evidence only</strong>
<div> <span>These 2024 values come from the partition used to fit Platt calibration. They are sample-based calibration checks—not independent final-performance estimates, population estimates, or locked-test results.</span>
<p class="eyebrow">Intentionally disabled</p> </aside>
<h2 id="estimator-status-title">An approved prediction lookup is not available</h2>
<p>The estimator will remain off until a privacy-reviewed, suppressed <code>prediction_lookup</code> is published. The current model diagnostics are not enough to serve individual or row-level estimates.</p> <div class="unavailable-state" data-unavailable hidden role="status">
</div> <div class="unavailable-state__icon" aria-hidden="true">!</div>
<div><h2>Model diagnostics unavailable</h2><p data-unavailable-message>The approved diagnostic summary could not be validated.</p></div>
</div> </div>
<form class="estimator-form" aria-describedby="estimator-disabled-copy"> <section class="model-comparison" data-requires-data aria-label="Development model comparison">
<fieldset disabled> <article class="model-card model-card--final">
<legend>Coarsened cohort attributes</legend> <div class="model-card__topline"><span class="decision-badge decision-badge--final">Final prototype</span><span>2024 calibration-fit check</span></div>
<div class="form-grid"> <h2>Calibrated logistic regression</h2>
<label class="field"><span>County</span><select><option>Select a covered county</option></select></label> <p>The selected model for the research conclusion and any prototype-level discussion. Platt scaling calibrates its probabilities.</p>
<label class="field"><span>Supported make and model</span><select><option>Select a cohort</option></select></label> <dl class="metric-list">
<label class="field"><span>Vehicle-age band</span><select><option>Select an age band</option></select></label> <div><dt>PR-AUC</dt><dd id="final-ap">—</dd></div>
<label class="field"><span>Fuel</span><select><option>Select a supported fuel</option></select></label> <div><dt>ROC-AUC</dt><dd id="final-roc">—</dd></div>
<label class="field"><span>Prior episode outcome</span><select><option>Pass / fail / reject / abort</option></select></label> <div><dt>Brier ↓</dt><dd id="final-brier">—</dd></div>
<label class="field"><span>Time since prior episode</span><select><option>Select a coarsened interval</option></select></label> </dl>
<label class="field"><span>Season</span><select><option>Select a season</option></select></label> </article>
<label class="field"><span>Program category</span><select><option>Select an approved program</option></select></label>
</div>
<button class="button" type="button">Estimate cohort risk</button>
</fieldset>
</form>
<p id="estimator-disabled-copy" class="form-note">No VIN, plate, exact address, station, free text, or current-test diagnostic will ever be requested.</p>
<section class="method-preview" aria-labelledby="future-output-heading"> <article class="model-card">
<div class="model-card__topline"><span class="decision-badge">Benchmark only</span><span>2024 calibration-fit check</span></div>
<h2>Calibrated boosted tree</h2>
<p>Retained for development comparison only. It does not replace the logistic model or define dashboard outputs.</p>
<dl class="metric-list">
<div><dt>PR-AUC</dt><dd id="benchmark-ap">—</dd></div>
<div><dt>ROC-AUC</dt><dd id="benchmark-roc">—</dd></div>
<div><dt>Brier ↓</dt><dd id="benchmark-brier">—</dd></div>
</dl>
</article>
</section>
<aside class="callout" data-requires-data>
<span class="callout__icon" aria-hidden="true">i</span>
<p><strong>Calibration diagnostic—not a final performance estimate:</strong> Platt scaling was fit on this same 2024 partition. PR-AUC and ROC-AUC summarize discrimination; lower Brier is better for probability error. Opened 2025 holdout metrics are intentionally not displayed here.</p>
</aside>
<section class="publication-boundary" aria-labelledby="publication-boundary-heading">
<div> <div>
<p class="eyebrow">Future approved output</p> <p class="eyebrow">Finished publication boundary</p>
<h2 id="future-output-heading">Designed for calibrated context</h2> <h2 id="publication-boundary-heading">No vehicle-level prediction service</h2>
<p>This static dashboard serves supported aggregate sample rates and diagnostic summaries only. It contains no vehicle lookup, personal inputs, operational connection, or row-level prediction output.</p>
</div> </div>
<ul class="check-list"> <ul class="check-list">
<li>Calibrated non-pass probability and uncertainty</li> <li>Returning-vehicle next-episode target only</li>
<li>Relevant aggregate baseline</li> <li>First attempt; pass versus non-pass</li>
<li>High-level factor contributions</li> <li>Final model: calibrated logistic regression</li>
<li>Support and coverage limitations</li> <li>Boosted tree: benchmark only</li>
</ul> </ul>
</section> </section>
</div> </div>
@ -278,11 +279,16 @@
<section class="view" id="view-methods" data-view="methods" aria-labelledby="methods-heading" hidden> <section class="view" id="view-methods" data-view="methods" aria-labelledby="methods-heading" hidden>
<div class="page-shell"> <div class="page-shell">
<header class="page-heading"> <header class="page-heading">
<p class="eyebrow">Transparent by design</p> <p class="eyebrow">Private 10,000-vehicle development sample</p>
<h1 id="methods-heading">Data &amp; methods</h1> <h1 id="methods-heading">Data &amp; methods</h1>
<p class="lede">How episodes, labels, temporal evaluation, coverage limits, and privacy controls shape every published result.</p> <p class="lede">How episodes, labels, temporal evaluation, coverage limits, and privacy controls shape every published result.</p>
</header> </header>
<aside class="scope-strip" aria-label="Result scope">
<strong>Sample-based, not population-based</strong>
<span>All displayed rates and diagnostics come from the development sample. County displays communicate feed coverage only.</span>
</aside>
<div class="unavailable-state" data-unavailable hidden role="status"> <div class="unavailable-state" data-unavailable hidden role="status">
<div class="unavailable-state__icon" aria-hidden="true">!</div> <div class="unavailable-state__icon" aria-hidden="true">!</div>
<div><h2>Live diagnostics unavailable</h2><p data-unavailable-message>Static methodology remains below; no performance values are being shown.</p></div> <div><h2>Live diagnostics unavailable</h2><p data-unavailable-message>Static methodology remains below; no performance values are being shown.</p></div>
@ -330,13 +336,13 @@
<div class="dashboard-grid" data-requires-data> <div class="dashboard-grid" data-requires-data>
<article class="panel"> <article class="panel">
<header class="panel__header"><div><p class="eyebrow" id="diagnostic-scope-label">Model diagnostics</p><h2>Published model diagnostics</h2></div></header> <header class="panel__header"><div><p class="eyebrow">Model governance</p><h2>Evidence status</h2></div></header>
<div class="diagnostic-kpis"> <ul class="limitation-list">
<div><span>PR-AUC</span><strong id="diagnostic-ap">—</strong></div> <li>Calibrated logistic regression is the declared final prototype model.</li>
<div><span>Brier score</span><strong id="diagnostic-brier">—</strong></div> <li>The boosted tree remains a benchmark only.</li>
<div><span>Support</span><strong id="diagnostic-n">—</strong></div> <li>Published diagnostics cover train, tune, and calibration partitions.</li>
</div> <li>No locked-test result or population performance claim is published.</li>
<div id="calibration-chart" class="chart" role="img" aria-label="Predicted versus observed non-pass risk calibration"></div> </ul>
</article> </article>
<article class="panel"> <article class="panel">
@ -344,7 +350,7 @@
<ul class="limitation-list" id="manifest-limitations"> <ul class="limitation-list" id="manifest-limitations">
<li>Participating inspection feeds do not cover all 29 counties.</li> <li>Participating inspection feeds do not cover all 29 counties.</li>
<li>Feed and program changes can resemble real-world trends.</li> <li>Feed and program changes can resemble real-world trends.</li>
<li>DMV history has a 2021 gap and ends in March 2024.</li> <li>The 10,000-vehicle draw retained 9,996 vehicles; page sampling can over-represent vehicles with more inspection records.</li>
<li>Reject and abort can reflect process or readiness issues.</li> <li>Reject and abort can reflect process or readiness issues.</li>
<li>Association is not causation or a mechanical diagnosis.</li> <li>Association is not causation or a mechanical diagnosis.</li>
</ul> </ul>
@ -358,9 +364,9 @@
<span aria-hidden="true">→</span> <span aria-hidden="true">→</span>
<span>Suppression<br><small>Minimum support + complements</small></span> <span>Suppression<br><small>Minimum support + complements</small></span>
<span aria-hidden="true">→</span> <span aria-hidden="true">→</span>
<span class="privacy-flow__public">Public assets<br><small>Aggregate JSON only</small></span> <span class="privacy-flow__public">Public assets<br><small>Suppressed summaries only</small></span>
</div> </div>
<p>Direct identifiers, private vehicle tokens, plates, ZIPs, stations, raw records, and row-level predictions are outside the public data contract.</p> <p>VINs, plates, ZIPs, stations, technician identifiers, private vehicle tokens, raw source JSON, credentials, operational records, and row-level predictions are outside this site and its public data contract.</p>
</section> </section>
</div> </div>
</section> </section>
@ -368,8 +374,8 @@
<footer class="site-footer"> <footer class="site-footer">
<div class="page-shell site-footer__inner"> <div class="page-shell site-footer__inner">
<div><strong>Utah Vehicle Health</strong><p>Aggregate emissions-inspection research.</p></div> <div><strong>Utah Vehicle Health</strong><p>Private 10,000-vehicle development sample.</p></div>
<p>Not a diagnosis, certification, safety assessment, or guarantee of an inspection outcome.</p> <p>Sample-based research only—not a population estimate, diagnosis, certification, safety assessment, or guarantee of an inspection outcome.</p>
</div> </div>
</footer> </footer>

107
dashboard/js/app.js
View File

@ -6,15 +6,14 @@ import {
renderAgeRisk, renderAgeRisk,
renderCohortDotPlot, renderCohortDotPlot,
renderCoverageHeatmap, renderCoverageHeatmap,
renderNoCalibration,
renderOutcomeTrend, renderOutcomeTrend,
renderUtahCoverage, renderUtahCoverage,
} from "./charts.js"; } from "./charts.js";
const ROUTES = Object.freeze({ const ROUTES = Object.freeze({
overview: "Overview", overview: "Overview",
reliability: "Reliability explorer", cohorts: "Sample cohorts",
estimator: "Next-test estimator", model: "Model & benchmark",
methods: "Data & methods", methods: "Data & methods",
}); });
@ -66,13 +65,6 @@ function normalizeCounty(value) {
return String(value).trim().toLowerCase().replaceAll("_", " ").replace(/\s+/g, " "); return String(value).trim().toLowerCase().replaceAll("_", " ").replace(/\s+/g, " ");
} }
function displayCategory(value) {
return String(value)
.replaceAll("_", " ")
.toLowerCase()
.replace(/\b\w/g, (letter) => letter.toUpperCase());
}
function routeFromHash() { function routeFromHash() {
const candidate = window.location.hash.replace(/^#/, "").toLowerCase(); const candidate = window.location.hash.replace(/^#/, "").toLowerCase();
return Object.hasOwn(ROUTES, candidate) ? candidate : "overview"; return Object.hasOwn(ROUTES, candidate) ? candidate : "overview";
@ -120,7 +112,7 @@ function setPreviewBanner(manifest) {
setText("preview-title", "Development preview"); setText("preview-title", "Development preview");
setText( setText(
"preview-copy", "preview-copy",
"These suppressed sample aggregates are not population estimates and must not be used for individual decisions.", "Private 10,000-vehicle development sample · suppressed aggregates only · not population estimates.",
); );
} }
@ -144,25 +136,6 @@ function setUnavailable(error) {
setHeaderStatus("error", "Approved data unavailable"); setHeaderStatus("error", "Approved data unavailable");
} }
function populateSelect(id, values, formatter = displayCategory) {
const select = byId(id);
if (!select) return;
for (const value of values) {
const option = document.createElement("option");
option.value = String(value);
option.textContent = formatter(value);
select.append(option);
}
}
function disableUnsupportedFilter(id, explanation) {
const select = byId(id);
if (!select) return;
select.disabled = true;
select.title = explanation;
select.options[0].textContent = explanation;
}
function renderCountyList(coveredCounties) { function renderCountyList(coveredCounties) {
const container = byId("county-coverage-list"); const container = byId("county-coverage-list");
container.replaceChildren(); container.replaceChildren();
@ -195,12 +168,12 @@ function renderOverview(data) {
setText("data-cutoff", latest ? `${latest.year} Q${latest.quarter}` : "Unavailable"); setText("data-cutoff", latest ? `${latest.year} Q${latest.quarter}` : "Unavailable");
setText( setText(
"model-version", "model-version",
`Release ${data.manifest.release_id.slice(0, 8)} · ${data.manifest.model_versions.join(" / ")}`, `Final: calibrated logistic · release ${data.manifest.release_id.slice(0, 8)}`,
); );
renderOutcomeTrend(byId("outcome-trend-chart"), rows); renderOutcomeTrend(byId("outcome-trend-chart"), rows);
setText( setText(
"outcome-trend-note", "outcome-trend-note",
"The current approved bundle publishes binary non-pass rates only. Four-class outcome mix and blank rates are not inferred or displayed.", "Development-sample aggregates only, not a population trend. The bundle publishes binary non-pass rates; four-class outcome mix and blank rates are not inferred.",
); );
renderAgeRisk(byId("age-risk-chart"), data.ageRisk.rows); renderAgeRisk(byId("age-risk-chart"), data.ageRisk.rows);
renderUtahCoverage(byId("utah-coverage-map"), counties); renderUtahCoverage(byId("utah-coverage-map"), counties);
@ -230,7 +203,7 @@ function renderScorecardCards(rows) {
heading.textContent = scorecardLabel(row); heading.textContent = scorecardLabel(row);
const meta = document.createElement("div"); const meta = document.createElement("div");
meta.className = "cohort-card__meta"; meta.className = "cohort-card__meta";
meta.textContent = `${compactNumber(row.support_rounded)} rounded support`; meta.textContent = `≈${compactNumber(row.support_rounded)} eligible sample episodes (rounded)`;
const bar = document.createElement("div"); const bar = document.createElement("div");
bar.className = "risk-bar"; bar.className = "risk-bar";
bar.setAttribute("aria-hidden", "true"); bar.setAttribute("aria-hidden", "true");
@ -242,7 +215,7 @@ function renderScorecardCards(rows) {
const strong = document.createElement("strong"); const strong = document.createElement("strong");
strong.textContent = percent(row.nonpass_rate); strong.textContent = percent(row.nonpass_rate);
const small = document.createElement("small"); const small = document.createElement("small");
small.textContent = "Observed non-pass"; small.textContent = "Observed sample non-pass";
value.append(strong, small); value.append(strong, small);
article.append(heading, meta, bar, value); article.append(heading, meta, bar, value);
container.append(article); container.append(article);
@ -259,33 +232,14 @@ function updateExplorer() {
visibleScorecards = sortedScorecards(matches, sortMode); visibleScorecards = sortedScorecards(matches, sortMode);
setText( setText(
"result-summary", "result-summary",
`${visibleScorecards.length} supported cohort${visibleScorecards.length === 1 ? "" : "s"}; showing up to 18 cards and 12 chart rows.`, `${visibleScorecards.length} supported sample cohort${visibleScorecards.length === 1 ? "" : "s"}; showing up to 18 cards and 12 chart rows.`,
); );
byId("empty-results").hidden = visibleScorecards.length > 0; byId("empty-results").hidden = visibleScorecards.length > 0;
renderCohortDotPlot(byId("cohort-dot-plot"), visibleScorecards); renderCohortDotPlot(byId("cohort-dot-plot"), visibleScorecards);
renderScorecardCards(visibleScorecards); renderScorecardCards(visibleScorecards);
} }
function initializeExplorer(data) { function initializeExplorer() {
populateSelect("filter-county", data.filters.public_counties);
populateSelect("filter-age", data.filters.age_bands, (value) => String(value));
populateSelect(
"filter-period",
data.filters.periods.map((period) => period.year),
(value) => String(value),
);
// Current scorecards are make/model aggregates only. These planned controls
// remain visible but disabled so the UI never implies unsupported slicing.
disableUnsupportedFilter("filter-county", "Unavailable at current scorecard grain");
disableUnsupportedFilter("filter-age", "Unavailable at current scorecard grain");
disableUnsupportedFilter("filter-fuel", "Fuel not published in this bundle");
disableUnsupportedFilter("filter-program", "Program not published in this bundle");
disableUnsupportedFilter("filter-period", "Period not published in this scorecard");
const adjusted = document.querySelector('input[name="risk-view"][value="adjusted"]');
adjusted.disabled = true;
adjusted.parentElement.title = "Model-adjusted cohort scorecards are not published.";
byId("cohort-search").addEventListener("input", updateExplorer); byId("cohort-search").addEventListener("input", updateExplorer);
byId("result-sort").addEventListener("change", updateExplorer); byId("result-sort").addEventListener("change", updateExplorer);
byId("reset-filters").addEventListener("click", () => { byId("reset-filters").addEventListener("click", () => {
@ -297,42 +251,37 @@ function initializeExplorer(data) {
updateExplorer(); updateExplorer();
} }
function chooseDiagnostic(rows) { function findDiagnostic(rows, model) {
const partitionRank = { calibrate: 3, tune: 2, train: 1 }; return rows.find((row) => row.model === model && row.partition === "calibrate");
return [...rows].sort((left, right) => { }
const preferredLeft = left.model === "logistic_platt" ? 1 : 0;
const preferredRight = right.model === "logistic_platt" ? 1 : 0; function renderModel(data) {
return ( const finalModel = findDiagnostic(data.diagnostics.rows, "logistic_platt");
preferredRight - preferredLeft || const benchmark = findDiagnostic(data.diagnostics.rows, "hist_gradient_boosting_platt");
(partitionRank[right.partition] ?? 0) - (partitionRank[left.partition] ?? 0) const metric = (row, key) => (row ? row[key].toFixed(3) : "—");
);
})[0]; setText("final-ap", metric(finalModel, "average_precision"));
setText("final-roc", metric(finalModel, "roc_auc"));
setText("final-brier", metric(finalModel, "brier"));
setText("benchmark-ap", metric(benchmark, "average_precision"));
setText("benchmark-roc", metric(benchmark, "roc_auc"));
setText("benchmark-brier", metric(benchmark, "brier"));
} }
function renderMethods(data) { function renderMethods(data) {
renderCoverageHeatmap(byId("coverage-heatmap"), data.coverage.rows); renderCoverageHeatmap(byId("coverage-heatmap"), data.coverage.rows);
const diagnostic = chooseDiagnostic(data.diagnostics.rows);
setText(
"diagnostic-scope-label",
diagnostic
? `${diagnostic.partition === "calibrate" ? "Calibration cohort" : "Development diagnostics"} · ${diagnostic.model}`
: "Development diagnostics · model unavailable",
);
setText("diagnostic-ap", diagnostic ? diagnostic.average_precision.toFixed(3) : "—");
setText("diagnostic-brier", diagnostic ? diagnostic.brier.toFixed(3) : "—");
setText("diagnostic-n", "Not published");
renderNoCalibration(byId("calibration-chart"));
} }
function renderDashboard(data) { function renderDashboard(data) {
dashboardData = data; dashboardData = data;
setPreviewBanner(data.manifest); setPreviewBanner(data.manifest);
renderOverview(data); renderOverview(data);
initializeExplorer(data); initializeExplorer();
renderModel(data);
renderMethods(data); renderMethods(data);
setHeaderStatus( setHeaderStatus(
"ready", "ready",
data.manifest.development_preview ? "Validated development aggregates" : "Validated public aggregates", data.manifest.development_preview ? "Validated sample aggregates" : "Validated public aggregates",
); );
} }
@ -349,4 +298,4 @@ async function initialize() {
initialize(); initialize();
export { displayCategory, normalizeCounty, routeFromHash }; export { normalizeCounty, routeFromHash };

17
dashboard/js/charts.js
View File

@ -109,7 +109,7 @@ export function renderOutcomeTrend(container, rows) {
</svg>`; </svg>`;
container.setAttribute( container.setAttribute(
"aria-label", "aria-label",
`Quarterly aggregate non-pass trend. ${description}`, `Development-sample quarterly aggregate non-pass rates, not population estimates. ${description}`,
); );
} }
@ -155,7 +155,7 @@ export function renderAgeRisk(container, rows) {
</svg>`; </svg>`;
container.setAttribute( container.setAttribute(
"aria-label", "aria-label",
`Non-pass risk by vehicle-age band. ${points `Observed development-sample non-pass rates by vehicle-age band, not population estimates. ${points
.map((point) => `${point.age_band}: ${percent(point.nonpass_rate)}`) .map((point) => `${point.age_band}: ${percent(point.nonpass_rate)}`)
.join("; ")}`, .join("; ")}`,
); );
@ -199,7 +199,7 @@ export function renderCohortDotPlot(container, rows) {
</svg>`; </svg>`;
container.setAttribute( container.setAttribute(
"aria-label", "aria-label",
`Ranked supported cohort non-pass risk. ${points `Supported development-sample cohort associations, not reliability rankings or population estimates. ${points
.map((point) => `${point.prior_make} ${point.prior_model}: ${percent(point.nonpass_rate)}`) .map((point) => `${point.prior_make} ${point.prior_model}: ${percent(point.nonpass_rate)}`)
.join("; ")}`, .join("; ")}`,
); );
@ -234,7 +234,7 @@ export function renderUtahCoverage(container, coveredCounties) {
container.setAttribute( container.setAttribute(
"aria-label", "aria-label",
coveredCounties.length coveredCounties.length
? `Utah feed coverage includes ${coveredCounties.join(", ")}. Other counties are unavailable.` ? `Development-sample feed availability includes ${coveredCounties.join(", ")}. This is not an outcome ranking; other counties are unavailable.`
: "No county feed coverage is available.", : "No county feed coverage is available.",
); );
} }
@ -267,14 +267,7 @@ export function renderCoverageHeatmap(container, rows) {
</div>`; </div>`;
container.setAttribute( container.setAttribute(
"aria-label", "aria-label",
`Source-era coverage from ${years[0]} through ${years[years.length - 1]} for ${eras.join(", ")}.`, `Development-sample source-era coverage from ${years[0]} through ${years[years.length - 1]} for ${eras.join(", ")}.`,
);
}
export function renderNoCalibration(container) {
emptyChart(
container,
"Calibration-bin data is not published in this development bundle. No calibration curve is shown.",
); );
} }

30
dashboard/js/data.js
View File

@ -1,5 +1,10 @@
const SCHEMA_VERSION = "dashboard_data_v1"; const SCHEMA_VERSION = "dashboard_data_v1";
const APPROVED_PARTITIONS = Object.freeze(["train", "tune", "calibrate"]); const APPROVED_PARTITIONS = Object.freeze(["train", "tune", "calibrate"]);
const APPROVED_TARGET = "first-attempt next-episode binary non-pass rate";
const REQUIRED_CALIBRATION_MODELS = Object.freeze([
"hist_gradient_boosting_platt",
"logistic_platt",
]);
const ENVELOPE_KEYS = Object.freeze([ const ENVELOPE_KEYS = Object.freeze([
"development_preview", "development_preview",
"population_estimate_allowed", "population_estimate_allowed",
@ -20,15 +25,22 @@ export const REQUIRED_ASSETS = Object.freeze({
const SENSITIVE_KEY_PARTS = new Set([ const SENSITIVE_KEY_PARTS = new Set([
"address", "address",
"certificate", "certificate",
"credential",
"email", "email",
"inspector",
"internal", "internal",
"ip", "ip",
"owner", "owner",
"password",
"pid", "pid",
"plate", "plate",
"prediction",
"probability",
"raw", "raw",
"secret",
"session", "session",
"station", "station",
"technician",
"token", "token",
"user", "user",
"vin", "vin",
@ -337,6 +349,11 @@ export function validateAssetSet(rawAssets) {
requireExactKeys(rawAssets, Object.keys(REQUIRED_ASSETS), "asset set"); requireExactKeys(rawAssets, Object.keys(REQUIRED_ASSETS), "asset set");
const manifest = rawAssets.manifest; const manifest = rawAssets.manifest;
validateEnvelope(manifest, "data_manifest"); validateEnvelope(manifest, "data_manifest");
if (!manifest.development_preview || manifest.population_estimate_allowed) {
throw new DataContractError(
"The dashboard accepts development-sample, non-population assets only.",
);
}
requireExactKeys( requireExactKeys(
manifest, manifest,
[ [
@ -427,6 +444,9 @@ export function validateAssetSet(rawAssets) {
requireInteger(manifest.definitions[key], `data_manifest.definitions.${key}`, { min: 1 }); requireInteger(manifest.definitions[key], `data_manifest.definitions.${key}`, { min: 1 });
} }
requireString(manifest.definitions.target, "data_manifest.definitions.target"); requireString(manifest.definitions.target, "data_manifest.definitions.target");
if (manifest.definitions.target !== APPROVED_TARGET) {
throw new DataContractError("data_manifest.definitions.target is outside the approved scope.");
}
if (!Array.isArray(manifest.assets)) { if (!Array.isArray(manifest.assets)) {
throw new DataContractError("data_manifest.assets must be an array."); throw new DataContractError("data_manifest.assets must be an array.");
} }
@ -463,6 +483,16 @@ export function validateAssetSet(rawAssets) {
throw new DataContractError(`model_diagnostics.rows[${index}].model is not cataloged.`); throw new DataContractError(`model_diagnostics.rows[${index}].model is not cataloged.`);
} }
} }
for (const model of REQUIRED_CALIBRATION_MODELS) {
const matches = validated.diagnostics.rows.filter(
(row) => row.model === model && row.partition === "calibrate",
);
if (matches.length !== 1) {
throw new DataContractError(
`model_diagnostics must contain exactly one approved calibration row for ${model}.`,
);
}
}
const minimumSupport = manifest.definitions.suppression_min_support; const minimumSupport = manifest.definitions.suppression_min_support;
const supportRounding = manifest.definitions.support_rounding; const supportRounding = manifest.definitions.support_rounding;
for (const name of ["overview", "ageRisk", "scorecards", "coverage"]) { for (const name of ["overview", "ageRisk", "scorecards", "coverage"]) {

211
dashboard/styles.css
View File

@ -37,12 +37,16 @@
} }
html { html {
max-width: 100%;
overflow-x: clip;
scroll-behavior: smooth; scroll-behavior: smooth;
} }
body { body {
min-width: 320px; min-width: 320px;
max-width: 100%;
margin: 0; margin: 0;
overflow-x: clip;
color: var(--ink-950); color: var(--ink-950);
background: background:
radial-gradient(circle at 7% 2%, rgb(231 216 185 / 35%), transparent 30rem), radial-gradient(circle at 7% 2%, rgb(231 216 185 / 35%), transparent 30rem),
@ -165,6 +169,11 @@ h2 {
margin-right: 0.35rem; margin-right: 0.35rem;
} }
.preview-banner > div {
min-width: 0;
overflow-wrap: anywhere;
}
.preview-banner__icon { .preview-banner__icon {
color: var(--amber-700); color: var(--amber-700);
} }
@ -182,7 +191,8 @@ h2 {
.site-header__inner, .site-header__inner,
.primary-nav__inner, .primary-nav__inner,
.page-shell { .page-shell {
width: min(calc(100% - 2rem), var(--content)); width: calc(100% - 2rem);
max-width: var(--content);
margin-inline: auto; margin-inline: auto;
} }
@ -302,10 +312,15 @@ h2 {
} }
main { main {
width: 100%;
min-width: 0;
max-width: 100%;
min-height: 65vh; min-height: 65vh;
} }
.view { .view {
min-width: 0;
max-width: 100%;
padding: clamp(2.5rem, 6vw, 5rem) 0 5rem; padding: clamp(2.5rem, 6vw, 5rem) 0 5rem;
} }
@ -318,9 +333,17 @@ main {
} }
.page-heading { .page-heading {
min-width: 0;
margin-bottom: clamp(2rem, 5vw, 3.4rem); margin-bottom: clamp(2rem, 5vw, 3.4rem);
} }
.page-heading > *,
.dashboard-grid > *,
.kpi-grid > *,
.model-comparison > * {
min-width: 0;
}
.page-heading--split { .page-heading--split {
display: flex; display: flex;
align-items: end; align-items: end;
@ -370,6 +393,26 @@ main {
font-size: 1.04rem; font-size: 1.04rem;
} }
.scope-strip {
display: flex;
align-items: baseline;
gap: 0.7rem 1rem;
margin: -1.4rem 0 1rem;
padding: 0.75rem 1rem;
border: 1px solid #e9c77d;
border-radius: var(--radius-sm);
color: #503407;
background: rgb(249 236 205 / 72%);
font-size: 0.78rem;
}
.scope-strip strong {
flex: 0 0 auto;
font-size: 0.72rem;
letter-spacing: 0.06em;
text-transform: uppercase;
}
.unavailable-state { .unavailable-state {
display: flex; display: flex;
align-items: flex-start; align-items: flex-start;
@ -694,6 +737,14 @@ main {
margin: 0; margin: 0;
} }
.filter-panel__note {
margin: 1rem 0 0;
padding-top: 1rem;
border-top: 1px solid var(--sand-200);
color: var(--ink-650);
font-size: 0.75rem;
}
.text-button { .text-button {
padding: 0.2rem; padding: 0.2rem;
border: 0; border: 0;
@ -868,6 +919,124 @@ fieldset:disabled .field select {
text-align: center; text-align: center;
} }
.model-comparison {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 1rem;
}
.model-card {
position: relative;
padding: clamp(1.25rem, 3vw, 1.7rem);
overflow: hidden;
border: 1px solid var(--sand-200);
border-radius: var(--radius-lg);
background: rgb(255 255 255 / 86%);
box-shadow: var(--shadow-sm);
}
.model-card::before {
position: absolute;
top: 0;
right: 0;
left: 0;
height: 4px;
background: var(--sand-400);
content: "";
}
.model-card--final {
border-color: #9bc9c6;
background: linear-gradient(145deg, rgb(220 239 238 / 58%), rgb(255 255 255 / 92%));
box-shadow: var(--shadow-md);
}
.model-card--final::before {
background: var(--teal-700);
}
.model-card__topline {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.75rem;
margin-bottom: 1.2rem;
color: var(--ink-500);
font-size: 0.68rem;
}
.decision-badge {
padding: 0.25rem 0.55rem;
border: 1px solid var(--sand-400);
border-radius: 999px;
color: #6a4c18;
background: var(--amber-100);
font-size: 0.64rem;
font-weight: 800;
letter-spacing: 0.06em;
text-transform: uppercase;
}
.decision-badge--final {
border-color: #83bbb7;
color: var(--teal-800);
background: var(--teal-100);
}
.model-card > p {
min-height: 5.1em;
margin-bottom: 1.2rem;
color: var(--ink-650);
font-size: 0.82rem;
}
.metric-list {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 0.5rem;
margin: 0;
}
.metric-list div {
padding: 0.72rem;
border-radius: var(--radius-sm);
background: rgb(237 240 237 / 78%);
}
.metric-list dt {
color: var(--ink-500);
font-size: 0.62rem;
font-weight: 700;
}
.metric-list dd {
margin: 0.18rem 0 0;
font-family: var(--font-display);
font-size: 1.35rem;
}
.publication-boundary {
display: grid;
grid-template-columns: 1.15fr 0.85fr;
gap: 2rem;
align-items: start;
margin-top: 1rem;
padding: clamp(1.4rem, 3vw, 1.9rem);
border-radius: var(--radius-lg);
color: var(--white);
background: var(--ink-950);
}
.publication-boundary p:last-child {
margin: 0;
color: #cad7d5;
font-size: 0.82rem;
}
.publication-boundary .eyebrow {
color: #9bd4d1;
}
.locked-panel { .locked-panel {
display: flex; display: flex;
gap: 1.2rem; gap: 1.2rem;
@ -1242,6 +1411,8 @@ fieldset:disabled .field select {
.dashboard-grid--wide, .dashboard-grid--wide,
.explorer-layout, .explorer-layout,
.split-section, .split-section,
.model-comparison,
.publication-boundary,
.privacy-panel { .privacy-panel {
grid-template-columns: 1fr; grid-template-columns: 1fr;
} }
@ -1260,10 +1431,26 @@ fieldset:disabled .field select {
} }
@media (max-width: 660px) { @media (max-width: 660px) {
.preview-banner {
align-items: flex-start;
justify-content: flex-start;
text-align: left;
}
.preview-banner__icon {
flex: 0 0 auto;
}
.site-header__inner { .site-header__inner {
min-height: 68px; min-height: 68px;
} }
.page-shell {
width: auto;
max-width: calc(100% - 2rem);
margin-inline: 1rem;
}
.header-status, .header-status,
.brand__tagline { .brand__tagline {
display: none; display: none;
@ -1286,6 +1473,21 @@ fieldset:disabled .field select {
display: block; display: block;
} }
.scope-strip {
align-items: flex-start;
flex-direction: column;
}
.lede,
.data-stamp,
.scope-strip,
.kpi-card,
.model-card,
.panel {
max-width: 100%;
overflow-wrap: anywhere;
}
.data-stamp { .data-stamp {
width: 100%; width: 100%;
margin-top: 1.3rem; margin-top: 1.3rem;
@ -1296,10 +1498,15 @@ fieldset:disabled .field select {
.method-grid, .method-grid,
.cohort-cards, .cohort-cards,
.form-grid, .form-grid,
.method-preview { .method-preview,
.metric-list {
grid-template-columns: 1fr; grid-template-columns: 1fr;
} }
.model-card > p {
min-height: 0;
}
.kpi-card { .kpi-card {
min-height: 130px; min-height: 130px;
} }

122
dashboard/tests/contract.test.mjs
View File

@ -52,13 +52,25 @@ function filesRecursively(directory) {
}); });
} }
function viewSource(html, route) {
const marker = `<section class="view" id="view-${route}"`;
const start = html.indexOf(marker);
if (start < 0) return "";
const remaining = html.slice(start + marker.length);
const nextView = remaining.indexOf('<section class="view" id="view-');
const mainEnd = remaining.indexOf("</main>");
const candidates = [nextView, mainEnd].filter((index) => index >= 0);
const end = candidates.length ? Math.min(...candidates) : remaining.length;
return html.slice(start, start + marker.length + end);
}
test("static shell exposes four semantic navigable views", () => { test("static shell exposes four semantic navigable views", () => {
const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8"); const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8");
assert.match(html, /<header\b/); assert.match(html, /<header\b/);
assert.match(html, /<nav\b[^>]*aria-label="Dashboard views"/); assert.match(html, /<nav\b[^>]*aria-label="Dashboard views"/);
assert.match(html, /<main\b/); assert.match(html, /<main\b/);
assert.match(html, /<footer\b/); assert.match(html, /<footer\b/);
for (const route of ["overview", "reliability", "estimator", "methods"]) { for (const route of ["overview", "cohorts", "model", "methods"]) {
assert.match(html, new RegExp(`data-route="${route}"`)); assert.match(html, new RegExp(`data-route="${route}"`));
assert.match(html, new RegExp(`data-view="${route}"`)); assert.match(html, new RegExp(`data-view="${route}"`));
} }
@ -67,20 +79,45 @@ test("static shell exposes four semantic navigable views", () => {
assert.doesNotMatch(html, /<strong>Locked test<\/strong>|<small>Final evaluation<\/small>/); assert.doesNotMatch(html, /<strong>Locked test<\/strong>|<small>Final evaluation<\/small>/);
}); });
test("estimator remains disabled and contains no identifying input", () => { test("narrow layouts constrain body content while preserving local nav scrolling", () => {
const css = readFileSync(path.join(DASHBOARD_ROOT, "styles.css"), "utf8");
assert.match(css, /html\s*{[^}]*max-width:\s*100%;[^}]*overflow-x:\s*clip;/s);
assert.match(css, /body\s*{[^}]*max-width:\s*100%;[^}]*overflow-x:\s*clip;/s);
assert.match(css, /\.primary-nav__inner\s*{[^}]*overflow-x:\s*auto;/s);
const mobile = css.match(/@media \(max-width: 660px\)\s*{[\s\S]*?(?=@media|$)/)?.[0];
assert.ok(mobile, "mobile breakpoint is present");
assert.match(mobile, /\.page-shell\s*{[^}]*width:\s*auto;[^}]*max-width:\s*calc\(100% - 2rem\);/s);
assert.match(mobile, /\.preview-banner\s*{[^}]*justify-content:\s*flex-start;/s);
assert.match(mobile, /overflow-wrap:\s*anywhere;/);
});
test("model view declares the final model, benchmark role, and publication boundary", () => {
const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8"); const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8");
const estimator = html.match(/<section class="view" id="view-estimator"[\s\S]*?<\/section>/)?.[0]; const modelView = viewSource(html, "model");
assert.ok(estimator, "estimator section is present"); assert.ok(modelView, "model section is present");
assert.match(estimator, /<fieldset disabled>/); assert.match(modelView, /Calibrated logistic regression/);
const controlAttributes = [...estimator.matchAll(/<(?:input|select|textarea)\b([^>]*)>/g)].map( assert.match(modelView, /Final prototype/);
(match) => match[1], assert.match(modelView, /Calibrated boosted tree/);
); assert.match(modelView, /Benchmark only/);
for (const attributes of controlAttributes) { assert.match(modelView, /partition used to fit Platt calibration/i);
assert.doesNotMatch( assert.match(modelView, /not independent final-performance estimates/i);
attributes, assert.match(modelView, /No vehicle-level prediction service/);
/(?:name|id)\s*=\s*["'][^"']*(?:vin|plate|address|station|free.?text|zip)[^"']*["']/i, assert.doesNotMatch(modelView, /<(?:input|select|textarea|form)\b/i);
); assert.doesNotMatch(modelView, /prediction_lookup|planned tool|future approved output/i);
});
test("every result-facing view visibly labels the sample and non-population scope", () => {
const html = readFileSync(path.join(DASHBOARD_ROOT, "index.html"), "utf8");
assert.match(html, /Private 10,000-vehicle development sample/);
for (const route of ["overview", "cohorts", "model", "methods"]) {
const view = viewSource(html, route);
assert.ok(view, `${route} view is present`);
assert.match(view, /scope-strip/);
assert.match(view, /sample/i);
assert.match(view, /not[^.]{0,80}population|population estimate/i);
} }
assert.match(html, /not a county outcome ranking/i);
assert.match(html, /not a reliability ranking or population comparison/i);
}); });
test("all required generated assets satisfy the browser contract", () => { test("all required generated assets satisfy the browser contract", () => {
@ -91,6 +128,12 @@ test("all required generated assets satisfy the browser contract", () => {
assert.equal(validated.manifest.definitions.locked_test_metrics_published, false); assert.equal(validated.manifest.definitions.locked_test_metrics_published, false);
assert.match(validated.manifest.release_id, /^[0-9a-f]{64}$/); assert.match(validated.manifest.release_id, /^[0-9a-f]{64}$/);
assert.ok(validated.manifest.model_versions.length > 0); assert.ok(validated.manifest.model_versions.length > 0);
assert.ok(validated.diagnostics.rows.some(
(row) => row.model === "logistic_platt" && row.partition === "calibrate",
));
assert.ok(validated.diagnostics.rows.some(
(row) => row.model === "hist_gradient_boosting_platt" && row.partition === "calibrate",
));
assert.deepEqual( assert.deepEqual(
validated.filters.partitions, validated.filters.partitions,
["train", "tune", "calibrate"], ["train", "tune", "calibrate"],
@ -151,10 +194,37 @@ test("contract fails closed for missing, inconsistent, or sensitive data", () =>
inconsistent.ageRisk.population_estimate_allowed = true; inconsistent.ageRisk.population_estimate_allowed = true;
assert.throws(() => validateAssetSet(inconsistent), DataContractError); assert.throws(() => validateAssetSet(inconsistent), DataContractError);
const populationClaim = assetSet();
populationClaim.manifest.population_estimate_allowed = true;
assert.throws(() => validateAssetSet(populationClaim), DataContractError);
const broadenedTarget = assetSet();
broadenedTarget.manifest.definitions.target = "any inspection outcome";
assert.throws(() => validateAssetSet(broadenedTarget), DataContractError);
const sensitive = assetSet(); const sensitive = assetSet();
sensitive.scorecards.rows[0].vehicle_token = "not-public"; sensitive.scorecards.rows[0].vehicle_token = "not-public";
assert.throws(() => validateAssetSet(sensitive), DataContractError); assert.throws(() => validateAssetSet(sensitive), DataContractError);
for (const deniedField of [
"vin",
"plate",
"zip",
"station",
"technician_id",
"inspector_id",
"row_prediction",
"probability",
"raw_json",
"credential",
"password",
"secret",
]) {
const denied = assetSet();
denied.overview.rows[0][deniedField] = "not-public";
assert.throws(() => validateAssetSet(denied), DataContractError, deniedField);
}
const extraRowField = assetSet(); const extraRowField = assetSet();
extraRowField.overview.rows[0].note = "unapproved"; extraRowField.overview.rows[0].note = "unapproved";
assert.throws(() => validateAssetSet(extraRowField), DataContractError); assert.throws(() => validateAssetSet(extraRowField), DataContractError);
@ -167,6 +237,20 @@ test("contract fails closed for missing, inconsistent, or sensitive data", () =>
extraPartition.filters.partitions.push("locked_test"); extraPartition.filters.partitions.push("locked_test");
assert.throws(() => validateAssetSet(extraPartition), DataContractError); assert.throws(() => validateAssetSet(extraPartition), DataContractError);
const missingFinalDiagnostic = assetSet();
missingFinalDiagnostic.diagnostics.rows = missingFinalDiagnostic.diagnostics.rows.filter(
(row) => !(row.model === "logistic_platt" && row.partition === "calibrate"),
);
assert.throws(() => validateAssetSet(missingFinalDiagnostic), DataContractError);
const duplicatedBenchmarkDiagnostic = assetSet();
duplicatedBenchmarkDiagnostic.diagnostics.rows.push({
...duplicatedBenchmarkDiagnostic.diagnostics.rows.find(
(row) => row.model === "hist_gradient_boosting_platt" && row.partition === "calibrate",
),
});
assert.throws(() => validateAssetSet(duplicatedBenchmarkDiagnostic), DataContractError);
const unsortedVersions = assetSet(); const unsortedVersions = assetSet();
unsortedVersions.manifest.model_versions = ["z_v1", "a_v1"]; unsortedVersions.manifest.model_versions = ["z_v1", "a_v1"];
assert.throws(() => validateAssetSet(unsortedVersions), DataContractError); assert.throws(() => validateAssetSet(unsortedVersions), DataContractError);
@ -194,3 +278,15 @@ test("dashboard source has no external runtime dependency or credential marker",
assert.doesNotMatch(source, /<script[^>]+src=["']https?:|@import\s+url\(["']?https?:/i); assert.doesNotMatch(source, /<script[^>]+src=["']https?:|@import\s+url\(["']?https?:/i);
} }
}); });
test("public data directory contains only the approved suppressed aggregate bundle", () => {
const expected = Object.values(REQUIRED_ASSETS).sort();
const observed = readdirSync(PUBLIC_DATA).sort();
assert.deepEqual(observed, expected);
const serialized = observed
.map((filename) => readFileSync(path.join(PUBLIC_DATA, filename), "utf8"))
.join("\n");
assert.doesNotMatch(serialized, /\b[A-HJ-NPR-Z0-9]{17}\b/);
assert.doesNotMatch(serialized, /(?:postgres(?:ql)?:\/\/|PGPASSWORD|BEGIN [A-Z ]*PRIVATE KEY)/i);
});

179
docs/architecture.mmd
View File

@ -1,115 +1,84 @@
graph TD flowchart TD
%% Data Sources SOURCE[(Private inspection histories<br/>read-only source)]
PG[(PostgreSQL<br/>countydata)] SAMPLE[Private page-sampled histories<br/>up to 10,000 vehicles<br/>keyed vehicle tokens]
DMV[("DMV Data<br/>~18M records<br/>2011-2024")] SOURCE -->|read-only bounded export| SAMPLE
INSP[("Inspection Data<br/>~19.3M records<br/>2010-2026")]
PG --- DMV SAMPLE --> STAGE[Normalize labels<br/>deduplicate and validate]
PG --- INSP STAGE --> EPISODES[Build 30-day episodes<br/>first attempt is the target]
EPISODES --> MART[Point-in-time feature mart<br/>returning vehicles only]
%% Extraction subgraph SPLITS[Private development chronology]
PG -->|"export_history_sample.py<br/>(VIN hashing, page sampling)"| CSV["history_sample_10000.csv.gz<br/>+ manifest"]
%% Feature Mart Build
CSV -->|"build_feature_mart.py"| DUCKDB
subgraph DuckDB["DuckDB Local Warehouse"]
direction TB
SQL20["20_stage_events.sql<br/>Normalize & validate"]
SQL21["21_build_episodes.sql<br/>Group episodes (30-day gap)"]
SQL22["22_build_features.sql<br/>Point-in-time feature windows"]
SQL20 --> SQL21 --> SQL22
end
DUCKDB --> MART["inspection_feature_mart.parquet<br/>(episode × features, temporal partitions)"]
%% Temporal Partitions
subgraph Partitions["Temporal Splits"]
TRAIN["Train: 2016–2022"]
TUNE["Tune: 2023"]
CAL["Calibrate: 2024"]
TEST["Locked Test: 2025"]
DRIFT["Shadow Drift: 2026"]
end
MART -.- Partitions
%% Model Training
MART -->|"train_baselines.py"| BL["artifacts/baselines/baseline_v1/<br/>Logistic Regression +<br/>Prevalence & Prior-Outcome"]
MART -->|"train_tree_model.py"| TREE["artifacts/tree/hist_gradient_boosting_v1/<br/>Histogram Gradient Boosting"]
%% Model Outputs
BL --> ARTIFACTS["Model Artifacts<br/>manifest.json, metrics.json,<br/>model.joblib, calibration_bins.csv"]
TREE --> ARTIFACTS
%% Dashboard Export
MART -->|"export_dashboard_data.py<br/>(privacy suppression, aggregation)"| JSON
subgraph JSON["dashboard/public/data/"]
direction TB
DM["data_manifest.json"]
OV["overview_period_county.json"]
CS["cohort_scorecard.json"]
ARC["age_risk_curve.json"]
CQ["coverage_quality.json"]
FC["filter_catalog.json"]
MD["model_diagnostics.json"]
end
ARTIFACTS -.->|metrics & manifest| JSON
%% Dashboard Frontend
JSON --> SERVER["server.mjs<br/>Node.js HTTP server"]
subgraph Dashboard["Browser SPA"]
direction TB
APP["app.js<br/>Routing & pages"]
CHARTS["charts.js<br/>D3 visualizations"]
DATA["data.js<br/>Data loading & filters"]
APP --- CHARTS
APP --- DATA
end
SERVER --> Dashboard
subgraph Pages["Dashboard Pages"]
P1["Overview<br/>KPIs, trends, county map"]
P2["Reliability Explorer<br/>Make/model comparison"]
P3["Next-Test Estimator<br/>Risk prediction"]
P4["Data & Methods<br/>Coverage, limitations"]
end
Dashboard --> Pages
%% Privacy Controls
subgraph Privacy["Privacy Controls"]
direction LR direction LR
S1["≥100 inspections per cell"] CONTEXT[2010-15<br/>history context]
S2["≥100 distinct vehicles"] TRAIN[2016-22<br/>train]
S3["≥10 outcome records"] TUNE[2023<br/>tune]
S4["No VINs/plates/ZIPs"] CAL[2024<br/>calibrate]
HOLDOUT[2025<br/>one-time holdout]
SHADOW[2026 partial<br/>shadow context]
end end
MART -.-> SPLITS
JSON -.- Privacy MART --> LOGISTIC[Regularized logistic + Platt<br/><b>final prototype model</b>]
MART --> TREE[Histogram gradient boosting + Platt<br/><b>benchmark only</b>]
%% Testing LOGISTIC --> PRIVATE[Private model artifacts<br/>manifests, checksums, metrics]
subgraph Tests["Test Suite"] TREE --> PRIVATE
HOLDOUT -.-> REPORT[Private/report-only<br/>one-time comparison]
PRIVATE -.-> REPORT
MART --> EXPORT[Aggregate, suppress,<br/>round, validate]
PRIVATE -.->|pre-2025 diagnostics only| EXPORT
subgraph PUBLIC[dashboard/public/data - public boundary]
direction TB
MANIFEST[data_manifest.json]
OVERVIEW[overview_period_county.json]
COHORTS[cohort_scorecard.json]
AGE[age_risk_curve.json]
COVERAGE[coverage_quality.json]
FILTERS[filter_catalog.json]
DIAGNOSTICS[model_diagnostics.json]
HASHES[sha256_manifest.json]
end
EXPORT --> PUBLIC
PUBLIC --> CONTRACT[Fail-closed browser validation]
CONTRACT --> SITE[Static dashboard<br/>semantic HTML + CSS<br/>vanilla ES modules + inline SVG]
subgraph VIEWS[Presentation-ready views]
direction TB
V1[Overview]
V2[Sample cohorts]
V3[Model & benchmark]
V4[Data & methods]
end
SITE --> VIEWS
subgraph GUARDRAILS[Publication guardrails]
direction TB
G1[Private 10,000-vehicle sample<br/>not population estimates]
G2[Minimum episode, vehicle,<br/>and binary-class support]
G3[No identifiers, raw/operational rows,<br/>credentials, or row-level predictions]
G4[No county population rankings<br/>or vehicle-level prediction service]
end
EXPORT -.-> GUARDRAILS
SITE --> BOLT[Separate dashboard-only<br/>Bolt project]
subgraph TESTS[Verification]
direction LR direction LR
PT["Python unittest<br/>(feature mart, models,<br/>export, baselines)"] PY[Python pipeline/model/export tests]
CT["Node.js contract tests<br/>(dashboard JSON schemas)"] JS[Node dashboard contract tests]
end end
MART -.-> PY
PUBLIC -.-> JS
MART -.- PT classDef private fill:#fff3e0,stroke:#b45309,color:#3b2415
JSON -.- CT classDef model fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e
classDef public fill:#ecfdf5,stroke:#047857,color:#064e3b
classDef warning fill:#fef2f2,stroke:#b91c1c,color:#7f1d1d
%% Styling class SOURCE,SAMPLE,STAGE,EPISODES,MART,PRIVATE,REPORT private
classDef source fill:#e1f5fe,stroke:#0288d1 class LOGISTIC,TREE model
classDef process fill:#fff3e0,stroke:#f57c00 class PUBLIC,CONTRACT,SITE,VIEWS,BOLT public
classDef storage fill:#e8f5e9,stroke:#388e3c class GUARDRAILS warning
classDef dashboard fill:#fce4ec,stroke:#c62828
classDef test fill:#f3e5f5,stroke:#7b1fa2
class PG,DMV,INSP source
class CSV,MART,ARTIFACTS,DUCKDB storage
class BL,TREE process
class SERVER,Dashboard,Pages,JSON dashboard
class Tests test

187
docs/bolt_deployment.md Normal file
View File

@ -0,0 +1,187 @@
# Dashboard-only Bolt deployment
Last reviewed: 2026-07-21
## Non-negotiable publication boundary
Publish a **separate Bolt project or GitHub repository containing only the
contents of `dashboard/`**. In that project, `index.html` must be at the project
root.
Do not import the full Utah Vehicle Health source repository into Bolt. Do not
depend on Bolt supporting a `dashboard` working-root setting. The source
repository contains private-pipeline structure that a static host does not need,
even when local secrets and data are Git-ignored.
The deployment project must never contain `.env` files, credentials, private
data or marts, database files, model artifacts, SQL, pipeline scripts, repository
history from outside `dashboard/`, raw JSON, operational records, or row-level
predictions.
## 1. Verify the source dashboard locally
From the Utah Vehicle Health repository root:
```bash
npm --prefix dashboard test
node dashboard/server.mjs
```
Open `http://127.0.0.1:4173` and verify:
- the development-preview banner says the values come from the private
10,000-vehicle sample and are not population estimates;
- the routes are Overview, Sample cohorts, Model & benchmark, and Data & methods;
- the final model is calibrated logistic regression and the boosted tree is
benchmark-only;
- model cards show 2024 calibration-fit diagnostics, not the opened 2025
holdout comparison;
- no vehicle-level prediction service or input form exists;
- county displays communicate feed coverage rather than population rankings;
and
- a missing or invalid aggregate bundle produces an unavailable state, not
fallback estimates.
Stop the local server with `Ctrl-C` after verification.
## 2. Create a fresh dashboard-only artifact
Use a new directory and, preferably, a new deployment repository. Copy the
**contents** of `dashboard/`, not the directory's parent and not the main
repository's `.git` history.
The reviewed project tree is:
```text
index.html
styles.css
package.json
server.mjs
README.md
js/
app.js
charts.js
data.js
public/data/
age_risk_curve.json
cohort_scorecard.json
coverage_quality.json
data_manifest.json
filter_catalog.json
model_diagnostics.json
overview_period_county.json
sha256_manifest.json
tests/
contract.test.mjs
```
Do not add any other repository directory. In the fresh artifact root, run:
```bash
npm test
HOST=0.0.0.0 npm start
```
No dependency install or build step is required. The `PORT` environment
variable is honored automatically by `server.mjs` when a host supplies it.
Before creating the deployment repository, inspect its entire file tree. Apart
from `package.json` project metadata, the only JSON allowed is the reviewed
aggregate bundle under `public/data/`. Confirm that `data_manifest.json` still
declares:
```text
development_preview: true
population_estimate_allowed: false
locked_test_metrics_published: false
```
Do not hand-edit an aggregate file to make validation pass. Regenerate it
through the private local exporter and repeat the review.
## 3. Import only the dashboard repository into Bolt
The recommended handoff is a fresh GitHub repository containing the reviewed
tree above. It may be private if the connected Bolt account has access.
1. On the Bolt homepage, choose the GitHub import control.
2. Select the separate dashboard-only repository or use **Import from URL**.
3. Confirm `index.html` appears at the imported project root.
4. Inspect the Bolt file tree before previewing. If SQL, scripts, artifacts,
private data, `.env`, or source-repository files appear, stop and delete that
Bolt project; create a clean dashboard-only project instead.
5. Use `HOST=0.0.0.0 npm start` if Bolt requests a preview command. There is no
build command.
Bolt's current GitHub import workflow is documented in
[GitHub for version control](https://support.bolt.new/integrations/git).
If a separate GitHub repository is not used, create a fresh Bolt/StackBlitz
project and upload only the reviewed dashboard files so the same tree appears at
the project root. Never upload the full source repository and then try to hide
or ignore its private-pipeline directories.
## 4. Review in Bolt before publishing
Use the preview URL and repeat the local visual checks. Also verify directly:
- `/public/data/data_manifest.json` resolves;
- an unknown file path returns `404`;
- `/.env`, `/data/`, `/artifacts/`, `/models/`, `/sql/`, and `/scripts/` return
`404`;
- the browser makes requests only for the static HTML, CSS, JavaScript, and
approved files under `/public/data/`; and
- no analytics, database, authentication, or third-party data service was added
by an automated Bolt edit.
Preview privately first. If Bolt proposes code changes, review the full diff and
rerun `npm test`; do not accept changes that add a database, environment secret,
tracking script, model API, prediction form, or remote data source.
## 5. Publish with Bolt Hosting
Bolt Hosting is the intended host for this prototype:
1. Open the dashboard-only project.
2. Select **Publish** in the upper-right corner.
3. Choose private visibility for presentation review when available.
4. Select **Publish** and wait for the generated `bolt.host` URL.
5. Run the post-publication checks below before changing visibility to public.
Bolt documents this flow in
[Publish your project to a live website](https://support.bolt.new/cloud/hosting/publish).
Bolt's Netlify integration is optional and is not required for this dashboard;
see [Netlify integration](https://support.bolt.new/integrations/netlify) only if
hosting requirements explicitly change.
## 6. Post-publication checks
At the exact published URL:
1. Hard-refresh and confirm the development-preview banner remains visible.
2. Visit all four routes and verify labels, charts, and keyboard navigation.
3. Confirm the Model & benchmark view identifies logistic as final, tree as
benchmark, and its values as 2024 calibration-fit checks.
4. Confirm no 2025 holdout metric appears in the site or public JSON.
5. Confirm there is no vehicle-level lookup, input form, or row-level output.
6. Confirm coverage language does not imply statewide representation or county
ranking.
7. Request the denied paths from step 4 again and require `404`.
8. Save the published URL, release ID, publication date, and reviewer decision;
do not save private paths, row values, or credentials in the release note.
If any check fails, use Bolt's Publish menu to unpublish or keep the site private
until a corrected dashboard-only artifact passes local and hosted review.
## Updating a published prototype
Changes in a Bolt project are not automatically live. For each update:
1. regenerate/review aggregates locally;
2. run the dashboard contract test;
3. update the separate dashboard-only repository/project;
4. inspect Bolt's diff and preview;
5. use **Publish > Update**; and
6. repeat every post-publication check.
Never synchronize private pipeline directories into the deployment repository.

173
docs/dashboard_spec.md
View File

@ -1,97 +1,126 @@
# Bolt dashboard specification # Static dashboard specification
## Product language ## Release status and language
Use **Utah Vehicle Health** as the brand, but call the modeled quantity The checked-in site is a **development preview** built from suppressed
**next-episode non-pass risk**. The estimator is a cohort estimate, not a aggregates derived from the private, page-sampled 10,000-vehicle development
diagnosis, certification, or guarantee. cohort. Every value is a sample result, **not a population estimate**. The site
must not present statewide prevalence, county rankings, causal comparisons, or
individual predictions.
## Four-page MVP Use **Utah Vehicle Health** as the project name and **next-episode first-attempt
non-pass** for the modeled outcome. Non-pass combines fail, reject, and abort.
The dashboard is not a diagnosis, certification, safety assessment,
roadworthiness assessment, or guarantee.
The sections below describe the full product target. The checked-in ## Implemented four-view prototype
development preview intentionally implements a narrower safe subset: binary
pass/non-pass aggregates, supported make/model scorecards, age bands, coverage
quality, and pre-2025 model diagnostics. Four-class charts, uncertainty
intervals, adjusted scorecards, and the prediction lookup remain disabled until
their own reviewed aggregate assets exist.
### Overview ### Overview
- Eligible inspections, pass rate, non-pass rate, and covered-period KPIs - Visibly persistent development-preview warning
- Quarterly pass/fail/reject/abort trend with blanks shown separately - Rounded published support and sample pass/non-pass rates
- Covered-county map; unavailable counties remain gray - Sample first-attempt non-pass trend for approved, pre-2025 periods
- Non-pass risk versus vehicle age with intervals - County **feed-coverage** map, with unavailable counties shown as unavailable
- Clear notices for partial periods and limited feed coverage rather than zero
- Sample non-pass pattern by coarsened vehicle-age band
- Plain-language target and limitations callout
### Reliability explorer County values provide sample/feed context only. They must not be sorted,
headlined, or described as population county performance.
- Search supported canonical make/model cohorts ### Sample cohorts
- Compare up to three cohorts across vehicle-age bands
- Observed versus model-adjusted risk toggle
- Outcome-mix bar and uncertainty-aware ranked dot plot
- County, make/model, age band, fuel, program, and period filters
- Support size and interval displayed for every estimate
### Next-test risk estimator - Search and compare only supported, suppression-cleared make/model cohorts
- Show observed sample non-pass rates and rounded support
- Default sorting by support rather than risk
- Disable filters or adjusted views that the aggregate bundle cannot support
- Explain that omitted cohorts may be suppressed or unavailable
Inputs are coarsened, non-identifying attributes: county, supported make/model, These are descriptive development-sample cohorts, not reliability ratings,
vehicle-age band, fuel, prior episode outcome, time-since-prior band, season, causal make/model effects, or recommendations.
and approved program category.
Output a calibrated non-pass probability, uncertainty interval, relevant ### Model & benchmark
baseline, and aggregate factor contributions. Never request VIN, plate, exact
address, station, free text, or current-test diagnostics.
### Data and methods - Identify calibrated logistic regression as the sole final prototype model.
- Identify calibrated histogram gradient boosting as benchmark-only.
- Show only pre-2025, 2024 calibration-fit diagnostics from the private sample.
- Explain that those same-partition values are calibration checks, not
independent final-performance or population estimates.
- Keep the opened 2025 holdout comparison in the private report/presentation,
not in the browser bundle.
- State plainly that the finished prototype has no vehicle-level prediction
service, personal input form, or row-level output.
- Coverage timeline and source/year missingness heatmap ### Data & methods
- Episode and target definitions
- `slc`/`slco` source-era explanation - First-attempt returning-vehicle episode and binary-target definitions
- Temporal split, PR-AUC, Brier score, and calibration plot - Point-in-time feature and leakage controls
- Subgroup/source-era performance - Coverage timeline by source era
- Leakage controls, DMV gaps, partial periods, and limitations - Chronological development-sample split and one-time 2025-gate disclosure
- Evidence-governance summary for the logistic final model and tree benchmark
- Known sampling, coverage, label, validation, and interpretation limits
- Private-to-public publication flow
The boosted tree appears only in benchmark context, never as a second final
model. The dashboard never requests a VIN, plate, ZIP, exact address, station,
technician, free text, or current-test diagnostic.
## Public data contract ## Public data contract
| Dataset | Safe grain | | Asset | Allowed content |
| --- | --- | | --- | --- |
| `data_manifest` | Data cutoff, deterministic release ID, model versions, definitions and exclusions | | `data_manifest.json` | Release ID, model versions, scope flags, definitions, and suppression settings |
| `overview_period_county` | Quarter/year × public county with rounded support and outcome rates | | `overview_period_county.json` | Suppressed quarter x covered-feed county sample aggregates |
| `cohort_scorecard` | Approved make/model × age band, optionally coarsened county/fuel | | `cohort_scorecard.json` | Suppressed make/model sample aggregates with rounded support |
| `age_risk_curve` | Approved cohort × age point/band with risk, interval and support | | `age_risk_curve.json` | Suppressed sample rates by coarsened vehicle-age band |
| `prediction_lookup` | Only supported coarsened input combinations and calibrated outputs | | `filter_catalog.json` | Publishable categories and supported periods |
| `filter_catalog` | Publishable categories and valid combinations | | `model_diagnostics.json` | Train/tune/calibration development metrics only; no 2025 metrics |
| `model_diagnostics` | Approved partition-level metrics; locked metrics require a separate release gate | | `coverage_quality.json` | Source-era availability by year |
| `coverage_quality` | Source era × year volume, blank rate, linkage and availability | | `sha256_manifest.json` | Checksums for the approved bundle |
Use purpose-built outputs rather than a single high-dimensional browser cube. Every JSON asset must declare `development_preview=true` and
Bolt receives only these sanitized, versioned assets—never countydata `population_estimate_allowed=false`. The public manifest must declare
credentials or private analytical rows. `locked_test_metrics_published=false`. The browser validates the complete bundle
and shows no estimates if any required asset fails its contract.
## Publication controls The browser bundle must never contain raw analytical rows, prediction lookup
rows, hidden unsuppressed chart layers, or values recoverable only through the
developer console.
- Suppress cells below 100 eligible inspections. ## Suppression and privacy controls
- Also require at least 100 distinct private vehicle tokens in every published
cell; tokens and distinct counts never enter the public asset.
- Suppress when an outcome or its complement has fewer than 10 records.
- Require at least 10 distinct vehicles contributing each binary class.
- Apply complementary suppression so totals cannot reconstruct hidden cells.
- Combine rare categories, coarsen model years, and round displayed counts.
- Recheck thresholds after every filter combination.
- Do not include suppressed rows in browser bundles, API responses, downloads,
analytics logs, or hidden chart layers.
- Downloads contain only the sanitized summary currently displayed.
## Visual direction - Require at least 100 eligible episodes and 100 distinct private vehicles per
published cell.
- Require at least 10 observations and 10 distinct vehicles in both binary
outcome classes.
- Apply complementary suppression when a visible total could reveal a hidden
cell.
- Combine rare categories, coarsen age/year fields, and round support.
- Recheck thresholds for the exact grain of every released asset.
- Omit suppressed rows entirely; do not send them to the browser.
- Publish no VIN, plate, ZIP, exact address, station, technician identifier,
private token, raw JSON, credential, operational record, free text, or
row-level prediction.
- Make no browser or hosted-service connection to `countydata`.
Use a restrained Utah/desert palette: teal pass, red fail, amber reject, purple ## Accessibility and visual behavior
abort, and gray missing. Use probability bars, calibrated dot plots, confidence
bands, and cohort comparisons instead of gauges. Do not rely on color alone.
## Stretch pages - Use semantic headings, landmarks, labels, status messages, and keyboard-
reachable navigation.
- Provide text alternatives or accessible labels for charts.
- Do not rely on color alone.
- Use a restrained Utah/desert palette and direct probability/rate encodings;
avoid gauges and decorative risk scores.
- Preserve the warning banner and fail-closed unavailable state on desktop and
mobile layouts.
- Failure-to-pass journeys with funnels, attempts-to-pass and survival curves ## Deployment boundary
- Four-class outcome probabilities
- Source-scoped OBD early-warning analysis Bolt receives a separate dashboard-only project whose root is the contents of
- Automated aggregate refresh with a dedicated read-only role `dashboard/`. Do not import or upload the full private-pipeline repository and
- Model-drift monitoring do not rely on a configurable subdirectory working root. The deployable project
contains no `.env`, private `data/`, model artifacts, SQL, pipeline scripts, or
repository history outside the dashboard directory.
See [bolt_deployment.md](bolt_deployment.md) for the reviewed handoff and
post-publication checks.

13
docs/data_inventory.md
View File

@ -1,8 +1,15 @@
# Countydata inventory # Countydata inventory
Inventory date: 2026-07-15; live schema revalidated 2026-07-21. All inspection Inventory date: 2026-07-15; live schema revalidated 2026-07-21. All inventory
was performed with read-only transactions, metadata queries, aggregate queries, work used read-only transactions, metadata queries, aggregate queries, and
and bounded JSON field-path sampling; no identifier values were exported. bounded JSON field-path sampling; no identifier values were exported.
> **Source-discovery context only:** Database-wide metadata and aggregate counts
> below describe the accessible source, not Utah Vehicle Health research
> results. The finished prototype uses a private, page-sampled 10,000-vehicle
> development cohort; all model/dashboard findings are sample-based and not
> population estimates. See the authoritative [project charter](project_charter.md)
> and [final report](final_report.md).
## Live access and full-schema boundary ## Live access and full-schema boundary

113
docs/demo_script.md Normal file
View File

@ -0,0 +1,113 @@
# Utah Vehicle Health dashboard demo script
Target length: **2 minutes 45 seconds**, embedded in the
[10-minute presentation](presentation_outline.md).
## Before the audience arrives
1. From the repository root, run `node dashboard/server.mjs`.
2. Open `http://127.0.0.1:4173/#overview`.
3. Confirm the header says the sample aggregates validated.
4. Confirm the private-sample development-preview banner is visible.
5. Visit all four routes: Overview, Sample cohorts, Model & benchmark, and Data
& methods.
6. Reset Sample cohorts and leave its sort on largest support.
Do not open developer tools, private files, model artifacts, database clients,
or environment variables during the presentation.
## Live talk track
### 0:00-0:25 — Establish the boundary
**Action:** Start on **Overview**. Point to the development-preview banner before
pointing to a chart.
**Say:**
> The most important element is this banner. Every value in the dashboard comes
> from the private 10,000-vehicle development sample. These are suppression-
> reviewed sample aggregates, not population estimates, and they cannot support
> individual decisions or county rankings.
### 0:25-1:00 — Explain the overview
**Action:** Point to published support and the observed first-attempt non-pass
trend, then the coverage map and vehicle-age chart. Do not rank counties by
outcome.
**Say:**
> The overview reports rounded support and sample first-attempt outcomes for
> returning vehicles. Non-pass combines fail, reject, and abort. The trend is
> descriptive of sampled records only. This county graphic communicates feed
> availability: gray means unavailable, not zero and not better. The age pattern
> is also an observed sample association, not a causal claim or diagnosis.
### 1:00-1:35 — Use Sample cohorts safely
**Action:** Open **Sample cohorts**. Search for `Toyota` or another currently
supported make. Leave sorting on largest support. Point to rounded support and
the “observed sample non-pass” label.
**Say:**
> This view contains only prior make-and-model cohorts that cleared the
> publication thresholds. I can search supported cohorts, but these are observed
> development-sample associations—not reliability grades, rankings, or
> recommendations. Unsupported slices are not inferred in the browser, and
> suppressed rows are absent rather than hidden.
**Action:** Reset the search before leaving the view.
### 1:35-2:15 — Separate the final model from its benchmark
**Action:** Open **Model & benchmark**. Point first to the role badges, then the
2024 calibration-fit scope strip, then the “No vehicle-level prediction service”
boundary.
**Say:**
> The governance decision is explicit: calibrated logistic regression is the
> final prototype model, while the boosted tree is benchmark-only. The numbers
> shown here are pre-2025 checks on the same 2024 sample partition used to fit
> calibration. They are not independent final-performance or population
> estimates, so we do not select between the cards from these values. The
> one-time 2025 comparison shown earlier favored logistic and stays in the report
> and presentation, not the browser bundle. This finished dashboard contains no
> vehicle lookup, personal inputs, or row-level prediction output.
### 2:15-2:40 — Close on method and privacy
**Action:** Open **Data & methods**. Briefly point to Prediction unit, the
chronological timeline, evidence status, and the privacy flow.
**Say:**
> The method page makes the contract visible: the target is the first attempt of
> a returning vehicle's next episode, and every feature ends before that episode.
> Training, tuning, and calibration are chronological. It also records that 2025
> was opened once after choices were frozen. Finally, the public path ends in
> suppressed summaries. No private rows or database connection reach this site.
### 2:40-2:45 — Transition
**Say:**
> That is the prototype: useful sample evidence, with model and privacy
> boundaries kept visible.
Return to the slide deck's privacy architecture.
## If something goes wrong
If the dashboard shows **data unavailable**, do not bypass validation or edit
JSON. Say:
> The site rejected an aggregate-contract mismatch and is failing closed, so it
> shows no estimates. That behavior is part of the privacy and integrity design.
Then continue with a backup screenshot or slides. If a cohort search returns no
rows, explain that unsupported or suppressed cohorts are intentionally absent
and reset the filter. Never substitute remembered values or improvise a vehicle-
level example.

181
docs/development_results.md
View File

@ -1,67 +1,66 @@
# Development run results # Development run results
Run date: 2026-07-15 Run date: 2026-07-15; presentation wording reviewed 2026-07-21
These results validate the engineering and modeling pipeline. They are **not > **Scope of every value below:** the private, page-sampled 10,000-vehicle
population estimates**: the development extract samples inspection-table pages > development cohort. These are sample results, not population estimates. Page
and then retrieves complete histories for up to 10,000 vehicles, which > sampling over-represents vehicles with more inspection records. Nothing in
over-represents vehicles with more inspection records. > this document supports statewide/county prevalence or county rankings.
## Label audit ## Label audit
The raw Utah County feed leaves `overall_result` blank for 3,747,862 records. The binary target is pass versus non-pass, where non-pass combines recognized
An aggregate-only server audit found 3,469,102 Utah `obd`/`OBD` records with a fail, reject, and abort outcomes. Blanks and unrecognized values are unlabeled,
controlled pass, fail, reject, or abort value in `obd_result`. Across non-Utah not passes.
feeds where both fields were recognized, the two fields agreed on binary pass
versus non-pass 99.30% of the time, but their fail-versus-reject categories were
not interchangeable.
Label contract v3 therefore permits the Utah OBD value only as a provenance- For the older Utah County feed, the documented source/program/test rule permits
tagged binary proxy. TSI, `other/C`, `B`, blank, and unknown values stay a controlled OBD result to fill a blank overall result as a binary proxy. It
unlabeled. Four-class analysis must use `overall_result` exclusively. This also does not support a four-class interpretation. In the private 10,000-vehicle
matches the official program distinction between a readiness rejection and a sample, the rule restored 11,334 labels and left 1,281 events unlabeled. These
failed inspection; see the [Utah inspection requirements](https://dmv.utah.gov/register/inspections/) sample counts are not population estimates. Label provenance stays in private
and [program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf). audit metadata and is not a predictor.
In the development extract, the rule restored 11,334 labels and left 1,281
events unlabeled. The label-source field remains private audit metadata and is
not a model feature.
## Pipeline reconciliation ## Pipeline reconciliation
| Stage | Rows | This table reports private 10,000-vehicle development-sample pipeline counts,
not population totals.
| Sample pipeline stage | Rows |
| --- | ---: | | --- | ---: |
| Source events written after VIN validation | 83,552 | | Source events written after identifier validation | 83,552 |
| Clean events after duplicate/conflict handling | 80,190 | | Clean events after duplicate/conflict handling | 80,190 |
| Inspection episodes | 69,588 | | Inspection episodes | 69,588 |
| Eligible returning targets | 44,659 | | Eligible returning-vehicle targets | 44,659 |
The sample contains 9,996 retained vehicle tokens. The extraction manifest, The sample retained 9,996 private vehicle tokens. Extraction and mart manifests,
compressed-file SHA-256, mart manifest, Parquet SHA-256, and row counts all compressed-file and Parquet checksums, and stage counts reconcile. Every mart
reconcile. Every mart invariant reports zero violations. Raw VIN and raw OBD invariant reports zero violations. Raw identifiers and raw OBD values are absent
result are absent from the staging output schema, feature mart, and model from staging output, feature mart predictors, and model features.
features.
| Temporal partition | Eligible targets | Never-fit audit targets | The following partition counts are also from the private 10,000-vehicle
development sample and are not population estimates.
| Sample temporal partition | Eligible targets | Never-fit audit targets |
| --- | ---: | ---: | | --- | ---: | ---: |
| Train, 2016–2022 | 31,745 | 3,129 | | Train, 2016-2022 | 31,745 | 3,129 |
| Tune, 2023 | 4,972 | 526 | | Tune, 2023 | 4,972 | 526 |
| Calibrate, 2024 | 2,633 | 277 | | Calibrate, 2024 | 2,633 | 277 |
| Locked test, 2025 | 4,451 | 480 | | One-time holdout, 2025 | 4,451 | 480 |
| Shadow, 2026 partial | 858 | 106 | | Shadow, 2026 partial | 858 | 106 |
Audit-bucket vehicles are excluded from fitting, tuning, and calibration. The Audit-bucket vehicles are excluded from fitting, tuning, and calibration. Normal
normal trainers do not calculate 2025 outcomes or metrics without the explicit training does not calculate 2025 outcomes or metrics without the explicit
locked-evaluation flag. locked-evaluation flag.
## Baseline results before the locked test ## Pre-2025 development diagnostics
The table below uses only non-audit vehicles. Average precision is the project's The table uses non-audit rows from the private 10,000-vehicle development
PR-AUC summary. The calibrated logistic row is shown only on the partition used sample. Average precision is the project's PR-AUC summary. The calibrated 2024
to fit the calibrator and is therefore a calibration diagnostic, not an row is evaluated on the same sample partition used to fit Platt scaling, so it
independent final estimate. is a calibration diagnostic rather than independent final performance. None of
these values is a population estimate.
| Partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% | | Sample partition | Model | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | --- | ---: | ---: | ---: | ---: | | --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 | | 2023 tune | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 | | 2023 tune | Previous episode, literal | 0.166 | 0.1723 | 0.582 | 0.285 |
@ -71,66 +70,62 @@ independent final estimate.
| 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 | | 2024 calibrate | Logistic | 0.257 | 0.0936 | 0.690 | 0.308 |
| 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** | | 2024 calibrate | Logistic + Platt | **0.257** | **0.0936** | **0.690** | **0.308** |
The selected logistic regularization was `C=0.03`. It converged in 2,390 of the The selected logistic regularization is `C=0.03`. It converged in 2,390 of the
5,000 allowed iterations; Platt calibration converged in five iterations. All 5,000 allowed iterations; Platt calibration converged in five iterations. These
five candidates and all stored probabilities passed explicit convergence and are development-run diagnostics. All candidates and stored probabilities passed
finite-value checks. explicit convergence and finite-value checks.
The literal previous-outcome baseline is useful as a ranking sanity check but On the 2023 sample partition, logistic regression more than doubled prevalence
produces overconfident zero/one probabilities, explaining its poor Brier and log PR-AUC and raised top-10% precision from 12.5% to 31.5%. This describes the
loss. The transparent logistic model is the current development leader: on the private development sample only.
2023 tuning partition it more than doubles prevalence PR-AUC and raises top-10%
precision from 12.5% to 31.5%.
## One-time development holdout ## One-time development-sample holdout
The nonlinear live-verification command invoked the explicit 2025 gate after The nonlinear verification command invoked the explicit 2025 gate only after
its feature contract and four-candidate grid had already been fixed and its the feature contract and benchmark grid had been frozen and candidate choices
candidate had been selected only on 2023. No test-driven model change was made. had been made from pre-2025 sample data. No model was changed in response. These
At that point the development specification was frozen and the already-fixed are one-time development-sample holdout results, not pristine future-test or
logistic model was evaluated once for a direct comparison. These are population estimates.
development-sample holdout results, not final population claims.
| 2025 non-audit model | PR-AUC | Brier | ROC-AUC | Precision at top 10% | | 2025 private-sample model | Role | PR-AUC | Brier | ROC-AUC | Precision at top 10% |
| --- | ---: | ---: | ---: | ---: | | --- | --- | ---: | ---: | ---: | ---: |
| Training prevalence | 0.123 | 0.1076 | 0.500 | 0.123 | | Training prevalence | Baseline | 0.123 | 0.1076 | 0.500 | 0.123 |
| Previous episode, literal | 0.158 | 0.1731 | 0.576 | 0.268 | | Previous episode, literal | Baseline | 0.158 | 0.1731 | 0.576 | 0.268 |
| Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** | | Logistic + Platt | **Final model** | **0.261** | **0.1011** | **0.693** | **0.312** |
| Histogram gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 | | Histogram gradient boosting + Platt | Benchmark only | 0.238 | 0.1024 | 0.690 | 0.292 |
The holdout contains 3,971 non-audit episodes from 3,734 vehicles. The The sample holdout contains 3,971 non-audit episodes from 3,734 vehicles. Its
never-fit audit contains another 480 episodes from 450 vehicles; calibrated never-fit audit adds 480 episodes from 450 vehicles; calibrated logistic PR-AUC
logistic PR-AUC is 0.253 and Brier is 0.0859 there. The logistic model beats is 0.253 and Brier score is 0.0859 there. All are private-sample diagnostics,
both simple baselines on the project's headline metrics and remains the chosen not population estimates or external validation.
development model. The tree did not provide a decisive pre-test improvement
that justified its added complexity, and no further 2025-informed tuning is
permitted.
## Remaining gates The calibrated logistic regression remains the final development model. It
beats both simple baselines on the headline metrics and beats the boosted-tree
1. Run a complete, contiguous bounded extraction for publishable population benchmark in the one-time comparison. The benchmark did not justify its added
aggregates; the page-sampled development cohort cannot support dashboard complexity. No further 2025-informed tuning is permitted.
prevalence or county rankings.
2. Treat any later full-data 2025 result as confirmatory rather than a pristine
unseen test, because the development sample's holdout has now been opened.
3. Run the frozen subgroup/source-era report and clustered uncertainty
analysis on the complete extraction.
4. Replace the checked-in development-preview bundle with complete-data
aggregates only after the population-publication review passes.
## Sanitized dashboard preview ## Sanitized dashboard preview
The repository now includes a fail-closed exporter and a static four-view The static preview uses only suppression-cleared sample aggregates and
dashboard shell. The preview exporter reads only train, tune, and calibration development diagnostics dated before 2025. It intentionally excludes the
rows dated before 2025. It rejects any model manifest that says the holdout was one-time holdout metrics. Every asset declares `development_preview=true` and
evaluated, rejects any metric row outside the three approved partitions, and `population_estimate_allowed=false`; the manifest declares
marks every asset `development_preview=true` and `locked_test_metrics_published=false`.
`population_estimate_allowed=false`.
Published cells must clear minimum episode and distinct-vehicle thresholds, Published cells must clear minimum episode, distinct-vehicle, and binary-class
including both binary-class complements. Supports are rounded, direct and thresholds. Supports are rounded, suppressed rows are omitted, and a checksum
pseudonymous identifiers stay private, and a checksum manifest covers the manifest covers the approved JSON bundle. The site fails closed if a contract
approved JSON bundle. Private model manifests bind the exact metrics files, and check fails and contains no vehicle-level prediction service.
the public manifest exposes a deterministic release ID plus model versions for
provenance. The site disables the estimator because no privacy-reviewed No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
prediction lookup exists. credential, operational record, or row-level prediction is in the public
bundle.
## Boundary of the finished prototype
The repository completes a presentation-ready development prototype, not a
population study or production service. A complete population-frame extraction,
fresh external/future validation, individualized prediction lookup, DMV
enrichment, multiclass modeling, and operational monitoring are outside this
prototype's deliverables. They would require separate approval, analysis, and
privacy review; they are not implied by the current results.

223
docs/final_report.md Normal file
View File

@ -0,0 +1,223 @@
# Utah Vehicle Health: final development-prototype report
Prepared July 21, 2026
> **Result scope:** Every analytical count, rate, chart, and model metric in this
> report comes from the private, page-sampled 10,000-vehicle development cohort.
> These are sample results, not population estimates. The cohort cannot support
> statewide or county prevalence, county rankings, causal conclusions, or
> decisions about an individual vehicle.
## Executive summary
Utah Vehicle Health asks one focused question:
> Using only information available before an inspection episode begins, how
> well can a calibrated logistic regression estimate whether a returning
> vehicle's next episode will have a first-attempt non-pass outcome?
The finished prototype builds leakage-safe inspection episodes, creates
point-in-time history features, evaluates models chronologically, and publishes
only suppression-reviewed sample aggregates to a static dashboard. Non-pass
combines recognized fail, reject, and abort outcomes; blank or unrecognized
results are not treated as passes.
The final model is regularized logistic regression with Platt calibration. On
the private sample's one-time 2025 holdout, it achieved PR-AUC 0.261 and Brier
score 0.1011. The calibrated histogram gradient-boosted benchmark achieved
PR-AUC 0.238 and Brier score 0.1024 on the same sample holdout. Logistic
regression remains the final model because it was both more transparent and
better on the declared headline metrics in this development comparison.
Those values demonstrate that the pipeline and model comparison work on this
development sample. They do not establish population performance, production
readiness, or external validity.
## Research scope
The prediction unit is the first attempt of the next inspection episode for a
returning vehicle. Consecutive attempts no more than 30 days apart form one
episode. This prevents a fail followed quickly by a retest from becoming two
nominally independent prediction targets. A vehicle is eligible only after a
prior episode has been observed.
The binary outcome is:
- pass; or
- non-pass, combining fail, reject, and abort.
Reject and abort can reflect process or readiness conditions, so this report
does not equate non-pass with mechanical failure. Cold-start prediction,
four-class modeling, fail-only sensitivity analysis, DMV enrichment,
station-level analysis, and individual prediction are outside the completed
prototype.
## Development data
The private extraction samples inspection-table pages, selects up to 10,000
vehicles, and then retrieves their complete available inspection histories. It
retained 9,996 private vehicle tokens after validation. The resulting sample
contains 83,552 source events, 69,588 episodes, and 44,659 eligible returning-
vehicle targets. These figures are pipeline-reconciliation counts for the
private sample, not estimates of Utah inspection volume.
The sample design intentionally makes end-to-end development manageable, but it
over-represents vehicles with more inspection records. Participating feeds also
change over time and do not cover all 29 Utah counties. The dashboard therefore
labels every value as a development-sample aggregate and uses county only to
communicate covered-feed context—not to publish population rankings.
The extraction and mart manifests bind hashes, configurations, and row counts.
Private vehicle linkage uses a one-way keyed token. Direct identifiers and raw
source payloads remain outside the analytical exports and public site.
## Leakage-safe method
Every feature exists before the target episode begins. The final model uses
vehicle age, prior episode and attempt counts, time since prior history, prior
non-pass rate, prior first/final outcomes, previously observed make/model,
public county context, and season.
The model excludes the target attempt's outcome and diagnostics, later attempts
from the target episode, station or technician information, future records,
full-history leakage, direct or pseudonymous identifiers, and preprocessing
learned from later partitions. DMV enrichment and rich current-test OBD fields
are not part of the final model.
The chronology is fixed:
| Period | Role in the private 10,000-vehicle development sample |
| --- | --- |
| 2010-2015 | Historical context only |
| 2016-2022 | Fit preprocessing and model parameters |
| 2023 | Select logistic regularization and compare fixed candidates |
| 2024 | Fit Platt calibration and inspect calibration behavior |
| 2025 | One-time development-sample holdout |
| 2026 partial | Shadow context only |
Random row splitting is prohibited. A separate set of private vehicle buckets
is never used for fitting and serves as a stress test.
## Model comparison and decision
The declared comparison includes training prevalence, the literal previous-
episode outcome, regularized logistic regression, and histogram gradient
boosting. Logistic regression uses `C=0.03`; Platt scaling is fit on the 2024
sample partition. The tree exists only to test whether nonlinearity provides a
meaningful development gain.
The table reports non-audit rows from the private, page-sampled 10,000-vehicle
development cohort. It is not population performance.
| Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
| --- | --- | ---: | ---: | ---: | ---: |
| 2023 tune | Prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
| 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
On the 2023 sample partition, logistic regression more than doubled the
prevalence baseline's PR-AUC and raised precision among the highest-scored 10%
from 12.5% to 31.5%. On the one-time 2025 sample holdout, calibrated logistic
also exceeded the simple baselines and the boosted-tree benchmark on PR-AUC and
Brier score.
The explicit 2025 gate was opened once after model specifications were frozen
from pre-2025 data. No model was changed in response. That evaluation is a
development holdout, not a pristine future test, and no further 2025-informed
tuning is allowed.
The final decision is therefore straightforward: keep calibrated logistic
regression. It satisfies the development objective with a simpler, more
explainable form, while the nonlinear benchmark did not show a gain that
justified extra complexity.
## Static dashboard
The presentation-ready dashboard is a dependency-free static site with four
views:
1. **Overview** shows rounded support, sample outcome patterns, vehicle-age
patterns, and county feed coverage.
2. **Sample cohorts** shows only supported make/model sample aggregates; it does
not issue reliability ratings or recommendations.
3. **Model & benchmark** identifies calibrated logistic regression as final and
the boosted tree as benchmark-only. It shows pre-2025, 2024 calibration-fit
checks—not the opened 2025 holdout comparison.
4. **Data & methods** explains the target, chronology, evidence status,
limitations, and privacy boundary.
The finished dashboard contains no vehicle lookup, personal input form, or
row-level prediction service.
The dashboard loads purpose-built JSON rather than private analytical rows. It
checks schemas, cross-file consistency, publication flags, and checksums. If the
bundle fails validation, the interface shows an unavailable state instead of
fallback estimates.
Every published cell requires minimum support for episodes, distinct vehicles,
and both binary outcomes. Supports are rounded and suppressed rows are omitted
entirely. The checked-in assets use only pre-2025 sample aggregates and
diagnostics; the one-time 2025 values in this report are not in the browser
bundle.
## Privacy and release boundary
The public boundary is intentionally narrow:
```text
private read-only source
-> private local extraction and model pipeline
-> suppression-reviewed aggregate JSON
-> dashboard-only static Bolt project
```
The public dashboard and presentation materials contain no VIN, plate, ZIP,
station, technician identifier, private vehicle token, raw JSON, credential,
operational record, or row-level prediction. The browser never connects to the
source database.
Bolt publication must use a separate project or repository containing only the
contents of `dashboard/`, with `index.html` at its root. The full source
repository must not be imported into Bolt. The reviewed procedure is in
[bolt_deployment.md](bolt_deployment.md).
## Limitations
- The private page sample is not population-representative.
- Results cover returning vehicles in participating feeds, not all Utah
vehicles or counties.
- Geography, source era, program mix, and time are entangled.
- Reject and abort are heterogeneous non-pass outcomes.
- The 2025 comparison is a one-time development holdout, not external
validation.
- Sample calibration and ranking do not establish operational usefulness.
- Observed cohort differences are descriptive associations, not causal effects
or recommendations.
- The absence of protected attributes does not establish fairness.
## Conclusion
The prototype answers its development question: a leakage-safe, calibrated
logistic regression produces materially better discrimination and probability-error metrics
than simple baselines in the private 10,000-vehicle sample, and it performs
better than the boosted-tree benchmark in the one-time holdout comparison. The
more important product result is the disciplined boundary around that finding:
one target, chronological evaluation, transparent model selection, persistent
sample labeling, fail-closed aggregate publication, and no row-level exposure.
This is a completed development prototype—not a population estimate, county
ranking, production model, diagnosis, or individual decision tool.
## Supporting documentation
- [Project charter](project_charter.md)
- [Modeling protocol](modeling_protocol.md)
- [Detailed development results](development_results.md)
- [Model card](model_card.md)
- [Dashboard specification](dashboard_spec.md)
- [Private-to-public architecture](architecture.mmd)
- [10-minute presentation outline](presentation_outline.md)
- [Dashboard demo script](demo_script.md)
- [Dashboard-only Bolt deployment](bolt_deployment.md)

272
docs/model_card.md
View File

@ -1,197 +1,161 @@
# Utah Vehicle Health model card # Utah Vehicle Health model card
Last updated: 2026-07-15 Last updated: 2026-07-21
Status: **development prototype; not approved for production or population claims** Status: **final development-prototype model; not approved for production or
population claims**
> Every count, rate, and performance value in this model card comes from the
> private, page-sampled 10,000-vehicle development cohort. These are sample
> results, not population estimates.
## Model summary ## Model summary
Utah Vehicle Health estimates the probability that the first attempt of a Utah Vehicle Health estimates whether the first attempt of a returning
returning vehicle's next emissions-inspection episode will be a non-pass. The vehicle's next emissions-inspection episode will be a non-pass. The final model
current selected model is a regularized logistic regression followed by Platt is regularized logistic regression (`C=0.03`) followed by Platt probability
probability calibration. It is an inspection-history model, not a mechanical calibration. Histogram gradient boosting is retained only as a benchmark.
health, safety, roadworthiness, or legal-compliance model.
This model card summarizes the current page-sampled development run. The full This is an inspection-history model. It is not a model of overall mechanical
research and product contracts are in the [project charter](project_charter.md), health, safety, roadworthiness, or legal compliance.
[modeling protocol](modeling_protocol.md), and
[dashboard specification](dashboard_spec.md).
## Intended use and prohibited use ## Intended use
Intended uses are to validate the data-engineering and modeling pipeline, The development model supports:
compare leakage-safe model candidates, study aggregate patterns, and support a
clearly marked development dashboard preview.
Do not use the model to: - validating leakage-safe longitudinal feature engineering;
- comparing a transparent final model with simple and nonlinear benchmarks;
- explaining calibration and chronological evaluation; and
- powering aggregate, suppression-reviewed development-preview diagnostics.
- make decisions about an individual vehicle, owner, registration, inspection, Do not use it to:
or station;
- diagnose a vehicle, guarantee an inspection result, or infer safety or
roadworthiness;
- rank or penalize people, counties, programs, or inspection stations;
- make causal claims from observed associations; or
- report statewide prevalence, county rankings, or production performance from
the current development sample.
The public estimator remains disabled because no privacy-reviewed prediction - decide anything about an individual vehicle, owner, registration, inspection,
lookup has been approved. county, program, station, or technician;
- diagnose a vehicle or guarantee an inspection outcome;
- rank counties, programs, stations, vehicle cohorts, or people;
- make causal claims from observed sample associations;
- report statewide/county prevalence or production performance; or
- serve a prediction lookup or row-level score.
## Prediction unit and target ## Prediction unit and target
The prediction is made immediately before a new inspection episode begins. An The prediction is made immediately before a new inspection episode begins.
episode groups consecutive attempts no more than 30 days apart, and only its Attempts no more than 30 days apart form one episode. The supervised target is
first attempt is the supervised target. A vehicle must have at least one prior the episode's first attempt, and a vehicle must have at least one prior completed
completed episode to enter the returning-vehicle cohort. episode.
The binary target is: The binary target is recognized pass (`0`) versus recognized fail, reject, or
abort (`1`, collectively **non-pass**). Blank, null, and unrecognized results
remain unlabeled. Reject and abort may reflect process/readiness conditions, so
non-pass must not be paraphrased as mechanical failure.
- `0`: recognized pass; A recognized overall result is preferred. The documented older Utah County OBD
- `1`: recognized fail, reject, or abort; and exception may fill a blank overall result under a narrow source/program/test
- unlabeled: blank, null, or unrecognized results. contract. That value is a binary proxy only. Its provenance remains private
audit metadata and is not a predictor. See the
[modeling protocol](modeling_protocol.md#binary-label-contract).
A recognized overall result is preferred. For the older Utah County OBD feed, ## Development data
a controlled OBD result may fill a blank overall result only under the narrow
source/program/test contract documented in the
[modeling protocol](modeling_protocol.md#source-specific-label-contract). That
proxy is approved only for binary pass versus non-pass; it must not support a
four-class interpretation. Label provenance is private audit metadata and is
not a predictor.
## Development data and representativeness The extraction page-samples inspection records and retrieves complete histories
for up to 10,000 vehicles. After validation, the private run retained 9,996
vehicle tokens, 83,552 source events, 69,588 episodes, and 44,659 eligible
returning targets. These sample counts describe pipeline reconciliation only;
they do not imply population coverage.
The current extract samples inspection-table pages and then retrieves complete The sampling design over-represents vehicles with more inspection records. Feed
histories for up to 10,000 vehicles. This over-represents vehicles with more coverage changes by source and time and does not include all 29 Utah counties.
inspection records. The run contains 83,552 retained source events, 69,588 Accordingly, the model card makes no statewide prevalence claim, no county
episodes, and 44,659 eligible returning targets, but those counts do not make ranking, and no claim that sample subgroup differences generalize.
the sample population-representative.
Coverage is limited to participating county/source feeds and changes over time. ## Features and exclusions
It does not represent all 29 Utah counties. Source-system transitions, missing
years, and partial periods can resemble real changes in risk. See the
[data inventory](data_inventory.md) and
[development results](development_results.md) for the audited scope.
## Features and leakage exclusions The final model uses only pre-episode inspection-history information: vehicle
age, prior episode/attempt counts, time since prior history, prior non-pass rate,
prior first/final outcomes, previously observed make/model, public county
context, and season.
The selected model uses information available before the target episode: The model excludes current-test results and diagnostics, later attempts, station
and technician information, direct/pseudonymous identifiers, future records,
full-history leakage, DMV enrichment, and preprocessing learned from evaluation
periods. It does not use VIN, plate, ZIP, private token, raw JSON, credentials,
or operational records. The complete contract is in
[modeling_protocol.md](modeling_protocol.md#leakage-and-privacy-exclusions).
- vehicle age; ## Chronological evaluation
- prior episode and attempt counts;
- days since the prior episode and prior adverse outcome;
- prior non-pass rate;
- prior first and final outcomes;
- previously observed make and model;
- public county; and
- target season.
Categorical mappings and all preprocessing are fit on training data only. The | Period | Development-sample role |
current model does not yet include DMV enrichment or rich same-test OBD fields.
Excluded inputs include direct or pseudonymous identifiers, station
information, the target attempt's result or diagnostics, later attempts in the
target episode, future records, full-history aggregates, label provenance, and
preprocessing learned from evaluation periods. The complete exclusion contract
is in [modeling_protocol.md](modeling_protocol.md#leakage-exclusions).
## Chronology and holdout status
The fixed temporal design is:
| Period | Role |
| --- | --- | | --- | --- |
| 2010-2015 | Historical context only | | 2010-2015 | Historical context only |
| 2016-2022 | Fit preprocessing and models | | 2016-2022 | Fit preprocessing and logistic parameters |
| 2023 | Select hyperparameters and compare candidates | | 2023 | Select regularization and compare fixed candidates |
| 2024 | Fit probability calibration and inspect calibration behavior | | 2024 | Fit Platt scaling and inspect calibration behavior |
| 2025 | One-time development holdout | | 2025 | One-time development-sample holdout |
| 2026 partial | Shadow monitoring only | | 2026 partial | Shadow context only |
During live verification, the explicit 2025 gate was opened once after both The explicit 2025 gate was opened once after specifications were frozen from
candidate specifications and the tree search grid had already been fixed from pre-2025 data. No model was changed in response. This makes 2025 a one-time
pre-2025 data. No model was changed in response. The already-fixed logistic development holdout, not a pristine future test. No further 2025-informed tuning
model was then evaluated once for a direct comparison. Consequently, this is permitted.
page-sampled 2025 cohort is no longer a pristine unseen test, and no further
2025-informed tuning is permitted. Any complete-data 2025 analysis must be
described as confirmatory. The detailed audit trail is in
[development_results.md](development_results.md#one-time-development-holdout).
## Candidate comparison and selected model ## Development-sample performance
The candidates were training prevalence, the literal previous-episode outcome, The following values use non-audit rows from the **private 10,000-vehicle
regularized logistic regression, and histogram gradient boosting. Results below development sample**. They are **not population performance estimates**.
use non-audit development rows.
| Evaluation | Model | PR-AUC | Brier | ROC-AUC | Top-10% precision | | Sample evaluation | Model role | PR-AUC | Brier | ROC-AUC | Top-10% precision |
| --- | --- | ---: | ---: | ---: | ---: | | --- | --- | ---: | ---: | ---: | ---: |
| 2023 tuning | Training prevalence | 0.125 | 0.1092 | 0.500 | 0.125 | | 2023 tune | Training-prevalence baseline | 0.125 | 0.1092 | 0.500 | 0.125 |
| 2023 tuning | Previous episode | 0.166 | 0.1723 | 0.582 | 0.285 | | 2023 tune | Previous-episode baseline | 0.166 | 0.1723 | 0.582 | 0.285 |
| 2023 tuning | Logistic | **0.282** | **0.1012** | **0.707** | **0.315** | | 2023 tune | Logistic before calibration | **0.282** | **0.1012** | **0.707** | **0.315** |
| 2025 one-time holdout | Logistic + Platt | **0.261** | **0.1011** | **0.693** | **0.312** | | 2025 one-time holdout | Calibrated logistic, final | **0.261** | **0.1011** | **0.693** | **0.312** |
| 2025 one-time holdout | Gradient boosting + Platt | 0.238 | 0.1024 | 0.690 | 0.292 | | 2025 one-time holdout | Calibrated boosted tree, benchmark | 0.238 | 0.1024 | 0.690 | 0.292 |
The selected logistic model uses `C=0.03`. It satisfied explicit convergence Platt scaling was fit on the 2024 development-sample calibration partition. Its
and finite-value checks. Platt scaling was fit on the 2024 calibration cohort; same-partition PR-AUC of 0.257 and Brier score of 0.0936 are calibration
on that same cohort, PR-AUC was 0.257 and Brier score was 0.0936. Those 2024 diagnostics, not independent final performance.
values are calibration diagnostics, not independent final performance.
The logistic model remains selected because it beat both simple baselines and The final logistic model was retained because it beat the two simple baselines
the more complex tree on the one-time development holdout while remaining more on headline development metrics and outperformed the more complex tree in the
transparent. The separate never-fit vehicle audit also supported the logistic one-time sample holdout. The benchmark did not justify added complexity.
model, but the development sample is too limited for final generalization or
fairness claims.
## Coverage, fairness, and privacy limitations The never-fit private-vehicle stress-test subset contained 480 eligible 2025
sample episodes from 450 vehicles; calibrated logistic PR-AUC was 0.253 and
Brier score was 0.0859. These are also private-sample diagnostics, not
population or external-validation results.
- The model applies only to returning vehicles with recognizable labels and ## Limitations
sufficient prior history; cold-start behavior is not established.
- Geography and source era are entangled. Performance may shift when a feed,
county program, vehicle mix, or label process changes.
- Reject and abort outcomes can reflect readiness or process issues rather than
mechanical failure.
- No protected attributes are modeled, but their absence does not establish
fairness. Subgroup sample sizes, errors, and calibration still require review.
- Make/model normalization and complete-data subgroup analysis are unfinished.
- The current probability calibration has not yet received independent,
vehicle-clustered uncertainty analysis or external validation.
Private analytical artifacts remain local. Public assets contain only reviewed, - Results apply only to the sampled returning-vehicle cohort with recognizable
rounded aggregates that satisfy episode, distinct-vehicle, and binary-class labels and sufficient prior history.
suppression thresholds. Direct identifiers, private linkage values, raw rows, - Page sampling is not population-representative.
and row-level predictions are outside the public contract. See - Source, program, time, and geography are entangled.
[dashboard_spec.md](dashboard_spec.md#publication-controls). - Reject and abort are heterogeneous non-pass outcomes.
- Make/model aliases and incomplete feed coverage can distort cohorts.
- The one-time 2025 holdout is not an external validation dataset.
- No protected attributes are modeled, but that does not establish fairness.
- Sample calibration does not establish production calibration.
## Monitoring ## Privacy and public release
Before any deployment, monitoring must cover source-level volume, label Private analytical artifacts remain local and Git-ignored. Dashboard files
recognition, outcome prevalence, source/program mix, missingness, unseen contain only rounded aggregate cells that pass episode, distinct-vehicle, and
categories, score distributions, PR-AUC, Brier score, and calibration. Reviews binary-class suppression. The browser validates publication flags and checksums
must explicitly separate the Salt Lake source transition, newer rich feeds, DMV and fails closed.
coverage gaps, and the partial 2026 period. Alert thresholds and a response plan
remain to be defined. No VIN, plate, ZIP, station, technician identifier, private token, raw JSON,
credential, operational record, or row-level prediction belongs in a public
asset. The finished dashboard has no vehicle-level prediction service. See the
[dashboard specification](dashboard_spec.md#suppression-and-privacy-controls).
## Reproducibility ## Reproducibility
The pipeline uses fixed temporal boundaries and a fixed random seed. Python The pipeline uses fixed chronological boundaries and a fixed seed. Private
dependencies are pinned in [requirements.txt](../requirements.txt). Private manifests bind input lineage, configuration, convergence checks, row
manifests record input lineage, software/model configuration, convergence, row reconciliation, and checksums. Normal training commands keep the 2025 gate
reconciliation, and checksums without publishing private paths or values. SQL closed unless the explicit evaluation flag is supplied.
transforms and validation queries are versioned under [sql](../sql), model
runners are under [scripts](../scripts), and automated checks are under
[tests](../tests). Normal training commands leave the 2025 evaluation gate
closed unless an explicit flag is supplied.
## Remaining approval gates The detailed sample audit is in
[development_results.md](development_results.md), and the narrative conclusion
1. Run a complete, contiguous, bounded extraction and rebuild the frozen is in [final_report.md](final_report.md).
pipeline without page-sampling bias.
2. Treat complete-data 2025 results as confirmation and reserve a genuinely new
period or external dataset for future unseen evaluation.
3. Complete subgroup/source-era reporting, vehicle-clustered uncertainty,
calibration diagnostics, and fairness review.
4. Finish make/model normalization, episode-gap sensitivity checks, and the
preregistered inspection-only versus DMV-enhanced ablation.
5. Pass privacy and publication review before replacing the development-preview
aggregates or enabling any prediction lookup.
6. Define monitoring thresholds, ownership, rollback criteria, and a model
update policy before operational use.

198
docs/modeling_protocol.md
View File

@ -1,141 +1,133 @@
# Leakage-safe modeling protocol # Leakage-safe modeling protocol
## Reporting boundary
This protocol governs the private, page-sampled 10,000-vehicle development
cohort. All resulting counts, rates, comparisons, and performance metrics are
**sample results, not population estimates**. County/source reporting is for
coverage and drift context only; it must not be turned into population county
rankings.
## Prediction unit ## Prediction unit
Score the **first attempt of the next inspection episode** immediately before Estimate the probability that the **first attempt of a returning vehicle's next
check-in. Consecutive attempts for the same private vehicle token belong to the inspection episode is a non-pass**, immediately before the episode begins.
same episode when they are no more than 30 days apart. Re-run the analysis with Consecutive attempts for the same private vehicle token form one episode when
14- and 45-day gaps as sensitivity checks. they are no more than 30 days apart. Rapid retests are not separate targets.
This prevents rapid fail/retest sequences from dominating the target. Those This is the sole research target for the completed prototype. Cold-start,
within-episode attempts belong in the separate failure-to-pass journey analysis. four-class, fail-only, failure-to-pass, and individualized prediction tasks are
out of scope.
## Eligibility ## Eligibility
- Use 2010-2015 events only as historical context; supervised targets begin in - Use 2010-2015 events only as historical context; supervised targets begin in
2016. 2016.
- Exclude the four 1990 date outliers. - Exclude the four known 1990 date outliers.
- Require a recognized first-attempt outcome and at least one prior completed - Require a recognized first-attempt binary outcome and at least one prior
episode for the returning-vehicle model. completed episode.
- Keep first-observed vehicles as a separate cold-start cohort. - Keep first-observed vehicles outside the modeled returning-vehicle cohort.
- Normalize `slc` and `slco` to Salt Lake County for geography while preserving - Normalize `slc` and `slco` to Salt Lake County for geography while preserving
source era for drift reporting. source era for diagnostics.
- Deduplicate exact uploads and quarantine shared/test/placeholder identifiers - Deduplicate exact uploads and quarantine shared, test, or placeholder
using preregistered rules for impossible conflicts or extreme activity. identifiers under preregistered conflict/activity rules.
- Do not require a DMV match; retain explicit match and staleness indicators.
Every exclusion must appear in a cohort-flow report. Every exclusion appears in the private cohort-flow audit.
## Source-specific label contract ## Binary label contract
The normalized target prefers a recognized `overall_result` for every source. The target is `0` for a recognized pass and `1` for recognized fail, reject, or
The older Utah County feed is the sole exception: when its overall result is abort. Blank, null, and unrecognized values remain unlabeled rather than being
blank, `obd_result` may supply the binary target only when the source is treated as passes.
`utah`, the program is `obd`, the test type is `OBD`, and the controlled value
is pass, fail, reject, or abort. `B`, blank, TSI, `other/C`, and unknown values
remain unlabeled. Every target retains `target_outcome_label_source` as private
audit metadata, and that field is never a predictor.
This is a binary pass-versus-non-pass proxy, not a four-class substitution. The normalized target prefers a recognized `overall_result`. The older Utah
Across non-Utah feeds where both fields are recognized, an aggregate audit found County feed is the sole exception: when its overall result is blank,
99.30% agreement on pass versus non-pass, while the fail/reject distinction was `obd_result` may supply the binary label only when source is `utah`, program is
not interchangeable. Utah program rules also distinguish a readiness rejection `obd`, test type is `OBD`, and the controlled value is pass, fail, reject, or
from a failed inspection. Future four-class analysis must therefore require abort. `B`, blank, TSI, `other/C`, and unknown values remain unlabeled. Private
`target_outcome_label_source = 'overall_result'`. See the official label provenance is retained for audit and is never a predictor or public
[Utah inspection requirements](https://dmv.utah.gov/register/inspections/) and field.
[program definition of rejection](https://www.utah.gov/pmn/files/1155003.pdf).
This exception is a binary pass/non-pass proxy. It does not justify interpreting
fail, reject, and abort as interchangeable mechanical conditions.
## Point-in-time features ## Point-in-time features
Every historical window ends strictly before the target episode: Every feature window ends strictly before the target episode. The final
inspection-history feature set includes:
- vehicle age;
- prior episode and attempt counts; - prior episode and attempt counts;
- previous episode first/final outcome; - days since the prior episode and prior adverse outcome;
- expanding and trailing prior pass/fail/reject/abort counts and rates; - prior non-pass rate;
- days since the previous episode and prior adverse result; - prior episode first and final outcomes;
- attempts required in prior episodes; - previously observed make and model;
- vehicle age and canonical make/model from prior information; - public county context; and
- physical county and target month/season; - target season.
- latest DMV record dated before the target, registration count/recency, fuel,
and DMV match/staleness indicators; and
- missingness indicators.
The stable core model is inspection-only. The DMV-enhanced model is an explicit The final prototype does not use DMV enrichment or rich current-test OBD data.
ablation because DMV history ends in March 2024 and is nearly absent in 2021.
## Leakage exclusions ## Leakage and privacy exclusions
The MVP must not use: The model and public product exclude:
- the target attempt's overall/OBD result, result reason, DTC count, MIL or - the target attempt's result, reason, DTC count, MIL/readiness state, PIDs,
readiness values, PIDs, visual checks, measurements, certificate, or visual checks, measurements, certificate, or calibration fields;
calibration fields;
- later attempts or eventual outcome from the target episode; - later attempts or eventual outcome from the target episode;
- station or station-level outcome statistics; - station, technician, or station-level outcome information;
- raw VIN, private token, plate, ZIP, or other identifiers as features; - raw VIN, private vehicle token, plate, ZIP, exact address, or another direct
- target-row vehicle attributes when a prior/static source is available; or pseudonymous identifier as a feature;
- future DMV records; - future records, full-history aggregates, or target-row information that was
- full-history aggregates; or not available before the episode;
- preprocessing, category mappings, target encodings, or imputation learned - preprocessing, category mappings, imputation, or calibration learned from a
from validation/test data. later partition; and
- raw JSON, credentials, operational records, and row-level predictions in any
public artifact.
Timestamp ties must be resolved before lag/window calculations. Cumulative Timestamp ties are resolved before lag/window calculations. Cumulative windows
windows end at the preceding event or episode. end at the preceding event or episode.
## Fixed evaluation timeline ## Frozen chronology
| Partition | Target dates | Purpose | | Partition | Target dates | Role in the private 10,000-vehicle development sample |
| --- | --- | --- | | --- | --- | --- |
| Historical context | 2010-2015 | Lag features only | | Historical context | 2010-2015 | Lag features only |
| Train | 2016-2022 | Fit preprocessing and models | | Train | 2016-2022 | Fit preprocessing and model parameters |
| Tune | 2023 | Hyperparameters and selection | | Tune | 2023 | Select logistic regularization; compare fixed candidates |
| Calibrate | 2024 | Probability calibration and thresholds | | Calibrate | 2024 | Fit Platt scaling and inspect calibration behavior |
| Locked test | 2025 | Final reported performance | | One-time holdout | 2025 | Development comparison after specifications were frozen |
| Shadow drift | 2026-01-01 to 2026-06-22 | Monitoring only | | Shadow | 2026-01-01 to 2026-06-22 | Partial-period monitoring context only |
The page-sampled development extract's 2025 gate was opened once during live The 2025 gate was opened once during live verification. It is no longer a
verification after both candidate specifications were fixed. It is therefore pristine unseen test, no 2025-informed tuning is permitted, and every reported
a one-time development holdout, not a pristine future test. No model change was 2025 value must say “one-time development-sample holdout.” A later complete-data
made from it; a later complete-data 2025 run is confirmatory. See 2025 analysis would be confirmatory, not a new final test.
[development_results.md](development_results.md) for the audit trail. The table
continues to define the frozen chronology for a complete extraction.
Repeated vehicles may cross ordinary time partitions because returning-vehicle Repeated vehicles may cross chronological partitions because returning-vehicle
prediction is the deployment scenario. Separately reserve 10% of keyed vehicle prediction is the intended scenario. Separately held-out private vehicle buckets
buckets as a never-fit VIN audit and report its 2025 performance as an unseen- form a never-fit stress test. Random row splitting is prohibited.
vehicle stress test. Never use random row splitting.
## Baselines and candidate model ## Models and decision rule
1. Training prevalence 1. Training prevalence: probability baseline.
2. Repeat the previous episode's first outcome 2. Previous episode's first outcome: literal history baseline.
3. Regularized logistic regression with age splines and one-hot categoricals 3. Regularized logistic regression: selected using the 2023 sample partition,
4. A boosted-tree model then Platt-calibrated on the 2024 sample partition; **the final model**.
5. Inspection-only versus inspection-plus-DMV ablation 4. Histogram gradient boosting with Platt calibration: fixed nonlinear
**benchmark only**.
Keep validation and test sets at natural prevalence. If pass rows are sampled The benchmark does not power a separate dashboard experience. Added complexity
for training, preserve sampling probabilities and recalibrate on the untouched would require a clear, predeclared improvement to displace the logistic model;
2024 partition. the one-time 2025 sample comparison did not show that improvement.
## Metrics ## Metrics and interpretation
Headline binary metrics: Headline metrics are non-pass PR-AUC and Brier score. Log loss, ROC-AUC, and
precision at the highest-scored 10% are supporting diagnostics. Calibration
values computed on the 2024 partition used to fit Platt scaling are calibration
diagnostics, not independent final performance.
- non-pass PR-AUC; All tables and charts must state that they use the private 10,000-vehicle
- Brier score and log loss; development sample and are not population estimates. Report sample support with
- calibration intercept, slope, and reliability curve; every subgroup result. Do not describe observed differences as causal, do not
- precision, recall, and lift at fixed review capacities; and claim generalization to all Utah vehicles, and do not publish county rankings.
- ROC-AUC as secondary context.
Use vehicle-clustered bootstrap confidence intervals. Report results by county,
source era, vehicle-age band, prior outcome, fuel, history depth, and DMV
match/staleness. Four-class analysis adds class-specific and macro PR-AUC,
multiclass log loss/Brier score, calibration, and a confusion matrix.
## Drift contract
Monitor monthly/source-level volume, label recognition, outcome prevalence,
blank rate, source/program mix, missingness, DMV staleness, unseen categories,
prediction distribution, PR-AUC, Brier score, and calibration. Treat the
`slc`→`slco` transition, new rich feeds, DMV 2021 gap, DMV 2024 endpoint, and
partial 2026 period as explicit stress cases rather than ordinary random drift.

170
docs/presentation_outline.md Normal file
View File

@ -0,0 +1,170 @@
# Utah Vehicle Health: 10-minute presentation outline
## Presentation rule
Keep this sentence visible on every results or dashboard slide:
> Private 10,000-vehicle development sample; sample results, not population
> estimates.
Do not show county rankings, imply statewide coverage, or describe non-pass as
mechanical failure. Refer to calibrated logistic regression as the **final
model** and histogram gradient boosting as the **benchmark**.
## Timed outline
| Time | Slide / action | Core message |
| --- | --- | --- |
| 0:00-0:40 | 1. Title and boundary | One focused prediction question; development prototype, not a population study or decision tool |
| 0:40-1:25 | 2. Research question | Returning vehicle, next episode, first attempt, binary non-pass |
| 1:25-2:15 | 3. Development data | Private page-sampled 10,000-vehicle histories; complete sampled histories but nonrepresentative selection |
| 2:15-3:10 | 4. Leakage-safe design | 30-day episodes, only pre-episode history, chronological split, never random rows |
| 3:10-4:20 | 5. Model comparison | Logistic final versus prevalence/previous-outcome baselines and boosted-tree benchmark |
| 4:20-4:50 | 6. Model decision | Calibrated logistic wins the declared sample comparison and is easier to explain |
| 4:50-7:35 | Live dashboard demo | Overview, Sample cohorts, Model & benchmark, Data & methods |
| 7:35-8:30 | 7. Privacy architecture | Private pipeline to suppressed aggregate JSON to dashboard-only Bolt project |
| 8:30-9:20 | 8. Limitations | Sampling, feed coverage, source/time confounding, heterogeneous non-pass, one-time holdout |
| 9:20-10:00 | 9. Close | Prototype is complete and honest about what it can—and cannot—claim |
Total: **10:00**
## Slide notes
### 1. Title and boundary — 40 seconds
**Title:** Utah Vehicle Health
**Subtitle:** Predicting a returning vehicle's next-episode first-attempt
non-pass outcome
Say:
> This is a presentation-ready development prototype built from a private,
> page-sampled 10,000-vehicle cohort. Every result I show is sample-based, not a
> population estimate. The project does not rank counties or diagnose vehicles.
### 2. Research question — 45 seconds
Show the question:
> Using only information available before an inspection episode begins, how
> well can a calibrated logistic regression estimate whether a returning
> vehicle's next episode will have a first-attempt non-pass outcome?
Explain that attempts separated by no more than 30 days form one episode. This
keeps rapid fail/retest sequences from becoming repeated target rows. Non-pass
combines fail, reject, and abort; blanks remain unlabeled.
### 3. Development data — 50 seconds
Show a compact flow:
```text
sample up to 10,000 vehicles -> retrieve their histories -> build episodes
-> create 44,659 eligible returning targets
```
Say that 9,996 private tokens remained after validation, producing 83,552 source
events and 69,588 episodes in the private sample. Immediately repeat that these
are reconciliation counts, not Utah totals. Page sampling over-represents
vehicles with more inspection records, and participating feeds do not cover all
29 counties.
### 4. Leakage-safe design — 55 seconds
Show the chronology:
```text
2010-15 context | 2016-22 train | 2023 tune | 2024 calibrate
| 2025 one-time holdout | 2026 partial shadow
```
Explain that every feature window ends before the target episode. The model uses
prior outcomes, history depth, timing, age, previously observed make/model,
public county context, and season. It excludes current-test diagnostics, future
records, identifiers, stations, technicians, and later attempts.
Note that the 2025 sample gate was opened once after specifications were frozen;
it is a development holdout, not a pristine future test.
### 5. Model comparison — 70 seconds
Use a two-metric table. Label it “Private 10,000-vehicle development sample; not
population performance.”
| 2025 one-time sample holdout | Role | PR-AUC | Brier |
| --- | --- | ---: | ---: |
| Training prevalence | Baseline | 0.123 | 0.1076 |
| Previous episode | Baseline | 0.158 | 0.1731 |
| Logistic + Platt | **Final** | **0.261** | **0.1011** |
| Boosted tree + Platt | Benchmark | 0.238 | 0.1024 |
Say that higher PR-AUC and lower Brier are better. Logistic beat both baselines
and the boosted-tree benchmark in this one-time sample comparison. Avoid the
word “accuracy,” which obscures class imbalance and calibration.
### 6. Model decision — 30 seconds
Say:
> The final model is calibrated logistic regression. The boosted tree is a
> benchmark only. It did not improve the declared sample metrics enough to
> justify greater complexity, while logistic regression is easier to audit and
> explain.
### Live dashboard demo — 2 minutes 45 seconds
Follow [demo_script.md](demo_script.md). Keep the development-preview banner in
view at the start. Explicitly show calibrated logistic as final, the tree as
benchmark-only, and the absence of a vehicle-level prediction service.
### 7. Privacy architecture — 55 seconds
Show:
```text
private read-only data -> local modeling -> suppressed aggregate JSON
-> dashboard-only Bolt project
```
State that the public bundle has no VINs, plates, ZIPs, stations, technician
identifiers, private tokens, raw JSON, credentials, operational records, or
row-level predictions. The browser has no database connection and fails closed
if its aggregate contract does not validate.
### 8. Limitations — 50 seconds
Name the limitations directly:
- page-sampled cohort is not population-representative;
- feeds are partial and change over time;
- geography and source era are confounded;
- reject/abort do not necessarily mean mechanical failure;
- 2025 is a one-time development holdout, not external validation; and
- sample associations do not support county rankings or causal conclusions.
### 9. Close — 40 seconds
End with:
> The prototype is complete at the development stage: one precise question, a
> leakage-safe timeline, calibrated logistic regression as the final model, a
> boosted-tree benchmark, and a privacy-reviewed static dashboard. Its value is
> not just the sample performance; it is the discipline of making only the
> claims the data and release boundary can support.
Pause, then invite questions about episode construction, calibration, privacy,
or the dashboard—not about county “winners” and “losers,” which this sample
cannot establish.
## Presentation checklist
- Start the local dashboard before presenting: `node dashboard/server.mjs`.
- Open `http://127.0.0.1:4173/#overview` and close unrelated browser tabs.
- Use a fresh page load to confirm aggregate validation succeeds.
- Keep a screenshot or short recording as a backup, with the sample warning
visible.
- Never open private data, artifacts, terminal environment variables, database
tools, or browser developer tools during the talk.
- Do not type or display a VIN, plate, ZIP, station, technician identifier, raw
record, or row-level prediction.

174
docs/project_charter.md
View File

@ -1,122 +1,110 @@
# Utah Vehicle Health project charter # Utah Vehicle Health project charter
## Working title ## Prototype status
**Utah Vehicle Health** **Presentation-ready development prototype.** Every analytical finding and every
dashboard value comes from the private, page-sampled 10,000-vehicle development
*What millions of emissions inspections reveal about the next test across the cohort. These values are **sample results, not population estimates**. They must
state of Utah county feeds available in this dataset.* not be used to estimate statewide or county prevalence, create county rankings,
or make decisions about an individual vehicle.
## Product promise ## Product promise
Explain how vehicle age, type, location, and prior inspection history relate to Utah Vehicle Health explains how information available before an inspection
the chance of passing the first attempt of the next emissions-inspection relates to the first-attempt non-pass outcome of a returning vehicle's next
episode. inspection episode.
The product measures **emissions-inspection outcomes**. It must not describe its The product measures an emissions-inspection outcome. It does not measure
score as a diagnosis of overall mechanical reliability, roadworthiness, safety, overall vehicle health, mechanical reliability, safety, roadworthiness, or
or legal compliance. legal compliance.
## Primary research question ## Primary research question
> Using only information available before an inspection episode begins, how > Using only information available before an inspection episode begins, how
> accurately and reliably can we estimate whether its first attempt will pass? > well can a calibrated logistic regression estimate whether a returning
> vehicle's next episode will have a first-attempt non-pass outcome?
An episode groups attempts for the same internal vehicle token when the gap An episode groups consecutive attempts for the same private vehicle token when
between consecutive attempts is 30 days or less. The target is the first attempt the gap is 30 days or less. The target is the first attempt of a new episode,
of a new episode, not every rapid retest. not a rapid retest. A vehicle is eligible only after at least one completed
prior episode.
## Primary target ## Binary target
- `0 — pass`: normalized `PASS` or `P` - `0 - pass`: normalized `PASS` or `P`
- `1 — non-pass`: normalized `FAIL`, `F`, `REJECT`, or `ABORT` - `1 - non-pass`: normalized `FAIL`, `F`, `REJECT`, or `ABORT`
- Blank, null, and unrecognized results are unlabeled. They remain in chronology - unlabeled: blank, null, and unrecognized results
and coverage reporting but are excluded as supervised targets.
Because reject and abort can reflect process/readiness problems rather than Reject and abort can reflect process or readiness conditions rather than a
mechanical failure, the project will also report: mechanical failure. The prototype therefore says **non-pass**, never “vehicle
failure,” when referring to the combined target. Multiclass outcome modeling
and fail-only sensitivity analysis are outside the finished prototype scope.
- a four-class pass/fail/reject/abort analysis; and ## Data and product boundary
- a fail-versus-pass sensitivity analysis that excludes reject and abort.
## Scope The development cohort contains complete inspection histories for up to 10,000
sampled vehicles from participating Utah county/source feeds. Page sampling
over-represents vehicles with more inspection records, so neither the cohort nor
its dashboard aggregates are population-representative. The feeds do not cover
all 29 Utah counties.
The returning-vehicle MVP uses universal inspection-history fields and prior The final model is inspection-history only. DMV enrichment, rich current-test
DMV information where it is point-in-time valid. It covers participating OBD fields, station effects, cold-start prediction, and individualized lookup
inspection county/source feeds, not all 29 Utah counties. are outside scope. The static dashboard publishes only rounded, suppressed
sample aggregates and pre-2025 development diagnostics.
The rich OBD/odometer fields are a later, source-specific extension for newer ## Final model and benchmark
`slco`, `davis`, and `cache` records. They are not part of the statewide-style
historical baseline.
## Deliverables The final model is regularized logistic regression (`C=0.03`) with Platt
probability calibration fit on the 2024 development-sample partition. A
histogram gradient-boosted tree is retained only as a nonlinear benchmark. It
is not a second final model and is not used to drive the product.
1. A reproducible, read-only extraction and private pseudonymization pipeline. ## Completed deliverables
2. A point-in-time episode/feature mart with an auditable exclusion report.
3. Prevalence, previous-outcome, and logistic-regression baselines.
4. One calibrated boosted-tree model and an inspection-only versus DMV-enhanced
ablation.
5. A locked temporal evaluation with subgroup and source-era diagnostics.
6. Versioned, suppressed public aggregates for a four-page Bolt dashboard.
7. A model card and data/methodology page documenting limitations.
## Headline success criteria 1. Read-only extraction with private vehicle pseudonymization.
2. Leakage-safe episode and point-in-time feature mart.
3. Training-prevalence and previous-episode baselines.
4. Calibrated logistic regression as the final model.
5. Histogram gradient boosting as a benchmark only.
6. Chronological development evaluation with an explicit one-time 2025 gate.
7. Suppression-reviewed static dashboard assets with fail-closed validation.
8. Model card, final report, presentation materials, and dashboard-only Bolt
deployment instructions.
- Beat both the training-prevalence and previous-episode-outcome baselines on ## Acceptance criteria
2025 PR-AUC and Brier score.
- Produce calibrated probabilities, not just class labels.
- Report performance by county/source era, vehicle-age band, history depth, and
DMV match/staleness.
- Reproduce all published charts from versioned sanitized outputs.
- Export no VIN, plate, ZIP, station, raw JSON, operational record, or
row-level prediction to Bolt.
- Suppress public cells with fewer than 100 eligible inspections or fewer than
10 observations in an outcome or its complement, with complementary
suppression where totals could reveal a hidden cell.
## Initial feasibility result - The prediction unit remains a returning vehicle's next-episode first attempt.
- Model features exist before that episode begins.
Under label contract v3, the fixed-seed aggregate-only query in - The calibrated logistic model beats both simple baselines on development-
`sql/10_episode_cohort_feasibility.sql` produced 8,916 eligible sample PR-AUC and Brier score.
returning-vehicle episodes from 1,730 of 2,000 sampled vehicle histories. The - The boosted tree is presented only as a benchmark.
first-attempt non-pass rate was 11.80%, and the median gap from the prior - Every displayed value is labeled as a private-sample result and not a
episode was about 372 days. Of those targets, 1,732 use the documented Utah OBD population estimate.
binary proxy because that source leaves its overall-result field blank. This - County views communicate feed coverage and sample context; they do not claim
supports both the episode definition and a calibrated binary model. The sample population rankings or causal county differences.
is for pipeline feasibility, not a population estimate. - No VIN, plate, ZIP, station, technician identifier, raw JSON, credential,
operational record, private token, or row-level prediction reaches the
The 2025 sample showed a longer median gap and different outcome mix, reinforcing dashboard or presentation materials.
the need for source-era drift reporting and a locked chronological test. - The dashboard contains no vehicle-level estimator, input form, or prediction
service.
## Development holdout status
During live verification, the explicit 2025 gate was opened once on the
page-sampled development extract after both candidate specifications had been
fixed from pre-2025 data. No test-informed model change was made. The frozen
logistic model remains selected, and a later complete-data 2025 analysis will
be treated as confirmatory rather than described as a pristine unseen test.
The audit trail and results are recorded in
[development_results.md](development_results.md).
## Non-goals ## Non-goals
- Diagnosing an individual vehicle - Predicting the result of a first-observed vehicle
- Certifying that a vehicle will pass - Diagnosing, certifying, or guaranteeing an individual vehicle outcome
- Ranking or accusing inspection stations - Ranking counties, programs, stations, technicians, owners, or vehicles
- Identifying owners or accepting VIN/plate input - Making causal claims about geography, vehicle makes, or inspection programs
- Making causal claims about county programs - Accepting VIN, plate, address, station, or diagnostic inputs
- Treating missing outcomes as passes - Publishing population prevalence from the development sample
- Shipping a production decision service or live database connection
- Completing DMV enrichment, four-class modeling, or failure-to-pass journeys
## Staged build ## Evaluation status
1. **Foundation:** episode definition, label normalization, exclusions, secure The explicit 2025 gate was opened once after model specifications were frozen
extraction, and coverage checks. from pre-2025 data. No 2025-informed model change was made. The 2025 values are
2. **Baseline:** inspection-history feature mart and transparent baselines. therefore a one-time **development-sample holdout**, not a pristine future test
3. **Enrichment:** canonical vehicle dimension and point-in-time DMV features. and not population performance. See [development_results.md](development_results.md)
4. **Modeling:** tree model, calibration, locked test, drift and subgroup audits. for the audit trail and [modeling_protocol.md](modeling_protocol.md) for the
5. **Product:** sanitized aggregate export and Bolt MVP. frozen evaluation contract.
6. **Stretch:** failure-to-pass journeys, multiclass probabilities, and a
clearly scoped rich-OBD model.
See [modeling_protocol.md](modeling_protocol.md) and
[dashboard_spec.md](dashboard_spec.md) for the detailed contracts.

7
docs/project_options.md
View File

@ -2,6 +2,13 @@
Inventory date: 2026-07-15 Inventory date: 2026-07-15
> **Archived planning context:** This shortlist records ideas considered during
> discovery; it is not the scope of the finished prototype. DMV enrichment,
> multiclass outcomes, station analysis, forecasting, and population/county
> rankings below are not current deliverables. The authoritative scope is the
> [project charter](project_charter.md), and the completed development story is
> in the [final report](final_report.md).
> **Selected:** Option 1, Utah Vehicle Health. The implementation contract is in > **Selected:** Option 1, Utah Vehicle Health. The implementation contract is in
> [project_charter.md](project_charter.md). > [project_charter.md](project_charter.md).

13
scripts/export_dashboard_data.py
View File

@ -97,6 +97,14 @@ REQUIRED_MART_COLUMNS = {
"last_observed_model", "last_observed_model",
} }
DENIED_EXACT_KEYS = { DENIED_EXACT_KEYS = {
"credential",
"credentials",
"inspector",
"inspector_id",
"password",
"secret",
"technician",
"technician_id",
"vehicle_token", "vehicle_token",
"vehicle_bucket", "vehicle_bucket",
"is_vin_audit", "is_vin_audit",
@ -122,6 +130,11 @@ DENIED_EXACT_KEYS = {
"row_count", "row_count",
} }
DENIED_KEY_FRAGMENTS = ( DENIED_KEY_FRAGMENTS = (
"credential",
"inspector",
"password",
"secret",
"technician",
"vehicle_token", "vehicle_token",
"internal_event", "internal_event",
"license_plate", "license_plate",

28
scripts/train_baselines.py
View File

@ -89,6 +89,17 @@ PLATT_CALIBRATION_CONFIG = {
# These fields are forbidden even when they are not selected as model features. # These fields are forbidden even when they are not selected as model features.
# Their presence indicates that the input is not the approved episode mart. # Their presence indicates that the input is not the approved episode mart.
FORBIDDEN_EXACT_COLUMNS = { FORBIDDEN_EXACT_COLUMNS = {
"credential",
"credentials",
"inspector",
"inspector_id",
"password",
"prediction",
"probability",
"row_prediction",
"secret",
"technician",
"technician_id",
"vin", "vin",
"plate", "plate",
"zip", "zip",
@ -109,7 +120,14 @@ FORBIDDEN_EXACT_COLUMNS = {
"target_final_outcome", "target_final_outcome",
} }
FORBIDDEN_COLUMN_FRAGMENTS = ( FORBIDDEN_COLUMN_FRAGMENTS = (
"credential",
"inspector",
"license_plate", "license_plate",
"password",
"raw_json",
"row_prediction",
"secret",
"technician",
"certificate", "certificate",
"calibration_id", "calibration_id",
"visual_check", "visual_check",
@ -122,17 +140,27 @@ FORBIDDEN_COLUMN_FRAGMENTS = (
FORBIDDEN_UNKNOWN_COLUMN_TOKENS = { FORBIDDEN_UNKNOWN_COLUMN_TOKENS = {
"address", "address",
"certificate", "certificate",
"credential",
"dtc", "dtc",
"email", "email",
"inspector",
"ip", "ip",
"json",
"mil", "mil",
"obd", "obd",
"owner", "owner",
"password",
"payload",
"pid", "pid",
"plate", "plate",
"prediction",
"probability",
"raw",
"readiness", "readiness",
"secret",
"session", "session",
"station", "station",
"technician",
"user", "user",
"vin", "vin",
"zip", "zip",

14
tests/test_baselines.py
View File

@ -104,6 +104,20 @@ class SchemaTests(unittest.TestCase):
with self.assertRaisesRegex(baselines.SchemaError, "owner_zip_code"): with self.assertRaisesRegex(baselines.SchemaError, "owner_zip_code"):
baselines.validate_schema(columns) baselines.validate_schema(columns)
def test_prohibited_sensitive_columns_fail_closed(self) -> None:
for column in (
"technician_id",
"inspection_technician_code",
"inspector_identifier",
"database_password",
"api_secret_value",
"raw_inspection_json",
"row_prediction_probability",
):
with self.subTest(column=column):
with self.assertRaisesRegex(baselines.SchemaError, column):
baselines.validate_schema(list(valid_record().keys()) + [column])
def test_duplicate_columns_fail_closed(self) -> None: def test_duplicate_columns_fail_closed(self) -> None:
columns = list(valid_record().keys()) + ["vehicle_token"] columns = list(valid_record().keys()) + ["vehicle_token"]
with self.assertRaisesRegex(baselines.SchemaError, "duplicate"): with self.assertRaisesRegex(baselines.SchemaError, "duplicate"):

13
tests/test_dashboard_export.py
View File

@ -193,6 +193,19 @@ class DashboardExportTests(unittest.TestCase):
"development_preview": True, "development_preview": True,
"population_estimate_allowed": False, "population_estimate_allowed": False,
} }
for unsafe_field in (
"technician_id",
"inspection_technician_code",
"inspector_identifier",
"database_password",
"api_secret_value",
):
with self.subTest(unsafe_field=unsafe_field):
with self.assertRaises(dashboard_export.PublicSchemaError):
dashboard_export._scan_public_value(
{unsafe_field: "not-public"}
)
unsafe_key = { unsafe_key = {
**base, **base,
"rows": [ "rows": [