128 lines
6.7 KiB
Markdown
128 lines
6.7 KiB
Markdown
# Countydata inventory
|
|
|
|
Inventory date: 2026-07-15; live schema revalidated 2026-07-21. All inspection
|
|
was performed with read-only transactions, metadata queries, aggregate queries,
|
|
and bounded JSON field-path sampling; no identifier values were exported.
|
|
|
|
## Live access and full-schema boundary
|
|
|
|
The July 21 validation connected successfully from this VS Code workspace using
|
|
TLS 1.3 and a session forced into read-only mode. It found 24 visible non-system
|
|
relations in `countydata`. The repeatable metadata-only inventory is
|
|
`scripts/inventory_metadata.py`, exposed as the VS Code task **Countydata:
|
|
inventory metadata safely**.
|
|
|
|
The analytical `data` schema contains 10 normalized tables. Four
|
|
`fdw_countydata` relations and the `fdw_data.dmv_tax_record*` relations are
|
|
source mirrors rather than additional populations. The remaining operational
|
|
relations contain API-key names and secrets, application usernames/emails and
|
|
password hashes, invitations, sessions, IP addresses, event logs, upload
|
|
contents, and ingest state. Their field names were inventoried, but their row
|
|
values were not read because they are unrelated to the project and sensitive.
|
|
|
|
A bounded raw-JSON field-path sample found vehicle attributes plus VIN and
|
|
plate identifiers, station/certificate/calibration fields, technician-license
|
|
numbers, odometer, engine/fuel/GVWR/transmission fields, DTC/PID arrays,
|
|
readiness monitors, MIL state, and visual inspection results. No owner-name
|
|
field was observed in that sample. This does not prove that every historical
|
|
source file has the same shape, so raw JSON remains private and out of the
|
|
public application.
|
|
|
|
## Accessible databases
|
|
|
|
- `countydata`: the useful analytical database, approximately 93.4 GB.
|
|
- `postgres`: empty local/staging copies of the core vehicle table structure
|
|
plus foreign-table links. It is not the analytical source.
|
|
- `vattp`: a tiny, unrelated course/event registration and survey application.
|
|
It is not suitable for this project and its application records should remain
|
|
out of scope.
|
|
|
|
## DMV logical dataset
|
|
|
|
The following are one logical dataset split into projections joined one-to-one
|
|
by `id`; they are not four independent populations.
|
|
|
|
| Relation | Approximate role | Key analytical fields |
|
|
| --- | --- | --- |
|
|
| `data.dmv_tax_json` | Raw JSON and ingest metadata | Full DMV record, file/source metadata |
|
|
| `data.dmv_tax` | Core lookup | VIN, registration date, county |
|
|
| `data.dmv_tax_search` | Search projection | VIN, registration date, county |
|
|
| `data.dmv_tax_vehicle` | Vehicle projection | Make, model, model year, fuel, registration type/place, temporary flag, expiration/emission dates, ZIPs |
|
|
|
|
Exact current logical row count: **18,009,278**.
|
|
|
|
- Date bounds are 2011-09-07 through 2024-03-13, but the few pre-2016 rows are
|
|
outliers and there are only 117 records in 2021.
|
|
- The 29 Utah counties are represented. Salt Lake (35.66%), Utah (17.21%),
|
|
Davis (10.29%), and Weber (8.01%) account for most records.
|
|
- Canonicalized fuel mix by record is 86.23% gasoline, 7.97% diesel, 2.59%
|
|
flexible fuel, 2.06% hybrid, 0.70% electric, and 0.24% plug-in hybrid.
|
|
- A vehicle appears repeatedly over time: sampled DMV histories had a median of
|
|
five registration records.
|
|
|
|
## Inspection logical dataset
|
|
|
|
These are likewise projections of one logical inspection dataset joined by
|
|
`id`.
|
|
|
|
| Relation | Approximate role | Key analytical fields |
|
|
| --- | --- | --- |
|
|
| `data.inspection_json` | Raw JSON and ingest metadata | Full inspection, detailed OBD/readiness/visual fields where supplied |
|
|
| `data.inspection` | Core lookup | VIN, test timestamp, ingest timestamp |
|
|
| `data.inspection_search` | Outcome/search projection | County/source, overall and OBD results, test/program type |
|
|
| `data.inspection_vehicle` | Vehicle projection | Make, model, year, calibration/certificate, station |
|
|
| `data.inspection_obd` | OBD summary | Result-reason code, DTC count, permanent-DTC flag |
|
|
| `data.inspection_plate` | Identifier lookup | VIN, plate, test timestamp; sensitive and unnecessary for analytics |
|
|
|
|
Exact current logical row count: **19,357,287**.
|
|
|
|
- Four dates in 1990 are outliers. Normal coverage begins in 2010 and continues
|
|
through 2026-06-22; 2026 is partial.
|
|
- Real source/county labels are `slc`, `slco`, `utah`, `weber`, `davis`, and
|
|
`cache`. The two Salt Lake labels represent different source eras and should
|
|
not be blindly treated as different counties.
|
|
- Overall results are 71.06% pass, 3.61% fail, 3.56% reject, 2.31% abort, and
|
|
19.45% blank/null or other near-blank values in the raw `overall_result`
|
|
field. Most missing overall results belong to the older Utah County feed; its
|
|
audited OBD/OBD rows carry a separate result that is usable only as a
|
|
provenance-tagged binary pass-versus-non-pass proxy.
|
|
- About 89.5% of records use the OBD program and about 9.2% use TSI.
|
|
- DTC count is zero in about 90.4% of records and null in about 1.4%; the
|
|
remaining values are class-imbalanced and include rare data-quality outliers.
|
|
- Raw JSON can contain odometer, vehicle fuel/type/GVWR/cylinders/engine,
|
|
transmission, DTCs, PIDs, MIL/readiness status, communication protocol, and
|
|
visual inspection fields. These richer fields are concentrated in the newer
|
|
`slco`, `davis`, and `cache` feeds rather than statewide history.
|
|
|
|
## Longitudinal linkage
|
|
|
|
VIN is indexed in both logical datasets and makes longitudinal analysis
|
|
possible, but must never appear in public outputs.
|
|
|
|
- 88.2% of 10,000 sampled distinct inspection VINs had at least one DMV match.
|
|
- The sampled inspection history median was eight visits per vehicle.
|
|
- 74.2% of 10,000 sampled distinct DMV VINs had at least one inspection match.
|
|
- Extreme repeat counts exist and require invalid/shared-identifier filtering.
|
|
|
|
Use a salted one-way internal token if a stable identifier is needed during
|
|
feature engineering. Never send VINs or plates to the browser or Bolt.
|
|
|
|
## Operational and sensitive relations
|
|
|
|
The `imreports` and `fdw_data` schemas include users, invitations, sessions,
|
|
event/upload logs, file contents, client network metadata, and remote ingest
|
|
state. They are operational rather than analytical and should be excluded.
|
|
Foreign tables in `fdw_countydata` mirror source data and are unnecessary when
|
|
the normalized `data` tables are available.
|
|
|
|
## Primary quality risks
|
|
|
|
- Missing years and partial periods can masquerade as real trends.
|
|
- Make/model and fuel categories need canonicalization.
|
|
- Outcome blanks must not be treated as passes.
|
|
- County labels also encode source-system changes and therefore potential drift.
|
|
- Rich JSON features are missing by design in older feeds, not missing at
|
|
random.
|
|
- Same-test OBD/result fields cause target leakage in pre-test prediction.
|
|
- Direct identifiers and small groups require aggregation and suppression.
|