103 lines
5.2 KiB
Markdown
103 lines
5.2 KiB
Markdown
# Countydata inventory
|
|
|
|
Inventory date: 2026-07-15. All inspection was performed with read-only
|
|
transactions and aggregate queries; no identifiers were exported.
|
|
|
|
## Accessible databases
|
|
|
|
- `countydata`: the useful analytical database, approximately 93.4 GB.
|
|
- `postgres`: empty local/staging copies of the core vehicle table structure
|
|
plus foreign-table links. It is not the analytical source.
|
|
- `vattp`: a tiny, unrelated course/event registration and survey application.
|
|
It is not suitable for this project and its application records should remain
|
|
out of scope.
|
|
|
|
## DMV logical dataset
|
|
|
|
The following are one logical dataset split into projections joined one-to-one
|
|
by `id`; they are not four independent populations.
|
|
|
|
| Relation | Approximate role | Key analytical fields |
|
|
| --- | --- | --- |
|
|
| `data.dmv_tax_json` | Raw JSON and ingest metadata | Full DMV record, file/source metadata |
|
|
| `data.dmv_tax` | Core lookup | VIN, registration date, county |
|
|
| `data.dmv_tax_search` | Search projection | VIN, registration date, county |
|
|
| `data.dmv_tax_vehicle` | Vehicle projection | Make, model, model year, fuel, registration type/place, temporary flag, expiration/emission dates, ZIPs |
|
|
|
|
Exact current logical row count: **18,009,278**.
|
|
|
|
- Date bounds are 2011-09-07 through 2024-03-13, but the few pre-2016 rows are
|
|
outliers and there are only 117 records in 2021.
|
|
- The 29 Utah counties are represented. Salt Lake (35.66%), Utah (17.21%),
|
|
Davis (10.29%), and Weber (8.01%) account for most records.
|
|
- Canonicalized fuel mix by record is 86.23% gasoline, 7.97% diesel, 2.59%
|
|
flexible fuel, 2.06% hybrid, 0.70% electric, and 0.24% plug-in hybrid.
|
|
- A vehicle appears repeatedly over time: sampled DMV histories had a median of
|
|
five registration records.
|
|
|
|
## Inspection logical dataset
|
|
|
|
These are likewise projections of one logical inspection dataset joined by
|
|
`id`.
|
|
|
|
| Relation | Approximate role | Key analytical fields |
|
|
| --- | --- | --- |
|
|
| `data.inspection_json` | Raw JSON and ingest metadata | Full inspection, detailed OBD/readiness/visual fields where supplied |
|
|
| `data.inspection` | Core lookup | VIN, test timestamp, ingest timestamp |
|
|
| `data.inspection_search` | Outcome/search projection | County/source, overall and OBD results, test/program type |
|
|
| `data.inspection_vehicle` | Vehicle projection | Make, model, year, calibration/certificate, station |
|
|
| `data.inspection_obd` | OBD summary | Result-reason code, DTC count, permanent-DTC flag |
|
|
| `data.inspection_plate` | Identifier lookup | VIN, plate, test timestamp; sensitive and unnecessary for analytics |
|
|
|
|
Exact current logical row count: **19,357,287**.
|
|
|
|
- Four dates in 1990 are outliers. Normal coverage begins in 2010 and continues
|
|
through 2026-06-22; 2026 is partial.
|
|
- Real source/county labels are `slc`, `slco`, `utah`, `weber`, `davis`, and
|
|
`cache`. The two Salt Lake labels represent different source eras and should
|
|
not be blindly treated as different counties.
|
|
- Overall results are 71.06% pass, 3.61% fail, 3.56% reject, 2.31% abort, and
|
|
19.45% blank/null or other near-blank values in the raw `overall_result`
|
|
field. Most missing overall results belong to the older Utah County feed; its
|
|
audited OBD/OBD rows carry a separate result that is usable only as a
|
|
provenance-tagged binary pass-versus-non-pass proxy.
|
|
- About 89.5% of records use the OBD program and about 9.2% use TSI.
|
|
- DTC count is zero in about 90.4% of records and null in about 1.4%; the
|
|
remaining values are class-imbalanced and include rare data-quality outliers.
|
|
- Raw JSON can contain odometer, vehicle fuel/type/GVWR/cylinders/engine,
|
|
transmission, DTCs, PIDs, MIL/readiness status, communication protocol, and
|
|
visual inspection fields. These richer fields are concentrated in the newer
|
|
`slco`, `davis`, and `cache` feeds rather than statewide history.
|
|
|
|
## Longitudinal linkage
|
|
|
|
VIN is indexed in both logical datasets and makes longitudinal analysis
|
|
possible, but must never appear in public outputs.
|
|
|
|
- 88.2% of 10,000 sampled distinct inspection VINs had at least one DMV match.
|
|
- The sampled inspection history median was eight visits per vehicle.
|
|
- 74.2% of 10,000 sampled distinct DMV VINs had at least one inspection match.
|
|
- Extreme repeat counts exist and require invalid/shared-identifier filtering.
|
|
|
|
Use a salted one-way internal token if a stable identifier is needed during
|
|
feature engineering. Never send VINs or plates to the browser or Bolt.
|
|
|
|
## Operational and sensitive relations
|
|
|
|
The `imreports` and `fdw_data` schemas include users, invitations, sessions,
|
|
event/upload logs, file contents, client network metadata, and remote ingest
|
|
state. They are operational rather than analytical and should be excluded.
|
|
Foreign tables in `fdw_countydata` mirror source data and are unnecessary when
|
|
the normalized `data` tables are available.
|
|
|
|
## Primary quality risks
|
|
|
|
- Missing years and partial periods can masquerade as real trends.
|
|
- Make/model and fuel categories need canonicalization.
|
|
- Outcome blanks must not be treated as passes.
|
|
- County labels also encode source-system changes and therefore potential drift.
|
|
- Rich JSON features are missing by design in older feeds, not missing at
|
|
random.
|
|
- Same-test OBD/result fields cause target leakage in pre-test prediction.
|
|
- Direct identifiers and small groups require aggregation and suppression.
|