Centralized_I-M_Database/SummerProject2026
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4342cdbc6a
SummerProject2026/docs/data_inventory.md
kevinBell 05729fc6de initial code
2026-07-15 17:55:53 -06:00

5.2 KiB
Raw Blame History

Countydata inventory

Inventory date: 2026-07-15. All inspection was performed with read-only transactions and aggregate queries; no identifiers were exported.

Accessible databases

  • countydata: the useful analytical database, approximately 93.4 GB.
  • postgres: empty local/staging copies of the core vehicle table structure plus foreign-table links. It is not the analytical source.
  • vattp: a tiny, unrelated course/event registration and survey application. It is not suitable for this project and its application records should remain out of scope.

DMV logical dataset

The following are one logical dataset split into projections joined one-to-one by id; they are not four independent populations.

Relation Approximate role Key analytical fields
data.dmv_tax_json Raw JSON and ingest metadata Full DMV record, file/source metadata
data.dmv_tax Core lookup VIN, registration date, county
data.dmv_tax_search Search projection VIN, registration date, county
data.dmv_tax_vehicle Vehicle projection Make, model, model year, fuel, registration type/place, temporary flag, expiration/emission dates, ZIPs

Exact current logical row count: 18,009,278.

  • Date bounds are 2011-09-07 through 2024-03-13, but the few pre-2016 rows are outliers and there are only 117 records in 2021.
  • The 29 Utah counties are represented. Salt Lake (35.66%), Utah (17.21%), Davis (10.29%), and Weber (8.01%) account for most records.
  • Canonicalized fuel mix by record is 86.23% gasoline, 7.97% diesel, 2.59% flexible fuel, 2.06% hybrid, 0.70% electric, and 0.24% plug-in hybrid.
  • A vehicle appears repeatedly over time: sampled DMV histories had a median of five registration records.

Inspection logical dataset

These are likewise projections of one logical inspection dataset joined by id.

Relation Approximate role Key analytical fields
data.inspection_json Raw JSON and ingest metadata Full inspection, detailed OBD/readiness/visual fields where supplied
data.inspection Core lookup VIN, test timestamp, ingest timestamp
data.inspection_search Outcome/search projection County/source, overall and OBD results, test/program type
data.inspection_vehicle Vehicle projection Make, model, year, calibration/certificate, station
data.inspection_obd OBD summary Result-reason code, DTC count, permanent-DTC flag
data.inspection_plate Identifier lookup VIN, plate, test timestamp; sensitive and unnecessary for analytics

Exact current logical row count: 19,357,287.

  • Four dates in 1990 are outliers. Normal coverage begins in 2010 and continues through 2026-06-22; 2026 is partial.
  • Real source/county labels are slc, slco, utah, weber, davis, and cache. The two Salt Lake labels represent different source eras and should not be blindly treated as different counties.
  • Overall results are 71.06% pass, 3.61% fail, 3.56% reject, 2.31% abort, and 19.45% blank/null or other near-blank values in the raw overall_result field. Most missing overall results belong to the older Utah County feed; its audited OBD/OBD rows carry a separate result that is usable only as a provenance-tagged binary pass-versus-non-pass proxy.
  • About 89.5% of records use the OBD program and about 9.2% use TSI.
  • DTC count is zero in about 90.4% of records and null in about 1.4%; the remaining values are class-imbalanced and include rare data-quality outliers.
  • Raw JSON can contain odometer, vehicle fuel/type/GVWR/cylinders/engine, transmission, DTCs, PIDs, MIL/readiness status, communication protocol, and visual inspection fields. These richer fields are concentrated in the newer slco, davis, and cache feeds rather than statewide history.

Longitudinal linkage

VIN is indexed in both logical datasets and makes longitudinal analysis possible, but must never appear in public outputs.

  • 88.2% of 10,000 sampled distinct inspection VINs had at least one DMV match.
  • The sampled inspection history median was eight visits per vehicle.
  • 74.2% of 10,000 sampled distinct DMV VINs had at least one inspection match.
  • Extreme repeat counts exist and require invalid/shared-identifier filtering.

Use a salted one-way internal token if a stable identifier is needed during feature engineering. Never send VINs or plates to the browser or Bolt.

Operational and sensitive relations

The imreports and fdw_data schemas include users, invitations, sessions, event/upload logs, file contents, client network metadata, and remote ingest state. They are operational rather than analytical and should be excluded. Foreign tables in fdw_countydata mirror source data and are unnecessary when the normalized data tables are available.

Primary quality risks

  • Missing years and partial periods can masquerade as real trends.
  • Make/model and fuel categories need canonicalization.
  • Outcome blanks must not be treated as passes.
  • County labels also encode source-system changes and therefore potential drift.
  • Rich JSON features are missing by design in older feeds, not missing at random.
  • Same-test OBD/result fields cause target leakage in pre-test prediction.
  • Direct identifiers and small groups require aggregation and suppression.