Hey everyone,
I am working on a real-world data quality problem and would appreciate feedback on my modeling approach.
Context:
I have a dataset of meters and their associated transformers (utility infrastructure). Some of these associations are incorrect, and the goal is to both detect and correct them.
Training data:
I’m using ~20,000 manually reviewed meter–transformer associations:
- Correct association → label = 1
- Incorrect association → label = 0
For incorrect cases, I also augment the data with the correct transformer, e.g.:
Meter1 | Trans1 | 0 (incorrect)
Meter1 | Trans2 | 1 (corrected)
Meter2 | Trans3 | 1 (correct)
Current baseline:
I started with a logistic regression model (class_weight="balanced" due to ~37% incorrect vs 63% correct).
Using a 0.20 threshold gives strong true negative performance (~98%), but only moderate recall.
Candidate generation:
For inference, I generate candidate transformers within a 550 ft radius for each meter (including the currently assigned…