Gate the expensive work
Partition-level checks run first. Row hashing runs only when a partition looks suspicious, avoiding unnecessary detailed comparisons.
Data engineering · Reliability · Case study
Catch the drift before it spreads.
DataGuard compares a simulated core-banking database with a downstream reporting replica. It checks partitions first, then hashes rows where the cheap checks suggest a problem.
Source → validation → evidence
100%
Precision and recall on labeled, injected faults
6.4×
Faster at 1M rows than the naive SQL EXCEPT baseline
13/13
Automated pytest tests passing
7/8
Documented fault types classified
The problem
Two databases can disagree quietly.
A source system and its reporting replica can diverge without a single dramatic failure. Missing rows, duplicate records, null changes, timestamp drift, and changed values can all affect downstream reporting. DataGuard was built to find those differences, classify them, and leave evidence someone can investigate. The core-banking scenario and faults used here are simulated.
Architecture
Check cheaply. Investigate precisely.
Both datasets enter the same comparison path. Detailed row hashing is reserved for partitions flagged by cheaper checks.
Source
Simulated core-banking data
Replica
Downstream reporting data
Partition checks
Identify suspected drift with cheaper comparisons
Targeted row hashing
Inspect flagged partitions in detail
Classify and present
Fault evidence feeds DataGuard Insights
Core: Python · PostgreSQL · xxhash · pandas/polars
Insights: Neon Postgres · Streamlit · Plotly
Key decisions
Why it works this way.
Classify the difference
A mismatch should be actionable. DataGuard distinguishes fault types so someone investigating a problem knows what kind of drift occurred.
Separate engine and insights
DataGuard Core handles reconciliation. DataGuard Insights provides the BI layer for exploring trends and risks through a separate deployed dashboard.
Measured results
Tested, not just described.
These results come from the project’s labeled synthetic fault injections and benchmark runs.
Precision and recall
Measured against labeled, injected faults in the simulated source and replica datasets.
Faster than the baseline
Compared with a naive SQL EXCEPT approach at 100K and 1M rows respectively.
Automated tests passing
The project’s pytest suite passed all 13 documented tests.
Fault types classified
Seven documented fault types are classified. The eighth remains a known limitation described below.
Fault injection used for evaluation: 5 duplicate, 20 missing, 15 null, 20 timestamp, and 20 value-mismatch faults per run. The 100% figure refers to this labeled test setup; it is not a claim about production banking data.
What broke
Three bugs that changed the design.
The useful lessons came from testing the cases where the first implementation failed.
Hashing ran too early
Row hashing was not gated by the cheaper partition checks. The first version was slower than the naive baseline. Fix: run detailed hashing only for suspicious partitions.
Null drift was missed
The partition suspicion check ignored null-count drift. Recall fell to 77% for one fault type. Fix: include null-count differences when deciding which partitions need deeper inspection.
Duplicates looked missing
Duplicate rows were incorrectly classified as missing rows. Fix: correct the classification logic so the reported fault matches the evidence.
Limitations
What remains unfinished.
PARTITION_ANOMALY classification
The documented PARTITION_ANOMALY fault type
is not implemented. DataGuard classifies seven of the
eight documented fault types. The banking scenario is
simulated, so the benchmark and injected-fault results
should be understood in that test environment.
Explore the work
Inspect the code and dashboard.
The reconciliation engine and BI dashboard live in separate repositories. The dashboard uses Neon Postgres and Streamlit Cloud to present the findings.