Question 20 of 20 Advanced Scenario
How would you validate that an optimized SQL query gives exactly the same scientific cohort as the original?
Short answer
Run both queries on the same frozen snapshot and compare their results as multisets in both directions with EXCEPT ALL. Then check row counts, key uniqueness and targeted edge cases, because identical output on today's data does not prove the logic is equivalent.
Why
A cohort query defines who is in a study, so a rewrite that loses three patients changes results downstream. Comparing counts alone is not enough, because one patient dropped and another added leave the count unchanged. The comparison must be row by row and must respect duplicates.
Set operations do this directly. The PostgreSQL documentation explains that EXCEPT returns rows in the first result that are absent from the second and removes duplicates unless EXCEPT ALL is used, so EXCEPT ALL is the version that notices a patient appearing twice in one result and once in the other. Equivalence on data is evidence, not proof, so the check is paired with fixtures that exercise the boundaries the rewrite touched.
How to work it through
- Freeze a snapshot of the source tables so both queries see identical data.
- Run the original and the optimized query and save both results with the same column order and types.
- Compute original EXCEPT ALL optimized and optimized EXCEPT ALL original, and require both to be empty, because ALL keeps duplicates that plain EXCEPT would hide.
- Check row counts and that the cohort key is unique in both outputs.
- Build a small fixture with edge cases the live data may lack, such as NULL dates, patients on boundary dates, duplicates and missing visits, and compare again.
- Record the snapshot, both query versions and the comparison output so the equivalence claim can be audited.
An example
An original query defines a cohort of 12,418 patients with a qualifying diagnosis and two visits within 365 days. The optimized version returns 12,418 rows too. original EXCEPT ALL optimized returns 3 patients and optimized EXCEPT ALL original returns 3 others. Inspection shows the rewrite changed the window test from <= 365 to < 365 days, swapping patients exactly at the boundary.
After fixing it, both differences are empty on the snapshot and on a 40-row edge-case fixture.
Where the answer stops holding
Equal output on one snapshot shows equivalence for that data, not for every possible input. Edge-case fixtures reduce but cannot remove that gap. Floating-point columns may differ in the last digits between plans, so compare them with a tolerance or round explicitly.
Results that depend on the current time or on non-deterministic ordering need those inputs fixed first.
A strong answer covers
- Same frozen snapshot for both queries
- EXCEPT ALL in both directions
- Counts alone can hide swaps
- Edge-case fixture for boundaries and NULLs
- Record evidence for audit
Common mistakes
- Declaring equivalence because the two row counts match
- Using EXCEPT without ALL, which hides a patient duplicated in only one result
- Comparing results from different days so source data changes mask differences
The interviewer might ask next
How would you keep this equivalence check running as the cohort definition evolves?