Data quality and observability: catching failures that look healthy
The data failures that cost the most are the ones where every job reports success. Through my own consulting practice I was the CEO's data person at Rime, a Saudi edge-AI retail camera startup, and the biggest problem I found there was that kind of failure: most upstream data was never arriving, and nothing flagged it. I traced the cause, fixed it in stages, and then built the reconciliation and observability that would have caught it in the first place. My public projects push the same idea further. They measure which checks catch silent failures and which only catch the loud ones, such as a device going offline or a job erroring out, so that monitoring effort goes where the expensive problems are.
What I've done
- Found that 80% of upstream data from a 150-device camera fleet at Rime was being lost, and shipped the fix as a staged rollout.
- Wrote reconciliation and observability from scratch in Python and GitHub Actions; on its first run it surfaced two real sync issues.
- Brought a 137 GB PostgreSQL database back to health after its 8-hour Airbyte syncs kept failing.
- At Geidea, ran real-time anomaly detection inside Talend ETL over 1M+ payment transactions a day to flag fraud and settlement errors.
- Also at Geidea, automated a Tableau platform that five or more teams used for FinOps reconciliation and SLA tracking, which reduced manual reporting by 40%.
Public work
- Silent-failure detection for a distributed capture fleet: With 43 failures injected into a simulated fleet, row-count checks caught 18 while content, clock and coverage checks caught all 43, and bad device-days fell from 1,097 to 58.
- Ledger integrity under agent-speed traffic: Torn writes and out-of-order rows are contradictions inside data already held, so they were caught at a median of 0.00 hours with no waiting window.
- Marketplace payout reconciliation: On a book with zero injected breaks, daily totals still raised an alert on 31 of 31 days, while line-level checks raised none.
Contact
Email sabasharat.ali@gmail.com, or find me on LinkedIn and GitHub. Full history on the experience page.
Open to roles in New York or remote US.