Experimentation and causal inference: when the metric decides the answer
A causal estimate is only as meaningful as the metric it is computed on. For my capstone in the M.S. in Business Analytics at Mercy University, I estimated whether on-time delivery causes customers to buy again, using 101,684 JD.com orders and three estimators: g-computation, inverse probability weighting and augmented IPW. The raw data said on-time customers came back less often. Adjusting for confounders shrank that gap, and changing the definition of "on time" flipped its sign. I bring the same scrutiny to other measurement problems, including LLM judges whose confidence intervals look precise and miss the truth entirely, and policy questions where the metric a team optimizes is not the one that costs money.
What I've done
- In the raw JD.com data, customers whose orders arrived on time repurchased 0.76 points less often (3.70% versus 4.46%).
- After balancing 18 confounders chosen from a causal graph, the doubly robust AIPW estimate was -0.38 points, with zero inside its 95% interval.
- Under calendar, timestamp and strict definitions of on time the effect was -0.38, -0.06 and +0.51 points; only the strict definition gave an interval that excluded zero.
- Reported an E-value of 1.68, a low bar for a hidden confounder to explain the estimate away.
- At Geidea, merchant churn models fed a tiered retention strategy; churn on the portfolio went from 35% down to about 10%.
Public work
- Does on-time delivery make customers come back?: Three defensible definitions of on time produced effects of -0.38, -0.06 and +0.51 points on 21-day repurchase.
- Validating the LLM judge that decides what you bill for: The judge-only estimate's 95% interval contained the true resolution rate 0% of the time, and its bias by category ranged from +19.4 to -2.3 points.
- Queue economics for a fraud and compliance review team: Optimizing for dollars instead of alert counts let alert recall slip from 85.1% to 80.0% while dollar recall climbed from 76.8% to 93.0%.
Contact
Email sabasharat.ali@gmail.com, or find me on LinkedIn and GitHub. Full history on the experience page.
Open to roles in New York or remote US.