Data scientist for LLM evaluation and AI measurement
I am Syed Ahmed Basharat Ali, a data scientist and analytics engineer in New York City. My AI work sits on the measurement side: checking whether the numbers an LLM produces, or the numbers reported about it, can be trusted.
I also build and implement AI assistants, currently as a Graduate Assistant in AI Implementation at Mercy University in New York (Dec 2025 to present).
LLM evaluation and measurement projects
In llm-judge-validation, the judge-only estimate has the tightest confidence interval of four methods and covers the truth 0% of the time. A difference estimator using 250 human labels cuts RMSE from 5.20% to 2.89%, and the reviewers’ own 4% error rate sets a floor of about 2 points.
- Validating the LLM judge that decides what you bill for: The LLM judge overstates resolution by 5.2 points overall and +19.4 in one category; the judge-only confidence interval covers the truth 0% of the time.
- A trust harness for a semantic layer: 40 of 90 queries (44%) have no single answer; one revenue question returns $149M, $136M or $107M depending on the join.
- AI spend intelligence from a raw card feed: 231 card descriptors resolved to 15 vendors; metered spend is not callable before about day 12 of the month, and is plus or minus 100% at day 5.
Building and implementing AI systems
- Mercy University: university-wide Ethical AI Faculty Toolkit and GenAI literacy curriculum for 5,000+ students; implemented AI assistants for university departments; trained faculty.
- Through my own consulting practice: a bilingual Arabic/English AI assistant on OpenAI function calling for a KSA grocery-delivery startup.
Statistics background
The measurement work draws on experimentation and causal inference. In on-time-delivery-causal, a study on 101,684 JD.com orders, the definition of "on time" flipped the estimated effect on repurchase from -0.38 to +0.51 points. At Geidea I built real-time anomaly detection on 1M+ daily payment transactions that evolved into production fraud models.
Tools
Python, OpenAI and Anthropic APIs, RAG, MCP, Claude Code, Cursor, PyTorch, scikit-learn, Causal inference, SQL.
Contact
Email sabasharat.ali@gmail.com, or find me on LinkedIn and GitHub. Full history on the experience page.
Open to roles in New York or remote US.