Validating the LLM judge that decides what you bill for

The problem

Under outcome-based pricing, the resolution label on a support conversation is the invoice, and an LLM judge usually assigns it.

This project simulates 8,000 conversations across 6 intents with a biased judge and a small human review budget, and compares four estimators over 400 replications using 250 human labels.

Headline numbers

• The judge reports 81.1% resolved against a true 75.9%: an overstatement of 5.2 points.
• The judge-only estimate has the tightest interval of the four (plus or minus 0.86%) and covers the truth 0% of the time.
• For billing disputes the judge overstates by 19.4 points; for shipping status it understates by 2.3.
• A difference estimator using human labels cuts RMSE from 5.20% to 2.89%, but a 4% reviewer error rate leaves a floor of about 2 points.

What it means

A tight interval computed on the wrong quantity is a confident wrong answer. Correcting a judge needs human labels, stratified by segment when per-customer billing depends on it, and past roughly 1,000 labels the budget is better spent on label quality than on more labels.