Success Metrics and KPIs

Two numbers the claims director does not care about

A candidate finishes a design and closes with: "Success would be 95% accuracy on the summaries and under two seconds of latency." The interviewer, playing a claims director, asks two short questions. "Accuracy of what? And why would I care?"

It is a fair challenge. Both numbers describe the software, and neither says whether the director's business got better. A metric plan for an FDE engagement needs one number the sponsor already cares about, measured today, plus a few numbers that catch harm. Model metrics still have a job, just a different one from the one the candidate gave them.

Summaries for a property insurer, and what they should move

Take a fictional property insurer that wants an LLM to draft a summary of each new claim at first notice of loss (FNOL, the first report of a claim). The summary pulls from the caller's description, photos and the policy, so adjusters don't have to read everything from scratch. Every number below is made up for the example.

Why accuracy alone cannot tell good from useless

Suppose the plan measures only accuracy and latency, and the summaries reach 95% accuracy on a review set. If adjusters still read the whole file because they don't trust the summary, nothing changed for the director. Now flip it. If the summaries are slightly worse but adjusters call claimants sooner, the business improved. The model metric cannot tell these two worlds apart, and that difference is what the director is paying for.

Output, outcome and guardrail

An output metric counts what the system produced. An outcome metric measures what changed for the business because of it. Guardrails catch harm the primary metric would hide. Here is the insurer's list, sorted:

Pick one primary. Hours from FNOL to first contact is the best candidate, because the director cares about it and the claims system already timestamps both events. Accuracy stays in the plan, demoted to a release gate: summaries don't ship to the pilot team below an agreed score on the review set. The guardrails are reopened-claim rate and complaint rate for the same team over the same period. They don't need to improve; they must not get worse.

The minutes nobody records

Adjuster minutes per claim matters too, but no system records it. Rather than wish the data existed, run a small time study: an observer times 30 intake reviews before the pilot and 30 during it. Say intake review drops from 22 to 14 minutes and the pilot team handles 400 claims a week. That is 8 × 400 = 3,200 minutes, about 53 hours a week. Report it as a secondary estimate with its sample size.

Pulling the baseline before launch

A metric with no baseline cannot show improvement, so the baseline gets pulled from the claims system before launch, for the pilot team and the claim types in scope. In Postgres it looks like this:

SELECT date_trunc('week', fnol_at) AS week,
       count(*) AS claims,
       count(first_contact_at) AS contacted,
       percentile_cont(0.5) WITHIN GROUP (
         ORDER BY extract(epoch FROM first_contact_at - fnol_at) / 3600
       ) AS median_hours_to_contact
FROM claims
WHERE team_id = 7
  AND claim_type IN ('water', 'wind')
  AND fnol_at >= now() - interval '12 weeks'
GROUP BY 1
ORDER BY 1;

There is a trap hiding in that median. Rows with no contact yet are ignored by percentile_cont, which flatters recent weeks, since the slowest claims are exactly the ones not yet contacted. That is why the query reports claims and contacted side by side: when the two diverge for a recent week, that week's median is too optimistic to use. Say the stable weeks show a median of 26 hours. That is the number the pilot is measured against.

Choosing metrics for any engagement

Start from the sponsor's sentence about why they're paying. "Claimants wait too long" becomes hours to first contact, and "adjusters are overloaded" becomes minutes per claim. If you can't connect a metric to a sentence the sponsor said, it is probably an output metric.

Name one primary outcome. Two primaries invite a split verdict, where one goes up and the other down and nobody can say whether the pilot worked. Then add two or three guardrails, each tied to a specific harm: wrong information reaching a customer, rework, complaints, or a fairness or compliance concern the customer has named.

For each metric, name the system and field it comes from. If none exists, say how you'd measure it anyway (a time study, a manual sample, a short survey) and label the result as an estimate. Measure the baseline before launch, from the same population the pilot will touch, over enough weeks to see normal variation, and compare like with like afterwards: same team, same claim types, similar season.

Where does that leave accuracy, latency and cost? They become gates. They decide whether a version may ship, and the outcome metric decides whether shipping was worth it.

Four ways a metric plan fails the sponsor

The first is accuracy or latency as the only metric. It answers "does the model work?" and leaves "did the business change?" unasked. Interviewers reach for the director's question, "why would I care?", and a candidate with only model metrics has nothing to say.

The second is no baseline. "After launch, median time to contact was 19 hours" means nothing without the before. A baseline from memory is worse ("adjusters say it used to take a day or two"), because it can't be checked and tends to drift toward whatever makes the project look good.

The last two are subtler. A metric with no data source, like "customer trust" or "adjuster productivity", is a direction until someone names the field or the method. And counting outputs as wins misleads: "12,000 summaries generated" is a usage number, and if the reply is "and did anything change?", it was the wrong number to lead with.

Drill: tier-1 support tickets

Here is a fictional brief to practise on. A B2B software company's support lead says, "We want an AI agent to answer tier-1 tickets. We get about 3,000 a week." The ticketing system records created, first-response and resolved timestamps, a reopened flag, and an optional satisfaction survey. In 10 minutes, write the primary outcome metric, two guardrails, and the baseline query plan: which fields, which tickets, how many weeks. Then write one sentence the support lead could repeat to their VP.

A strong answer has a primary such as the share of tier-1 tickets resolved without a human and not reopened within 7 days, guardrails like reopen rate and satisfaction score on agent-handled tickets, and a baseline from the last 8 to 12 weeks of the same ticket categories. "Tickets deflected" alone is an output metric, since a deflected ticket that comes back reopened is a cost. Rehearse it aloud with the FDE AI coach in thita, or run it as an FDE mock round.

How this shows up in the interview

Metrics are a closing line in decomposition rounds. For Palantir, Aced lists iteration and success metrics in the reported rubric; for Databricks, Aced reports prompts that ask for a KPI outright. In AI-agent roles the metric is sometimes the whole question. A candidate report on Aced describes a Sierra final round asking for north-star success metrics for an AI agent on a healthcare financial platform, and a take-home asking how to diagnose a drop in one agent metric. OpenAI's system design round reportedly includes "what evaluation would catch failures before customers see them", per Aced.

The usual probes are "accuracy of what?", "what is it today?", and "where would that number come from?" Credit goes to answering all three at once, with model quality placed as a gate. A sentence that does it: "Primary metric is median hours from FNOL to first contact for team 7, which is 26 today from the claims timestamps; accuracy is a release gate, and reopen rate is the guardrail."

Check yourself

• Why is summary accuracy a gate in the insurer example rather than the primary metric? Answer: It describes the model, and it can rise while nothing changes for the director. Hours to first contact measures the business result and comes from data the insurer already has.
• What does the SQL baseline get wrong if you drop the count(first_contact_at) column, and why does it matter? Answer: You lose sight of uncontacted claims, which the median ignores. Recent weeks then look faster than they are, and the baseline flatters itself.
• How do you handle a metric no system records? Answer: Measure it with a small method you can state (a timed sample of 30 reviews before and during) and report it as a secondary estimate with its sample size.