ai//3 min read
Shipping AI features without breaking trust: evals, guardrails and review
A language model will be confidently wrong sometimes. The products that work design for that: narrow jobs, structured output, automatic checks, human review where it matters, and a test set that catches regressions.
In short
- Give the model one narrow job with a clear definition of a good answer.
- Ask for structured output and validate it with a schema before your code trusts it.
- Build a small evaluation set from real examples and run it on every prompt or model change.
- Put a person in the loop wherever a wrong answer is expensive, and log what the model suggested.
Adding a model call to a product takes an afternoon. Making that feature reliable enough that customers trust it takes design. Models are probabilistic: the same input can produce different outputs, and some outputs will be wrong in ways that look right. The goal is not to eliminate mistakes but to contain them.
1. Narrow the job
“Help users with their documents” is not a feature. “Extract the invoice number, date, supplier and total from an uploaded invoice” is. Narrow jobs have checkable answers, predictable costs and obvious failure modes. Broad ones have none of these.
2. Ask for structure, then validate it
Most model APIs can return JSON that follows a schema. Use it, and still validate the result in your own code before anything downstream relies on it.
import { z } from "zod";
const Invoice = z.object({
invoiceNumber: z.string().min(1),
issueDate: z.iso.date(),
supplier: z.string().min(1),
total: z.number().nonnegative(),
currency: z.string().length(3),
});
const parsed = Invoice.safeParse(JSON.parse(modelOutput));
if (!parsed.success) {
// Retry once with the validation errors, then fall back to manual entry.
return { status: "needs_review", issues: parsed.error.issues };
}Add business checks on top of the schema: does the total equal the sum of the line items? Is the date in a plausible range? These catch a surprising share of errors cheaply.
3. Build an evaluation set
Collect 50 to 200 real examples with the answers you expect. Include the awkward ones: blurry scans, unusual layouts, other languages. Every time you change the prompt, the model or the pre-processing, run the whole set and compare.
Metric | What it tells you |
|---|---|
Exact-field accuracy | How often each field is fully right |
Validation failure rate | How often output breaks the schema |
Needs-review rate | How much work lands on people |
Cost and latency per item | Whether the feature pays for itself |
Without an evaluation set, every prompt tweak is a guess. With one, you can upgrade models with confidence and catch regressions before users do.
4. Keep a person in the loop where it matters
Decide, per feature, what a wrong answer costs. A suggested email subject line can go straight to the user. An extracted payment amount should be confirmed before money moves. Design the interface so review is fast: highlight what the model changed, show the source next to the result, and make accepting or correcting a single action.
5. Budget for cost and latency
- Set a per-request token limit and a monthly spending cap per customer.
- Stream responses when people are waiting on screen; process in the background when they are not.
- Cache results for identical inputs, and use smaller models for easy cases.
- Time out and fall back gracefully when the provider is slow.
6. Log carefully
Record the input, the output, the model version and what the user did with the suggestion. That record is your best source of new evaluation examples. Strip or mask personal data before logging, and check your provider's terms on how API data is retained and used.
Trust is built by what happens when the model is wrong, not by how often it is right.
Release checklist
- To do: One narrow job with a written definition of a good answer
- To do: Structured output validated by a schema and business rules
- To do: An evaluation set run on every prompt or model change
- To do: Human review for high-cost decisions, designed to be fast
- To do: Cost limits, timeouts and a fallback path
- To do: Logging of inputs, outputs and user decisions, with personal data masked
