An LLM that proposes and a system that decides: thresholds, review and audit for production AI

A language model becomes safe to run in production when its output is treated as a proposal and something else decides what happens to that proposal. That "something else" is a set of ordinary engineering controls: a measured confidence signal, a threshold chosen from the cost of being wrong, a human review step that actually catches errors, and a record of every decision. None of it is exotic, and most of the work is outside the model.

The demo hides the question that matters

In a demo, the model reads twenty documents and gets nineteen right. Everyone is impressed. In production, the same model reads fifty thousand documents a month, and the question is no longer "how often is it right?" but "what happens on the occasions it is wrong, and who finds out?"

So start there. Before choosing a model or writing a prompt, write down for each task:

• What does a wrong output cause? A mislabeled support ticket costs a few minutes of rerouting. A wrong payee on a payment costs money and trust. A wrong statement about coverage or eligibility can have legal consequences.
• Is the action reversible? A draft can be discarded. An email that went out cannot be recalled. A posted ledger entry needs a correcting entry and an explanation.
• How quickly would anyone notice? Some errors surface in minutes because a person sees them. Others sit in a database until an audit.
• Who bears the cost? Your team, your customer, or a third party who never agreed to be part of your experiment.

This gives you a simple grid. Low cost and reversible: automate freely, sample for quality. High cost or irreversible: the model assists, a person or a deterministic rule commits. Most business tasks contain both kinds of case, which is exactly why you need a threshold instead of a single yes or no.

Note also that the two kinds of error are rarely equal. Wrongly approving a fraudulent refund and wrongly sending a legitimate refund to review have very different prices. Your controls should reflect that asymmetry.

A confidence number you can trust

A threshold needs a score, and the obvious source of a score is the wrong one. Asking the model "how confident are you, from 0 to 100?" produces a number, and that number is generated the same way as the rest of the text. It tends to be high, it clusters around a few values, and it is not reliably connected to whether the answer is correct. Fluent, assured wording is not evidence either. Models produce wrong answers in the same tone as right ones.

Useful confidence signals come from outside the model's opinion of itself:

1. Deterministic validation. Does the output parse against the schema? Does the extracted invoice total equal the sum of the line items? Does the customer ID exist? Would the extracted date put the invoice ten years in the future? Is the chosen category one of the allowed values? These checks are cheap, exact and catch a surprising share of errors.
2. Agreement. Run the task twice with different prompts, or with two different models, and compare. Agreement is not proof, but disagreement is a strong reason to route to a person.
3. Token probabilities, where available. For classification into a fixed set of labels, the probability the model assigned to its chosen label can be a usable signal, if your provider exposes it. It still needs calibration.
4. Grounding checks. If the answer is supposed to come from supplied documents, verify that each claim can be matched to a passage. An answer with no supporting passage is low confidence by definition.
5. Input quality signals. A blurry scan, an unusually long document, a language you did not test, a field that is usually present and is missing. Many errors are predictable from the input alone.
6. A separate checker. A second model or a small trained classifier whose only job is to estimate whether the first output is acceptable.

Whatever you combine into a score, calibrate it. Calibration means checking the score against reality: take a labeled set of real cases, group them by score, and measure the actual accuracy in each group. If the cases scoring above 0.9 are right 97 percent of the time, you know what 0.9 means. If they are right 80 percent of the time, the score is decoration. Without this step you do not have a confidence threshold, you have a number that feels like one.

Who gets to pick the cutoff

Once the score means something, choosing a cutoff means balancing three quantities against each other: how many cases are handled automatically, how many errors get through, and how much review work you create. Engineering can draw that curve. Engineering should not pick the point on it alone, because the price of an error belongs to the business.

A method that works:

1. Collect a few hundred to a few thousand real, labeled examples, including the awkward ones.
2. For each candidate threshold, compute the share of cases that would be automated and the error rate among them.
3. Put a cost on each outcome: a correct automation saves X, a review costs Y, an error that reaches a customer costs Z.
4. Choose the threshold with the owner of the process, who is the person accountable for Z.

A worked example, with invented numbers for illustration only. Picture an accounts payable team that receives 10,000 invoices a month and wants extracted fields posted automatically. At one threshold, 85 percent are automated and 1 in 100 of those contains an error. At a stricter threshold, 60 percent are automated and 1 in 1,000 contains an error. The first option sends 1,500 invoices to review and lets about 85 errors through. The second sends 4,000 to review and lets about 6 through. Which is better depends entirely on what a mis-posted invoice costs compared with a few minutes of a clerk's time. That is not a modeling question.

Three refinements are worth building from the beginning:

• Use bands, not one line. Above the high threshold: act automatically. Between the two: human review with the model's suggestion shown. Below the low threshold: route to a person without the suggestion, or reject, because a bad suggestion is worse than none.
• Set thresholds per field and per case type. A vendor name and a bank account number do not deserve the same tolerance. Neither do a $40 invoice and a $400,000 one.
• Re-measure on a schedule. Inputs drift, vendors change formats, and model providers update models. A threshold that was right in March may be wrong in September. Keep a labeled sample flowing and recheck.

Review that catches errors instead of approving them

"A human reviews it" is the most over-trusted sentence in AI projects. People who are shown a plausible suggestion and a queue of two hundred items tend to accept the suggestion. The effect is well known as automation bias, and a review step that approves 99.8 percent of what it sees in four seconds per item is providing a signature, not a check.

Design the review so that catching an error is the easy path:

• Show the evidence beside the proposal. The source document with the relevant passage highlighted, next to the extracted value. A reviewer should verify, not re-derive.
• Show why it was flagged. "Total does not match line items" directs attention. "Low confidence" does not.
• Ask for the high-risk value independently. For the fields that matter most, have the reviewer enter or select the value before seeing the model's answer, then compare.
• Seed known errors. Insert a small number of items with deliberate mistakes and measure how many reviewers catch. This is the only direct measure of whether your review works.
• Keep the queue honest. If volume exceeds capacity, people start rubber-stamping. Either raise capacity, tighten what gets automated elsewhere, or accept a slower turnaround. Do not quietly accept faster clicking.
• Give authority and time. A reviewer who is measured only on throughput will optimize throughput. Rejection must be a normal, unpunished outcome.
• Sample the automated lane. High-confidence cases skip review, so audit a random sample of them regularly. This is how you learn that the score has drifted.
• Feed corrections back. Every override is a labeled example. Store it and use it for evaluation, prompt changes and recalibration.

Remember what review cannot do: it only sees what was routed to it. Errors with high scores pass straight through, which is why calibration and sampling matter as much as the reviewers.

Log it as if someone will ask

Someone will. A customer disputes a decision, an auditor wants to know how a figure was produced, or an engineer needs to explain why Tuesday's results were different. If the answer is "the model did it", you have a problem.

For every model-assisted decision, record:

• The input, or a reference to it and a hash if the content is sensitive.
• The exact model identifier and version, and the parameters used.
• The prompt template version and any retrieved context that was supplied.
• The raw output and the parsed output.
• Each validation result and the final confidence score.
• The threshold configuration in force at that moment.
• The route taken: automatic, reviewed or rejected.
• The reviewer's identity, decision, any edits and the timestamp.
• The action finally taken in the downstream system, with its identifier.
• A correlation ID that ties all of the above together.

Two practical notes. First, version prompts and thresholds like code: changes go through review, and the log says which version produced which decision. Second, logs of prompts and outputs frequently contain personal or confidential data. Apply access control, retention limits and redaction to them as you would to the source data. An audit trail that leaks is its own incident.

With this in place you can answer the three questions that always come up: what did the system see, what did it propose, and who or what approved it.

What stays with people and with code

Some decisions should stay out of the model's hands regardless of the score, because the downside is not a matter of accuracy.

• Irreversible actions. Moving money, deleting data, sending external communications that create commitments, changing production configuration.
• Decisions with legal or significant personal effect. Hiring, termination, credit, insurance coverage, access to services, anything touching clinical judgment. Several jurisdictions regulate automated decision-making in these areas, so get proper legal guidance for your situation. This article is general information, not legal advice.
• Decisions that need an accountable name. If someone will have to stand behind it in front of a customer, a regulator or a court, a person makes it.
• Policy exceptions. The model can identify that a case is unusual. Granting the exception is a judgment about precedent.
• Arithmetic and business rules that code can do exactly. Do not ask a model to calculate tax, apply a discount table or enforce a limit. Compute those deterministically and let the model handle the language around them.

The design principle underneath all of this is least privilege. The model's credentials should not include the ability to perform the dangerous action at all. It writes to a "proposed" table, a draft folder or a queue. A separate component with its own rules, and where needed a person, performs the commit. Then no prompt, however cleverly crafted by an outsider, can talk the system into doing what it has no permission to do.

A model that reads, extracts, classifies, summarizes and drafts, inside a system that validates, routes, reviews and records, is useful on day one and stays defensible in year three.

If your model made a costly mistake tomorrow, could you show how it was approved?

If the honest answer is "not really", the gap is in the system around the model, and it can be closed. Trivo can work through your use case with your team, set the thresholds against real cost, and leave you with a written design for review and audit that you own.