How to set up sample checks of AI output in daily operations

The sample follows the impact of a mistake, not a random spread

Once AI moves from a pilot into daily operations, a one-off check before launch stops being enough. Output volume grows, the tasks being handled change, and nobody has the capacity to check everything. The real question is not whether to sample, but how to choose the sample so review lands where a mistake actually hurts, instead of being spread evenly across tasks with very different consequences.

This article covers setting up that ongoing routine: what gets sampled, at what rate, who carries out the check, what gets logged, and what a specific finding should trigger. It is written for a process where AI is already running in production, not for a one-off check carried out before rollout.

What to sample: by task risk, not a fixed number

A random sample across the whole operation has one major weakness: it is just as likely to pick a low risk task as a task with a serious consequence, so reviewer capacity ends up spent on tasks where a mistake would barely matter anyway. The fix is to sort tasks into risk levels based on what an undetected error would cause, and set the review rate accordingly.

High risk covers tasks with a direct effect on a customer, a legal or financial consequence, or output that moves forward without any further human check. Medium risk covers tasks whose output feeds further internal work but still passes through another approval or edit. Low risk covers purely supporting tasks, where an error is easy to spot and cheap to fix before it causes any harm.

This split does not need to be complicated or formal. It just needs the process owner to think it through once and write it down, because without it, review tends to drift toward whatever is easiest to reach rather than what actually carries the most risk.

Who checks it, what gets logged, what a finding triggers

A reviewer needs enough expertise to judge whether that type of output is correct, and should ideally be someone other than the person who produced it, so the check is not just the same person looking back at their own work. For high risk tasks, it makes sense to add a second independent reviewer or a formal sign-off step beyond routine practice.

Every sample that gets checked should leave a record, not just pass or fail silently. A record makes a pattern visible over time, rather than leaving each case isolated, and it is what a decision to raise or lower the checked share is actually based on.

A finding always closes with one of four outcomes: no action, if the output meets the standard; retraining, if the error came from a gap in understanding the tool or the brief, a pattern also covered from the training side in the mistakes that ruin company AI training; a process change, if the problem sits not in the people but in the brief, the template, or how the task is set up; or a temporary suspension of AI for that task type, if the severity or frequency of errors is not tolerable. Choosing between these four belongs to the process owner, not the reviewer, because the choice carries a wider operational effect.

How the checked share moves with what the check finds

A fixed checked share that never changes ignores the single most useful thing review actually provides: information about how the task or tool behaves over time. If review keeps finding no problem in a given task type, it is reasonable to consider lowering the checked share and moving the freed capacity elsewhere. If a repeating pattern of the same error shows up instead, it makes sense to temporarily raise the checked share for that task type, until the cause is addressed and the raised rate confirms the fix more than once.

This is not a formula or a statistically derived rule; it is operational judgement that review capacity should track wherever a problem is currently showing up, rather than staying fixed at a number set once at the start. The same logic, letting severity and pattern decide where attention goes rather than treating every item the same, shows up in how to prioritise security findings when you have hundreds of them to work through.

Artefact: sample check plan and finding record sheet

A. Sample check plan by risk level

Risk level	Example task	Starting checked share	Who checks it	What the check verifies	Trigger to raise the share
High risk	output goes straight to a customer, or carries a legal or financial consequence	company sets its own high share, potentially close to full review	independent reviewer, plus a second person for the most sensitive tasks	factual accuracy, rule compliance, absence of a serious error	any finding of a serious error
Medium risk	output feeds further internal work and still passes through approval	company sets its own medium share based on reviewer capacity	someone other than the author, with process knowledge	factual accuracy and whether the output meets the standard for the next step	a repeated finding of the same error type
Low risk	supporting task, where an error is cheap and quick to fix	company sets its own low share, enough to keep visibility	the person assigned to routine quality checks	basic usability of the output	a rise in the number of findings against the usual pattern

Each company needs to set its own actual checked share figures based on operating volume and reviewer capacity; the table deliberately leaves out a universal percentage, because no such figure has support in the cited sources.

B. Individual finding record sheet

Field	Entry
Sample ID	your own fixed identifier
Date checked
Task type and risk level
Reviewer
Finding	yes / no
Error type, if any
Severity	low / medium / high
Action taken	no action / retraining / process change / suspend use
Date to verify the fix

The record sheet works fine as a simple shared table. What matters is that the reviewer fills it in right after the check, not from memory later, and that it stays accessible to the process owner for a regular review of the pattern over time.

Where this fits in wider operations

Sample checking builds on how a company first assessed the quality of AI-assisted work when it rolled the tool out. Where that earlier step compares before and after, sample checking tracks steady-state operations continuously. A formal, broader review of an entire AI deployment, not just an ongoing sample, is covered separately in what an AI system audit covers and how it works, and how findings from a check like this should feed into wider operational security is covered in how to secure AI use across the enterprise.

Sources and limits

NIST’s TEVV framework describes general principles for testing, evaluating, validating and verifying AI systems, but it is not a sector standard and not a methodology written specifically for sample checking in a small or medium company’s daily operations. The Measure function in the NIST AI RMF Core supports the principle that AI output quality should be tracked continuously rather than checked only once at rollout, without pointing to any specific checked share. NIST AI 600-1 describes the risk profile of generative AI and is cited here only as support for sorting tasks by the severity of potential impact, not as a source of specific figures. The checked shares, escalation triggers and the structure of both tables above are a CIAD recommendation built on operational logic, not a statistically derived formula or a measured error rate; none of the cited sources states such a figure, and this article does not invent one.