Human in the loop is the most-claimed control in AI systems and one of the least designed. In most implementations it means a person sees a screen and clicks approve, on every case, with no information beyond the output itself, at a volume that guarantees the clicking becomes automatic within a fortnight. That is not a control. It is a signature, and it will not survive the first incident it was supposed to prevent.
This matters more than it sounds, because human review is not a marginal safeguard in this industry. It is the primary one. The Measuring Agents in Production study, which surveyed practitioners and ran twenty in-depth case studies across 26 domains, found that 74% of production agent deployments depend primarily on human evaluation, with model-based evaluation such as LLM-as-a-judge second at 52% and rule-based checks at 42%. The same study found 68% of production agents execute at most ten steps before human intervention, and that bounded, predefined workflows dominate over open-ended autonomy, which is the production reality behind the distinction we drew in what is an AI agent, and does your business actually need one?. (One caution on the figures: the published versions of that paper are inconsistent on sample size, with the abstract citing 306 practitioners in most versions and 86 in another, so treat the exact n with care. The pattern is corroborated across the case studies.)
So the industry's main quality mechanism is a person looking at things. It is worth designing properly, and it is one of the five connections an AI integration has to specify rather than a policy written afterwards.
Human in the loop: four conditions, or it is theatre
Review is genuine only if all four of these hold. Failing any one produces oversight that is nominally present and structurally incapable of changing an outcome.
The reviewer can do the underlying task. A person who cannot produce the output themselves cannot judge it. They can check it is present and formatted, which is not the same thing. This is also why the review role is difficult to staff with juniors alone, and why it tends to land on the people whose time the system was meant to free up.
The reviewer sees the working, not just the answer. Retrieved sources, the fields the extraction came from, a confidence signal where one is meaningful. Given only the conclusion, a reviewer is being asked whether it sounds right, and plausible-sounding wrong answers are exactly what these systems produce, which is why the demo survives and the deployment does not.
Disagreeing is cheap. If rejecting an item means filling in a form, explaining yourself to a manager, or blocking a queue that someone is chasing you about, the rejection rate will approach zero regardless of quality. Measure how long it takes to reject something. If it is longer than accepting, you have designed for approval.
The volume is survivable. A person cannot meaningfully review four hundred items a day. If the volume requires that, the design is wrong: route by confidence and sample the rest, rather than pretending to look at everything.
Why review decays, and what to do about it
Two well-evidenced effects work against you, and both are design problems rather than diligence problems.
The first is old. Lisanne Bainbridge's Ironies of Automation, published in Automatica in 1983, made the point that automating the routine leaves the human monitoring for exceptions, while removing the routine practice that kept them capable of recognising one. Her conclusion was that automation demands more skilled operators, not fewer, and that monitoring is itself a task humans perform badly over long stretches.
The second is recent and specific. The Microsoft Research and Carnegie Mellon study of 319 knowledge workers, published at CHI in 2025, found that higher confidence in the AI predicted less critical thinking, while higher confidence in one's own ability predicted more. Read as an engineering constraint: as your system gets better, your review gets worse, and it degrades fastest among reviewers least equipped to catch the errors.
Neither is fixed by asking people to be careful. Both are addressed structurally.
- Route by confidence, do not review uniformly. Low-confidence outputs go to a person by default. High-confidence outputs get sampled at a fixed rate. This is the pattern the production case studies describe: judges score each response, low scores route to experts, and a percentage of high-confidence outputs is randomly pulled for ongoing validation.
- Keep a seeded error rate. Periodically inject known-bad items into the review queue and record whether they are caught. This measures the reviewer, not the model, and it is the only way to know whether your control is still working. Tell people it happens; the point is calibration, not entrapment.
- Log every correction with a reason code. A correction with no reason attached is a fixed record. A correction with a reason is a dataset telling you which failure modes are recurring and which prompt or retrieval change would remove them. It is also the same reason the record, rather than the model, has to hold the answer, which we argued in why your AI should never be the source of truth.
- Rotate reviewers and keep them doing the work. The best reviewer is someone who still does the task some of the time. This is Bainbridge's prescription and forty years of subsequent practice has not improved on it.
- Scale the gate to reversibility, not to importance. The useful question is not how important the output is. It is how expensive it is to undo. A wrong internal summary is cheap. A sent email, a posted journal entry or a changed price is not, and irreversible actions should hold a mandatory gate regardless of measured accuracy.
Measure the review, not just the output
Most teams instrument the model and leave the review uninstrumented, which means they cannot tell a healthy control from a rubber stamp. Four numbers are worth having from the start.
- Rejection rate over time. A rate trending towards zero is a warning, not a success. It is the most reliable early signal that review has become clicking.
- Time per review. If median review time falls below the time it takes to actually read the item, you have your answer.
- Seeded error catch rate. The direct measure of whether the control works.
- Escaped defects found downstream. Anything a customer, an auditor or a colleague found that review did not. This is the number that matters and the one nobody wants to collect.
Those four are cheap to capture at the point where the exception queue is built, and nearly impossible to reconstruct afterwards, which is an argument for building the queue before the system goes live rather than after the first bad week. Putting that queue inside the tools people already use, rather than in a new interface, is AI integration work and it is where a meaningful share of the effort lands.
The part that is a business decision
How much review is enough is not an engineering question. Our working rule is that roughly a third of the hours an AI workflow gives back should be booked for review, higher early on and where errors are expensive. That figure is a judgement from our own project work rather than a research finding, and we would rather say so than dress it up. It also assumes somebody has said out loud what the recovered time is for, which is the silence we described in employee resistance to AI is usually a rational response.
What engineering can supply is the evidence to revise it. After three months of seeded error rates, correction logs and escaped defects, you can replace the guess with a measurement, and in most cases reduce the review burden with confidence rather than by hope. That is the actual argument for instrumenting review: not that checking is virtuous, but that it is the only route to eventually checking less. Designing that gate alongside everything else that has to hold once real users arrive is what our work on building AI that works in production covers.
FAQ
What does human in the loop actually mean? It means a person is positioned in the execution path with the authority and information to change the outcome. In practice it is frequently implemented as an approval click on a screen showing only the output, which satisfies the label without providing the control. The distinction is whether disagreement is possible, informed and cheap.
How common is human review in production AI? It is the dominant quality mechanism. The Measuring Agents in Production study found 74% of production agent deployments depend primarily on human evaluation, ahead of model-based evaluation at 52% and rule-based checks at 42%, and that 68% of agents execute at most ten steps before a human intervenes.
Does human review stop AI errors? Only if designed for. Review quality degrades predictably: research from Microsoft Research and Carnegie Mellon found higher confidence in AI predicts less critical scrutiny, and Bainbridge's 1983 work on automation established that sustained monitoring is a task humans perform poorly. Confidence-based routing, sampling, seeded errors and correction logging address this structurally.
Should a person review every AI output? Rarely, and attempting it usually produces worse review than sampling would. Route low-confidence outputs to a person by default, sample high-confidence outputs at a fixed rate, and hold a mandatory gate on anything irreversible. Uniform review across items of very different consequence is a sign the design was not considered.
How do we know whether our review is working? Track four things: rejection rate over time, median time per review, the catch rate on deliberately seeded errors, and defects that escaped review and were found downstream. A rejection rate trending to zero is the clearest early warning that review has become a formality.
Can we use another model to review the output instead? As a complement, not a replacement. Model-based evaluation is used by around half of production deployments, and in the documented case studies it is paired with human review rather than substituted for it, typically by scoring confidence and routing uncertain cases to people. A model reviewing a model shares the failure modes of both.
References
- Pan, M. Z. et al.: Measuring Agents in Production (survey of practitioners and 20 in-depth case studies across 26 domains; 74% human evaluation, 52% LLM-as-a-judge, 42% rule-based; 68% executing at most ten steps before human intervention; note the sample-size inconsistency between published versions). https://arxiv.org/abs/2512.04123
- Open review record for the same paper, stating 306 practitioners surveyed. https://openreview.net/forum?id=FxNCt9xtOZ
- Lee, H. et al.: The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (CHI 2025; 319 knowledge workers, 936 first-hand examples; confidence in AI against self-confidence). https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/
- Bainbridge, L.: Ironies of Automation, Automatica, Volume 19, Issue 6, pages 775 to 779 (November 1983). https://www.sciencedirect.com/science/article/abs/pii/0005109883900468
- KPMG International: 2026 Global AI in Finance: The Decision Advantage (1,013 senior finance leaders at organisations turning over at least US$250 million; 33% increasing human oversight in response to AI-related concerns). https://kpmg.com/xx/en/our-insights/ai-and-technology/kpmg-global-ai-in-finance-report.html
Written by JP, Sixees Labs. Last reviewed September 2026.