Human-in-the-Loop AI Testing Without Rubber-Stamp Approvals

Human-in-the-loop fails when there are more approvals than a person can really review. When that happens, approving becomes a click. So design it the other way round: work out how many real decisions your reviewers can make per release, and only send them those.

Make every approval come with its source and its diff, and keep an eye on the edit rate for signs of rubber-stamping.

“Keep a human in the loop” has become the most repeated advice in agentic testing, and the least explained. It rarely says what the human is reviewing, against what, or how many times a day.

A human who approves everything isn’t governing anything. They’re just the last step in the pipeline, with their name on it.

If you’re still working out where AI fits into your testing in the first place, our guide to software testing and AI covers the basics before you get into approvals and governance.

Why Does Human-in-the-Loop Turn into Rubber-Stamping?

People reviewing a capable automated system tend to trust it more than they should. Training doesn’t fix that, and it gets worse when reviewers are busy. Human factors researchers call this automation bias. Without limits built into the design, an approval step attached to a high-volume AI process will drift towards accepting whatever it’s shown.

The research goes back decades, well before generative AI. In a 2010 paper in the journal Human Factors, Raja Parasuraman and Dietrich Manzey reviewed the evidence and drew three conclusions that matter for agentic testing:

Now picture a QA lead in release week. An agent proposes requirements, journeys and repairs, and most of the time the proposals are good. The lead has other work. Every approval that turns out fine teaches them the next one will too. That isn’t a character flaw. It’s just how attention works under pressure.

The usual fix, telling people to review carefully, is the one thing the research says doesn’t work. You can’t instruct your way out of automation bias. You have to design your way out of it.

For a closer look at what testing agents do on their own, and where they still need a person to step in, read our guide to autonomous testing with agentic AI.

What is an Approval Budget?

An approval budget is the number of real decisions your reviewers can make in a release. A real decision means the proposal is read against its source, checked, and then accepted, edited or rejected. The budget depends on how many reviewers you have, how much time they actually have, and how much evidence each proposal comes with. Anything beyond the budget doesn’t get reviewed. It gets clicked.

For example, two reviewers with six hours each per release, making ten real decisions an hour, have a budget of about 120 decisions. If agents propose 400 changes, around 280 of them won’t get a proper look.

The last part of that formula is the one you control. A reviewer who has to open the ticket, find the policy, work out what changed and guess at the consequences makes few decisions an hour. A reviewer who’s shown the cited source, the exact diff and what the change affects makes many more, without being any less careful.

So the rule is simple. Don’t send reviewers more decisions than they have time for, and give them everything they need to make each one quickly.

This turns “human in the loop” from a slogan into a capacity plan. If agents propose more than the budget allows, you have three honest options:

• Raise the budget by improving the evidence that comes with each proposal.

• Lower the demand by tiering decisions, so low-risk ones need less review.

• Admit that some approvals are for show, and say so openly.

The dishonest option is pretending the extra clicks are reviews.

What Should a Real Approval Include?

A real approval puts five things in front of the reviewer:

1. The source the proposal answers to

2. The exact change against the previous version

3. What it touches

4. What happens if it’s wrong

5. How to undo it.

With all five, the reviewer makes a judgement. Without them, they make a guess and sign it.

These fields also give you your evidence. An approval that records these five things is exactly what a Release Record needs in its approval field. Designing good reviews and producing good evidence turn out to be the same job.

It saves time too. If a reviewer has to go and find the source themselves, most of their review time goes on searching rather than deciding.

Illustrative Scenario: The Week the Queue Overflowed

This scenario is illustrative. It describes how things can go wrong, not a specific company.

Picture an insurer’s QA team using agentic test generation for its policy portal. At first, the agents propose a manageable number of requirements and journeys each week. The QA lead reads each one against the underwriting guide and edits a fair number of them.

Then a product refresh lands, and the agents propose several hundred changes in a single release window. The lead’s calendar hasn’t changed. The proposals arrive as whole documents, with no cited clause and no diff. By Wednesday, the lead is approving in batches. By Friday, nothing has been edited for two days.

The release ships. Every proposal has an approval with a name and a timestamp. On paper, governance worked.

Now run the same week with an approval budget:

The lead makes fewer decisions, and every one of them is real. The edit rate stays above zero, which shows someone is still reading.

How Do You Keep Approvals Within the Budget?

There are four levers:

Tracking the edit rate is cheap. You don’t need an audit or a survey, and it shows you quickly whether reviewers are actually reading what they approve.

Where Does Virtuoso QA Touchstone Fit?

Touchstone is built on the idea that AI proposes and humans govern. It’s designed to make each approval quick to do well, so your approval budget goes further.

AI reasons, a deterministic engine executes, a person approves what matters, and every decision leaves evidence.

Touchstone is part of a wider approach to agents in testing, where every agent action has a person accountable for it. You can see how that works across the platform in Virtuoso QA’s agentic AI software testing.

What Touchstone Doesn’t Do Today

• It doesn’t decide your risk tiers or who can approve what. That’s a governance decision for your team.

• It doesn’t report an edit rate for you. Count edits and rejections from your approval history.

• A single audit timeline across every asset is planned but isn’t available today. Approvals and versions are recorded on requirements and journeys.

• It tests browser-based applications. Native mobile and desktop apps, and load or performance testing, are outside its scope.

To see how AI agents draft requirements and tests that your team reviews and signs off, take a look at Virtuoso QA’s agentic test generation.

Every approval Touchstone records feeds into Virtuoso QA’s release evidence, so you can show who approved each change and what they were looking at when they did.

What Can You Do This Quarter?

Here are four things you can do without buying anything:

None of this slows AI testing down. It just means you decide where your reviewers’ time goes, instead of hoping they keep up.

When you’re ready to bring AI into your own test suite with approvals built in from day one, take a look at how Virtuoso QA handles AI-powered test automation.