AI Agents in QA: Which Decisions to Hand Over

An AI agent in QA should own decisions that are easy to undo, low risk and backed by clear evidence, like fixing a locator after a UI change. It should never own decisions about what the business needs, or whether a release ships. Everything in between should be handed over one type of decision at a time, as the agent’s track record shows it gets them right.

The debate about autonomous testing usually gets framed as an on-off switch: either trust the AI, or keep a person in every loop. Both fail. Full autonomy hands judgement to a system nobody can hold accountable. A person in every loop brings back the bottleneck agents were supposed to remove.

The more useful question is narrower: which decisions, based on what evidence, and how should that change over time?

What Should AI Agents Own in QA, and What Shouldn’t They?

Decide based on three things about the decision itself, not on what the AI model can do:

Decisions that are easy to undo, low risk and backed by evidence can be handed to an agent. Decisions about what the business wants, or about accepting risk on its behalf, stay with people however good the agent gets.

We call this the QA Decision Rights Matrix:

The last three rows aren’t waiting for better AI. They’re judgements about intent and risk, and someone has to be accountable for them. A senior manager can be held accountable. An agent can’t. That’s about accountability, not capability, and it’ll still be true when AI models are far better than they are today.

For more on how to keep people’s approvals meaningful as AI proposes more changes, read our guide to human-in-the-loop AI testing.

How Should an AI Agent’s Autonomy Grow?

Through an Autonomy Ledger: a written record, for each type of decision, of what the agent can do on its own, where, and based on what track record.

A decision type moves up a level when the record shows the agent’s proposals are accepted without changes most of the time, over a meaningful period. It moves back down as soon as the record shows otherwise. The agent gets more freedom when the evidence supports it, not when the team just feels good about it.

Each entry in the ledger has five fields:

Here’s what a ledger entry might look like in practice:

Moving up a level should be specific and a bit dull. It’s not “the AI seems good now”. It’s “over the last three releases, locator repairs on non-critical journeys were accepted unchanged almost every time, and the one rejection was on a legacy page we’ve now excluded”. Moving down should be just as mechanical.

One wrong decision by the agent drops that decision type back a level until the cause is understood.

Two rules keep the ledger honest:

• Levels apply to one decision type and one scope at a time, never across the board. An agent trusted with locators on the marketing site hasn’t earned anything on the claims system.

• The “people only” decisions never go in the ledger. Interpreting intent, accepting risk and deciding a release is ready never move up a level.

This is what governed autonomy looks like day to day. AI proposes, people decide, and the line between them only moves when the evidence says it should.

Illustrative Scenario: Moving One Decision Type Up a Level

This scenario is illustrative. It describes a process, not a specific company.

A retailer’s QA team runs journeys across its ecommerce site, order management system and customer service platform. At first, every repair the testing agent suggests goes to a person to approve.

After a few months, the QA lead notices the review queue is mostly locator repairs on browsing and marketing journeys, and nearly all of them are accepted without changes.

So the lead opens a ledger entry:

• Decision type: locator repair

• Scope: browsing and content journeys only, with checkout, payments and returns excluded

• New level: acts and reports, so the agent applies the repair and it appears in a daily digest that a named tester reviews

• Track record: attached, and the lead signs it off

Two releases later, one repair in scope points a test at the wrong promotional banner. The daily digest catches it. That decision type goes back to “proposes only” for that journey while the team investigates. It turns out the page has two almost identical components. The page is taken out of scope, and the decision type moves back up.

Meanwhile, checkout repairs stay at “proposes only”, and release decisions never go in the ledger at all. The team spends less time reviewing low-risk repairs and more time on the decisions that matter.

The honest part: a ledger only works if someone keeps it up to date. It adds a little governance work, and teams under pressure will be tempted to move decisions up a level based on a hunch. Without a written record, an agent’s freedom creeps up rather than being earned, and nobody notices until a decision goes wrong in an area that was never formally trusted.

Where Does Virtuoso QA Touchstone Fit?

Touchstone is built on governed autonomy: AI proposes, and people decide. The line between them sits where the Decision Rights Matrix puts it.

AI suggests what to test and how to fix it, a deterministic engine runs the tests, a named person approves what matters, and every step is recorded.

Available Today

To see how agents work within these limits across your applications, take a look at Virtuoso QA’s agentic AI software testing.

Coming Next

The direction is Autonomous QA Intelligence: agents that take on more routine decisions as the application intelligence graph builds up evidence about your systems. Autonomy will grow one decision type and one scope at a time, with the track record visible and the people-only decisions staying with people. This is on the roadmap and isn’t available today.

What It Doesn’t Do

Touchstone doesn’t decide what your business needs, accept risk on your behalf or declare a release ready. It tests browser-based applications. Native mobile and desktop apps, and load or performance testing, are outside its scope.

When someone asks who decided a release was ready, and on what basis, Virtuoso QA’s release evidence shows each test, its result, who approved it and which build it ran on.

What Can You Do This Quarter?

Here are four things you can do without buying anything:

Full autonomy isn’t the right goal for now. Handing over decisions one at a time, based on a written track record, lets you move faster and still know who decided what.

Related Reads

• The Shift from Automation to Autonomy: What Are AI Agents?

  
  

• Agentic AI Vs AI Agents - 12 Key Differences

  
  

• AI Agents for Software Testing: Agentic Automation & Self-Healing