Salesforce

Agentforce Regression Testing After a Salesforce Release

Rishabh Kumar

Published On

Salesforce Release Cycle
eBook CTA Background
RECOMMENDED RESOURCES

Salesforce testing: Complex Element Selection Best Practice

While some applications are easy to automate, others, like Salesforce, need a little extra love and attention. This is our best practice guide to complex element selection.

Get the eBook →

Ebook image
eBook CTA Background
RECOMMENDED RESOURCES

Salesforce testing: Complex Element Selection Best Practice

While some applications are easy to automate, others, like Salesforce, need a little extra love and attention. This is our best practice guide to complex element selection.

Get the eBook →

Ebook image
eBook CTA Background
RECOMMENDED RESOURCES
Ebook image

Salesforce testing: Complex Element Selection Best Practice

While some applications are easy to automate, others, like Salesforce, need a little extra love and attention. This is our best practice guide to complex element selection.

Get the eBook →

eBook CTA Background
RECOMMENDED RESOURCES

Salesforce testing: Complex Element Selection Best Practice

While some applications are easy to automate, others, like Salesforce, need a little extra love and attention. This is our best practice guide to complex element selection.

Get the eBook →

Ebook image

Table of contents

Lorem ipsum

Lorem ipsum

Lorem ipsum

Lorem ipsum

Lorem ipsum

How Do You Regression Test Agentforce After a Salesforce Release?

Regression testing an Agentforce agent means checking, after every Salesforce release and every agent change, that the agent still routes, acts, hands off and refuses as it did before, and that its answers are still good enough. Because agents aren’t deterministic, you compare against a baseline, meaning pass rates and quality scores from repeated runs before the change, set against the same measures after it.

Some outcomes must still pass every single time, like the right record being created or a conversation reaching the right queue. Reasoning and wording are judged on whether a rate or score has moved further than the threshold you agreed in advance.

Run it in the sandbox preview window, before the release reaches your production agent.



What a Salesforce Release Can Change for an Agent

An agent is configuration sitting on a platform that moves three times a year. Even if nobody touches your topics, instructions or actions, a release can change what the agent does.

Area of change

How it can affect your agent

Agent platform capabilities

New reasoning behaviours, defaults or features change how topics and actions are chosen

Authoring and testing tools

The new Agentforce Builder, Agent Script, the Tests tab in Agentforce Studio and renamed concepts, such as topics becoming subagents, change how agents are built and tested

Release updates

Security and access changes affect what the agent user can read or do, and how integrations authenticate

Actions built on Flow and Apex

Platform changes to flows, Apex or screen components alter the actions the agent calls

Channels

Messaging, routing and embedded chat changes alter how conversations reach the agent

Connected systems

APIs the agent calls change on their own schedules

The consequence is simple. “We haven’t changed the agent” isn’t evidence that the agent hasn’t changed. The only evidence is a before-and-after comparison.



Two Events That Count as Regressions on Their Own

Moving to the New Agentforce Builder

Salesforce’s own guidance for migrating legacy agents is to build a small baseline suite against the old agent, convert it, then run the identical suite against the migrated agent. Anything that worked before and fails after is a migration regression, not something you introduced. Treat the migration as a release in its own right.

Changing the Test Runner

Legacy Testing Center suites and the newer Agentforce Studio tests store expectations differently. When you move suites between them, rerun the baseline, so you know a changed result comes from the agent and not from the test definition.



Building a Baseline for Something That Isn't Deterministic

A traditional regression suite compares one run with an expected result. An agent suite compares a spread of results with a baseline.

Regression Testing Agentforce After a Salesforce Release

Assemble a Golden Set

Choose the utterances and conversations that represent the agent’s real work, including common requests, boundary cases, handoffs and requests it must refuse. Include multi-turn conversations and the context your channels set.

Separate the Two Kinds of Check

Deterministic checks, such as the right record created, the right queue reached and no action taken on a refused request, must pass every time. Probabilistic checks, like topic routing across different phrasings and quality scores, are measured as rates and averages.

Run Each Case Several Times

One run of a non-deterministic system tells you very little. Repeat each case often enough to see its normal variation.

Set Thresholds Before You Look

Decide in advance what counts as a regression, such as any deterministic failure, a routing pass rate dropping by more than an agreed amount, or a quality score below an agreed floor. Thresholds chosen after seeing results tend to excuse whatever happened.

Keep the Baseline With Its Context

Store the results with the agent version, the Salesforce release and the date, so every comparison is like for like.



Flaky or Regressed?

The same failure can mean very different things, so classify it before reacting.

What you see

What it means

What to do

A case flips between pass and fail on an unchanged agent

Flaky. Normal variation is wider than expected.

Rewrite the case or widen its threshold, and keep it out of release gates until it’s stable

A case that always passed now fails consistently

Regressed

Block the release and find the cause

A deterministic check fails even once

Regressed

Treat it as a defect, whatever the overall pass rate

A quality score drifts down, but stays above the floor

Watch

Note it in the release evidence, and investigate before it crosses the line

The same split between noise and real failure matters in every suite. Learn to tell them apart with flaky tests.



Plan for the Cost of Repeated Runs

Testing Center uses AI to evaluate results, and those evaluations consume Agentforce credits. Repeating every case several times, across every release and every change, adds up. Size the golden set deliberately. A tightly chosen set of cases, run often, is worth more than a huge set nobody can afford to repeat.



The Release-window Plan for Agents

When

What to do for agents

Release notes publish

Read the Agentforce, Flow, Messaging and security sections, and list changes that could touch your agents

Before preview

Confirm your agent, its actions, data libraries and channels exist in the preview sandbox, matching production

First days of preview

Run the golden set and the end-to-end agent journeys, the same number of times as the baseline

Comparison

Any deterministic failure is a regression. Judge probabilistic measures against the thresholds set in advance.

Triage

For each regression, decide whether the cause is the platform, an action, the channel, the agent user or a connected system

Before production

Rerun the full set after fixes, and refresh the baseline only for changes you made on purpose

Production weekend

Run a small smoke set through the live channel straight after the upgrade

Afterwards

Keep the comparison, decisions and approvals as the agent’s release evidence

One rule saves a lot of confusion. Never refresh the baseline to make a regression disappear.

Check the exact dates for this window, from sandbox preview to production weekends, in Salesforce release cycle. And keep the comparison, with the approval that followed it, as release evidence.



Where Virtuoso QA Fits

Testing Center judges the conversation. Virtuoso QA checks what the agent actually did in the systems around it. Agent journeys are scheduled or triggered from CI/CD against the preview sandbox the day it upgrades, and the deterministic parts, such as records, routing, handoffs and refusals, pass or fail on exactly the same assertions as before the release.

Run history shows what each journey did on the previous release, next to your Testing Center results. When something fails, Touchstone separates an agent behaviour change from a moved element or an environment problem, and proposes repairs only for brittleness, for a named person to approve.

Keeping people in charge of what AI is allowed to change has a name. Explore it in human-in-the-loop AI testing.

Virtuoso QA's Salesforce test automation runs your Agentforce journeys on the preview sandbox the day it upgrades, compares records, routing and handoffs with the last release, and keeps a person in charge of every repair.



Related Reads

Frequently Asked Questions

Can a Salesforce Release Change My Agentforce Agent's Behaviour?

How Do You Regression Test a Non-deterministic Agent?

Is Migrating to the New Agentforce Builder a Regression Risk?

What is a Golden Set?

Who Should Approve an Agent After a Release?

Written by

Rishabh Kumar

AI Testing & Quality Engineering Contributor

Rishabh leads business development and marketing at Virtuoso QA. He writes about AI-powered test automation, end-to-end testing, self-healing tests, and quality engineering, exploring how these capabilities can improve test coverage, reduce test maintenance, and support reliable software releases.

See what your next release looks like with Virtuoso

Book a walkthrough on your applications and your workflows. Bring a requirement, a user journey, or a brittle Selenium script, and watch the loop run on something you recognise.

See what your next release looks like with Virtuoso

Book a walkthrough on your applications and your workflows. Bring a requirement, a user journey, or a brittle Selenium script, and watch the loop run on something you recognise.

See what your next release looks like with Virtuoso

Book a walkthrough on your applications and your workflows. Bring a requirement, a user journey, or a brittle Selenium script, and watch the loop run on something you recognise.

Virtuoso QA is establishing the standard of proof for software releases. Its governed QA loop turns business requirements into tests for any browser-based application: AI proposes, a deterministic engine executes, a person approves what matters, and every decision leaves evidence.

Trust Center

AICPA

SOC

WAVE STRONG PERFORMER

@ Copyright 2026 SpotQA, Creators of Virtuoso QA

Virtuoso QA is establishing the standard of proof for software releases. Its governed QA loop turns business requirements into tests for any browser-based application: AI proposes, a deterministic engine executes, a person approves what matters, and every decision leaves evidence.

Trust Center

AICPA

SOC

WAVE STRONG PERFORMER

@ Copyright 2026 SpotQA, Creators of Virtuoso QA

Virtuoso QA is establishing the standard of proof for software releases. Its governed QA loop turns business requirements into tests for any browser-based application: AI proposes, a deterministic engine executes, a person approves what matters, and every decision leaves evidence.

Trust Center

AICPA

SOC

WAVE STRONG PERFORMER

@ Copyright 2026 SpotQA, Creators of Virtuoso QA

Platform

Solutions

Resources

Platform

Solutions

Resources

Platform

Solutions

Resources