Salesforce
Agentforce Regression Testing After a Salesforce Release

Rishabh Kumar
Published On

Table of contents
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
How Do You Regression Test Agentforce After a Salesforce Release?
Regression testing an Agentforce agent means checking, after every Salesforce release and every agent change, that the agent still routes, acts, hands off and refuses as it did before, and that its answers are still good enough. Because agents aren’t deterministic, you compare against a baseline, meaning pass rates and quality scores from repeated runs before the change, set against the same measures after it.
Some outcomes must still pass every single time, like the right record being created or a conversation reaching the right queue. Reasoning and wording are judged on whether a rate or score has moved further than the threshold you agreed in advance.
Run it in the sandbox preview window, before the release reaches your production agent.
What a Salesforce Release Can Change for an Agent
An agent is configuration sitting on a platform that moves three times a year. Even if nobody touches your topics, instructions or actions, a release can change what the agent does.
Area of change | How it can affect your agent |
|---|---|
Agent platform capabilities | New reasoning behaviours, defaults or features change how topics and actions are chosen |
Authoring and testing tools | The new Agentforce Builder, Agent Script, the Tests tab in Agentforce Studio and renamed concepts, such as topics becoming subagents, change how agents are built and tested |
Release updates | Security and access changes affect what the agent user can read or do, and how integrations authenticate |
Actions built on Flow and Apex | Platform changes to flows, Apex or screen components alter the actions the agent calls |
Channels | Messaging, routing and embedded chat changes alter how conversations reach the agent |
Connected systems | APIs the agent calls change on their own schedules |
The consequence is simple. “We haven’t changed the agent” isn’t evidence that the agent hasn’t changed. The only evidence is a before-and-after comparison.
Two Events That Count as Regressions on Their Own
Moving to the New Agentforce Builder
Salesforce’s own guidance for migrating legacy agents is to build a small baseline suite against the old agent, convert it, then run the identical suite against the migrated agent. Anything that worked before and fails after is a migration regression, not something you introduced. Treat the migration as a release in its own right.
Changing the Test Runner
Legacy Testing Center suites and the newer Agentforce Studio tests store expectations differently. When you move suites between them, rerun the baseline, so you know a changed result comes from the agent and not from the test definition.
Building a Baseline for Something That Isn't Deterministic
A traditional regression suite compares one run with an expected result. An agent suite compares a spread of results with a baseline.

Assemble a Golden Set
Choose the utterances and conversations that represent the agent’s real work, including common requests, boundary cases, handoffs and requests it must refuse. Include multi-turn conversations and the context your channels set.
Separate the Two Kinds of Check
Deterministic checks, such as the right record created, the right queue reached and no action taken on a refused request, must pass every time. Probabilistic checks, like topic routing across different phrasings and quality scores, are measured as rates and averages.
Run Each Case Several Times
One run of a non-deterministic system tells you very little. Repeat each case often enough to see its normal variation.
Set Thresholds Before You Look
Decide in advance what counts as a regression, such as any deterministic failure, a routing pass rate dropping by more than an agreed amount, or a quality score below an agreed floor. Thresholds chosen after seeing results tend to excuse whatever happened.
Keep the Baseline With Its Context
Store the results with the agent version, the Salesforce release and the date, so every comparison is like for like.
Flaky or Regressed?
The same failure can mean very different things, so classify it before reacting.
What you see | What it means | What to do |
|---|---|---|
A case flips between pass and fail on an unchanged agent | Flaky. Normal variation is wider than expected. | Rewrite the case or widen its threshold, and keep it out of release gates until it’s stable |
A case that always passed now fails consistently | Regressed | Block the release and find the cause |
A deterministic check fails even once | Regressed | Treat it as a defect, whatever the overall pass rate |
A quality score drifts down, but stays above the floor | Watch | Note it in the release evidence, and investigate before it crosses the line |
The same split between noise and real failure matters in every suite. Learn to tell them apart with flaky tests.
Plan for the Cost of Repeated Runs
Testing Center uses AI to evaluate results, and those evaluations consume Agentforce credits. Repeating every case several times, across every release and every change, adds up. Size the golden set deliberately. A tightly chosen set of cases, run often, is worth more than a huge set nobody can afford to repeat.
The Release-window Plan for Agents
When | What to do for agents |
|---|---|
Release notes publish | Read the Agentforce, Flow, Messaging and security sections, and list changes that could touch your agents |
Before preview | Confirm your agent, its actions, data libraries and channels exist in the preview sandbox, matching production |
First days of preview | Run the golden set and the end-to-end agent journeys, the same number of times as the baseline |
Comparison | Any deterministic failure is a regression. Judge probabilistic measures against the thresholds set in advance. |
Triage | For each regression, decide whether the cause is the platform, an action, the channel, the agent user or a connected system |
Before production | Rerun the full set after fixes, and refresh the baseline only for changes you made on purpose |
Production weekend | Run a small smoke set through the live channel straight after the upgrade |
Afterwards | Keep the comparison, decisions and approvals as the agent’s release evidence |
One rule saves a lot of confusion. Never refresh the baseline to make a regression disappear.
Check the exact dates for this window, from sandbox preview to production weekends, in Salesforce release cycle. And keep the comparison, with the approval that followed it, as release evidence.
Where Virtuoso QA Fits
Testing Center judges the conversation. Virtuoso QA checks what the agent actually did in the systems around it. Agent journeys are scheduled or triggered from CI/CD against the preview sandbox the day it upgrades, and the deterministic parts, such as records, routing, handoffs and refusals, pass or fail on exactly the same assertions as before the release.
Run history shows what each journey did on the previous release, next to your Testing Center results. When something fails, Touchstone separates an agent behaviour change from a moved element or an environment problem, and proposes repairs only for brittleness, for a named person to approve.
Keeping people in charge of what AI is allowed to change has a name. Explore it in human-in-the-loop AI testing.
Virtuoso QA's Salesforce test automation runs your Agentforce journeys on the preview sandbox the day it upgrades, compares records, routing and handoffs with the last release, and keeps a person in charge of every repair.
Related Reads
Frequently Asked Questions
Can a Salesforce Release Change My Agentforce Agent's Behaviour?
How Do You Regression Test a Non-deterministic Agent?
Is Migrating to the New Agentforce Builder a Regression Risk?
What is a Golden Set?
Who Should Approve an Agent After a Release?
Written by

Rishabh Kumar
AI Testing & Quality Engineering Contributor
Rishabh leads business development and marketing at Virtuoso QA. He writes about AI-powered test automation, end-to-end testing, self-healing tests, and quality engineering, exploring how these capabilities can improve test coverage, reduce test maintenance, and support reliable software releases.









