Our Story

The more we have moved to the cloud, web, and mobile interfaces, the harder this has become for teams. Tests and test automation have become a burden to create, maintain, and scale.

We have reimagined how software is tested by developing a game-changing platform that is already being used by the biggest names in software. We passionately believe that anyone should be able to create and maintain tests regardless of their coding ability, and that quality is a key driver for change and growth. The latest advances in AI and Machine Learning have been leveraged to produce test automation software that thinks like a human, empowers everyone to test, and for the first time, delivers on the promise of codeless test automation.

About the Client

The Nutanix Cloud Platform comprises hybrid cloud infrastructure and multi-cloud management along with unified storage, database services, and desktop services to support multiple applications and workload across locations.

Key Requirement: Web Application Testing

Types of Testing: Functional testing, Automation testing

Total 2 teams: R&D Calm, PIT

Duration: Ongoing

Key challenges faced by Nutanix before Virtuoso

1. Calmtest (in-house automation suite) limitations:

• High skill requirement in coding (both design and implementation)
• Difficult to keep up pace with UI feature development, specifically when UI/UX change rate (common problem) is high
• This led to longer ship times or incomplete testing

2. A gap between an SDET and QA:

SDET with high coding skills and poor QA mindset is equally inefficient and ineffective to a seasoned QA with high QA mindset and poor coding skills. We needed to bridge this gap.

3. Less application coverage

This was a result of Calmtest limitations.

4. Test maintenance issues

Calmtest - based on popular open source framework Pytest - needed a lot of refactoring of scripts after any changes in the application.

Challenges resolved using Virtuoso

1. Ease of learning and setup:

• Virtuoso brings true codeless automation to the table
• Skill requirement is minimum with manual testers creating scripts using simple English-language statements
• Users transformed into Virtuoso automation champions within a week

2. Self-healing and quick test maintenance enables QA to keep pace with rapid changes to the UI

• Because it has self-healing properties, there is almost zero breakage when locators for UI element changes
• Pre-commit tests have been running smoothly without any breakage
• Re-usability: changes in one checkpoint reflects throughout

3. Shorter ship times compared to Calmtest

• Test Automation development now roughly 10X faster
• Direct saving of 15-20 mins on every code commit after using Virtuoso. That’s a direct impact on productivity

4. Bridges gap between SDET and QA:

• True shift-left. Even Developers can fix automation bugs when they break it

5. Greater application coverage:

• Approx. 300 complex test scenarios were automated using Virtuoso. On an average, three journeys were created every day. A total of two webapps and two browser-OS configurations were automated

6. Better scaling:

• Cross-browser, cross-device, and execution planner scales up execution rapidly

7. Lesser cost:

• Less infra costs as platform is purely cloud based
• Cost savings through improved efficiency and engineer productivity
• Hiring time significantly reduced

Some more Virtuoso benefits:

• Covered testing of applications that are not hosted publicly through Virtuoso bridge
• Covered testing of complex, end-to-end flows handling multiple iFrames and page load issues effectively through best practices like adaptive waits
• Live Authoring to the rescue: increase in accuracy through Live Authoring, ensuring tests don’t fail on reruns
• Dedicated support from Virtuoso team for help with test building and advanced support for API tests, etc. whenever required. Dedicated slack channel for quick query resolution and super fast turnarounds.

R&D Calm team leading the way with highest automation numbers

Users become automation testers in record time.

• A user new to Virtuoso became a pro in just one week of training and practice
• Authoring speed is fantastic at an average of 20 journeys created per day during peak authoring times
• The current level of automation coverage was achieved with just ten regular Virtuoso users - big savings on resource costs
• Currently, Virtuoso is being used by two teams across the company with the R&D Calm team emerging as the champion in terms of achieved automation coverage

Significant increase in automation coverage

Overview

A high volume of manual tests (approx. 300 complex test scenarios) were automated using Virtuoso. On an average, three journeys were created every day. A total of two webapps and two browser-OS configurations were automated

Virtuoso Stats

• 31 Projects
• 82 Goals
• 628 Active journeys (including drafts) as of June 2022
• Each journey testing almost 150-200 steps on an average