Functional to Visual: AI powered UI Testing from Testim and Applitools

Name: Functional to Visual: AI powered UI Testing from Testim and Applitools
Uploaded: 2021-10-20T15:09:07.000Z
Duration: 1 h 56 min 27 s

Introduction to the Webinar

The webinar is about functional to visual AI-powered UI testing. The speakers are Paz Yanover and Mark Lambert, who have extensive experience in software development and product management.

Speakers' Background

Paz Yanover is the VP of Product Marketing and Product Management at testim.io with over 18 years of enterprise software experience.

Mark Lambert is the Vice President of Product Marketing at aptly tools with over 16 years of experience in the software industry.

Agenda for the Webinar

The agenda for the webinar includes discussing challenges faced by teams in traditional test automation, leveraging AI to solve these problems, fast authoring of resilient automated tests using AI, integrating visual validation into test scenarios, running tests across multiple environments using cross-browser testing at scale, and streamlining delivery of visually perfect applications through Visual AI.

Challenges in Software Testing Today

There are three main challenges faced in software testing today - accelerated releases, increasing complexity of applications, and a wide range of browser-device combinations.

Accelerated releases have shifted from quarterly or monthly to daily or even intradaily releases.

Applications are becoming more complex with more features, pages, interactivity, and components outside of your control.

The number of browser-device combinations has exponentially increased over time.

These challenges lead to time constraints, lack of skilled resources for test automation, brittle and flaky UI tests requiring maintenance, time-consuming execution across different browsers and devices, reliance on manual testing to fill gaps which is slow and error-prone.

Current Approach to Test Automation is Broken

The current approach to test automation is not scalable and fails to meet the demands of modern software development.

Insufficient time and lack of skilled resources for creating sophisticated test automation.

UI tests are brittle and require frequent maintenance.

Execution across different browser-device combinations is time-consuming and unreliable.

Manual testing is used as a fallback, leading to late-cycle verification, errors, and defects slipping into production.

AI as a Solution for Test Automation Challenges

AI can help address the challenges in test automation and improve scalability.

Torsten Volk, an analyst at EMA, highlights the potential of AI in solving test automation challenges.

AI can streamline test creation, maintenance, and execution processes.

Visual AI can ensure visually perfect applications across various screens.

The webinar will demonstrate how Apple Tools and Testim have integrated AI to simplify test creation, maintenance, and execution.

Introduction to Integration between Products

In this section, the speaker introduces the new integration between products and discusses how it can accelerate test automation practices.

Integration Benefits

The integration between products aims to bring functional and visual validation together.

It offers a platform for super fast authoring of resilient tests.

The software provides different categories of functionality, including fast authoring, AI-powered stability, and robust root cause analysis tools.

It enables users to record the flow of tests, capture elements and objects in a logical visual flow, and edit them with various building blocks for validations, actions, or code flexibility.

AI-powered stability ensures that tests remain stable even when there are code changes.

Robust root cause analysis tools help in debugging tests.

Challenges in Test Automation

This section highlights the challenges related to test automation that often become bottlenecks for faster releases.

Test Automation Challenges

Test automation can become a bottleneck for faster releases due to various challenges.

Testing.ai's testimony platform addresses these challenges by providing an AI-based solution for super fast test authoring.

The diagram categorizes the software's functionality into different areas such as fast authoring, AI-powered stability, and robust root cause analysis tools.

Fast Authoring with Testing.ai

This section explains how Testing.ai enables super fast authoring of tests through recording flows and using out-of-the-box building blocks.

Fast Authoring Process

Users can record the flow of their tests to capture all elements and objects in a logical visual flow.

The recorded steps are displayed as thumbnails in the testing editor for easy understanding of the test flow.

Steps can be reordered or deleted as needed. They can also be grouped together for reuse in different tests.

Out-of-the-box building blocks, such as validations, multiple actions, and code flexibility, can be added to the recorded test.

The AI-powered stability ensures that tests remain stable even with code changes.

AI-Powered Stability

This section focuses on the AI-powered stability feature of Testing.ai.

AI-Powered Stability Features

The software uses AI-powered locators that represent multi-attributes to uniquely identify elements and keep tests stable.

It has a human-like mechanism for identifying and handling code changes that should not break tests.

Desktop Setups and Robust Root Cause Analysis

This section discusses the additional capabilities provided by desktop setups and robust root cause analysis tools.

Additional Capabilities

Desktop setups offer additional capabilities for organizing tests, managing their lifecycle, controlling changes, and gaining insights from test activities.

Robust root cause analysis tools enable anyone to debug a test, regardless of whether they wrote it or not.

Benefits of Testing.ai

This section highlights the benefits reported by customers who have used Testing.ai compared to their previous solutions.

Customer Benefits

Customers report faster authoring and lower maintenance with Testing.ai.

They are able to spend less time testing and accelerate their releases.

Testing.ai offers a low-code platform with maximum flexibility through the ability to add JavaScript code into tests.

Unique features like duplication autocomplete help in writing clean and maintainable tests.

Desktop capabilities go beyond reporting to efficiently manage teams and tests at scale.

Demo: Creating a Test

In this section, a live demo is conducted to showcase how to create a test using Testing.ai.

Test Creation Process

A new test is created with its own base URL as the starting point.

The recording feature captures every action performed during the test.

The recorded actions are displayed as thumbnails in the testing editor, providing a clear understanding of the test flow.

Steps can be reordered, deleted, or grouped together for better organization and reuse.

Out-of-the-box steps and custom actions can be added to enhance the test.

Demo: Test Editing and Validation

This section focuses on editing tests and performing validations using Testing.ai.

Test Editing and Validation

Recorded steps can be easily edited by changing their order or deleting irrelevant ones.

Grouping steps allows for reusing them in different tests.

Out-of-the-box validation steps are available to validate element visibility, text, properties, accessibility, and more.

Select Destination Page Loading and Text Validation

This section focuses on the process of selecting a destination and ensuring that the subsequent page is loaded correctly with the expected text displayed.

Selecting Destination and Validating Text

The test aims to ensure that after selecting a destination, the page loads properly and displays the specified text.

The expected value can be customized using parameters or JavaScript expressions.

Each step of the test is displayed in the editor, showing whether it passed or failed.

Screenshots are captured for each step, including baseline screenshots taken during initial recording and current screenshots during test execution.

Root cause analysis provides console and network logs for each step, aiding in identifying issues such as missing or invisible elements.

Running the Test

This section demonstrates running the test after setting up a simple flow and adding validation.

Running the Test

The test is executed, with each step's result displayed in the editor.

Screenshots are available for comparison between baseline (recorded) and current (test execution) states.

The test passes successfully if no failures occur.

Root Cause Analysis Features

This section highlights root cause analysis features provided by the testing tool.

Root Cause Analysis Features

Detailed analysis is available for each step, including console logs and network logs.

Failure causes can be easily identified by examining spreadsheets for missing or invisible elements.

Console.log and network log data can be reviewed for all test results.

Stability and AI Algorithm

This section discusses stability in tests through an AI algorithm that adapts to changes made by developers without causing unnecessary failures.

Stability and AI Algorithm

Elements are identified based on their properties, parents, and ancestors rather than specific selectors.

This robust mechanism allows the algorithm to recognize elements even if they are updated by developers.

The test can be paused at a specific point, allowing for updates to the element being tested.

Changes made to the element, such as text or background color modifications, are recognized by the AI algorithm.

Disabling breakpoints and continuing the test execution shows that the test still passes despite changes made to the element.

Testos Capabilities

This section briefly mentions additional capabilities of Testos that couldn't be demonstrated due to time constraints.

Testos Capabilities

Users are encouraged to explore Testos' app, tests, pull requests, branches, duplication level analysis, and identification of flaky tests.

Integration with Apple Tools for Visual Validation

This section introduces integration with Apple Tools for visual validation of elements and pages.

Integration with Apple Tools for Visual Validation

Visual validation steps can be added to verify the appearance of elements or pages.

Different environments (browsers, resolutions) can be selected for visual validation.

Match levels (strict, exact content, layout) can be chosen for comparison.

RCA (Root Cause Analysis) and displacement settings can also be configured.

The test is executed with visual validation performed using Apple Tools' ultra-fast grid on various environments.

New Section

In this section, the speaker discusses the main environment and demonstrates how tests can be run across multiple environments. They also mention the availability of screenshots and direct viewers to a tools application for more results.

Main Environment and Running Tests (0:23:57 - 0:24:14)

The speaker shows screenshots of the main environment.

Tests can be run across browsers or locally.

Low code features of testing are briefly mentioned, including custom coded steps.

Application testing integration is highlighted as a holistic approach to functional and visual testing.

New Section

This section provides a recap of what has been shown so far, emphasizing the ease of test creation, various low code features available for testing, and the integration of application testing.

Recap of Test Creation and Features (0:24:14 - 0:24:39)

Test creation is highlighted as being easy.

Tests can be run across browsers or locally.

Low code features are briefly mentioned, with an emphasis on custom coded steps.

Application testing integration is discussed as providing a holistic approach to functional and visual testing.

New Section

The speaker hands over control to another person who will continue demonstrating the application's side.

Transition to Application Demonstration (0:24:39 - 0:25:04)

The speaker hands over control to another person for the application demonstration.

New Section

The new speaker continues from where the previous one left off by providing more details about Apple Tools' interface and showcasing its capabilities in visual AI for validation and cross-browser testing.

Visual AI for Validation and Cross-Browser Testing (0:25:04 - 0:25:27)

The speaker mentions that visual validation can be done on the entire page, not just individual elements.

Apple Tools' dashboard is shown, displaying multiple browsers analyzing the same home page.

Integration with visual validation steps and setting up different environments is mentioned.

Options such as RCA (Root Cause Analysis) and element displacement are briefly discussed.

New Section

The speaker dives deeper into the capabilities of Apple Tools' visual AI, focusing on element displacement and demonstrating how it handles differences between versions of a page.

Element Displacement and Visual Differences (0:25:27 - 0:26:06)

Element displacement is explained as a challenge for traditional visual validation technology when elements have moved.

Apple Tools offers techniques to handle element displacement, including element displacement itself.

A full-page screenshot comparison is shown, highlighting visual differences in pink.

Zooming in reveals slight movement of images due to element displacement.

New Section

The speaker demonstrates the power of different match algorithms in Apple Tools' visual AI for handling dynamic content and defining custom regions for analysis.

Match Algorithms and Custom Regions (0:26:06 - 0:27:04)

Different match algorithms are introduced as a powerful technique in visual AI.

A custom region is defined on the page to focus on layout variations while ignoring content changes.

Applying layout match level reduces visual variations and focuses on specific areas of interest.

An animated GIF example is shown to demonstrate handling dynamic web content.

New Section

The speaker continues showcasing Apple Tools' capabilities by analyzing a full-page example using different match algorithms and addressing specific defects found.

Analyzing Full Page with Different Match Algorithms (0:27:04 - 0:28:06)

The speaker demonstrates how different match algorithms can be applied to analyze a full page.

Applying layout match level reduces visual variations and focuses on specific areas of interest.

Specific defects are identified, such as missing logos or unintended changes in text.

Defects can be annotated, assigned, and tracked within Apple Tools.

New Section

The speaker concludes the demonstration by summarizing the capabilities of Apple Tools' visual AI for analyzing full pages, defining regions, and handling element displacement.

Summary of Apple Tools' Visual AI Capabilities (0:28:06 - 0:30:13)

Analyzing full pages with different match algorithms helps identify visual differences.

Custom regions can be defined to focus on specific layout variations while ignoring content changes.

Element displacement is handled effectively by Apple Tools' visual AI techniques.

Defects can be easily annotated, assigned, and tracked within the tool.

Analyzing Browser Rendering Differences

In this section, the speaker discusses the differences in rendering between Chrome and Internet Explorer browsers. They explain that while these differences may not be visually impacting to humans, they are important to analyze for strict analysis purposes.

Understanding Rendering Differences

The combination of Chrome and Internet Explorer browsers can result in differences in how a document page is rendered.

These differences may not be visually noticeable to humans but can impact font rendering and other elements.

Applying a layout match algorithm helps identify these differences accurately.

Identifying CSS Interpretation Issues

This section focuses on a specific defect found in the bottom right of the display, highlighting how Chrome and Internet Explorer interpret CSS differently.

CSS Interpretation Differences

Chrome correctly renders white text on a blue background.

Internet Explorer incorrectly interprets the CSS, resulting in blue text on a blue background.

While this defect may not be noticeable to humans, it can be identified through visual AI analysis as a safety net issue.

Root Cause Analysis for Visual Defects

The speaker explains how root cause analysis helps identify the underlying elements causing visual defects on a webpage.

Using Root Cause Analysis

Enabling root cause analysis in Applitools allows for easy identification of elements causing visual defects.

By selecting an issue in Applitools, one can pinpoint the element responsible for the problem within the DOM (Document Object Model).

This feature significantly reduces the time required for analyzing and fixing visual defects.

Cross-Browser Testing at Scale

This section discusses challenges faced when conducting cross-browser testing at scale and how visual AI can help overcome them.

Challenges of Cross-Browser Testing at Scale

Conducting cross-browser testing across various devices, resolutions, and browsers can result in a large number of screenshots to review.

Changes that impact every page, such as rebranding or color scheme changes, require extensive review time.

Reviewing all these screenshots manually is time-consuming and may lead to missing defects or blind acceptance.

Leveraging Visual AI for Efficient Testing

Visual AI analysis helps categorize and group screenshots, reducing the effort required for review.

By analyzing differences across multiple screenshots, visual AI identifies visual regressions efficiently.

This approach allows for quick identification and remediation of issues during cross-browser testing.

Integration of Test Infrastructure with Applitools

The speaker explains how integrating visual validation and snapshots within the test infrastructure enables efficient analysis of HTML and CSS elements.

Integration Benefits

Integrating visual validation and snapshots in the test infrastructure extracts the document model (HTML and CSS) for analysis.

The extracted data is sent to Applitools' ultra-fast test cloud for further processing.

This integration streamlines the process of identifying visual defects and ensures efficient collaboration between Test M and Applitools.

New Section

In this section, the speaker discusses a platform that accelerates cross-browser testing and allows for faster execution. The integration of Test M and Apple Tools simplifies test creation, reduces test maintenance, streamlines root cause analysis, and shortens release cycles.

Complete and Scalable UI Testing Practice

The combination of Test M and Apple Tools provides a complete and scalable UI testing practice.

Test creation is simplified through Test M.

Single screenshot validation eliminates the need to manage element locators.

Smart locators from Test M help handle brittle element locators.

Single visual validation from Apple Tools reduces maintenance time.

Shorter Release Cycles

Leveraging Test M allows for full test lifecycle management.

Different tests can be managed in different states.

Integration with Apple Tools' ultra-fast test grid accelerates cross-browser testing.

New Section

In this section, the speaker mentions how users can get started with Apple Tools by creating a free account. They also encourage reaching out to their teams for more information on the integration between the two platforms.

Getting Started with Apple Tools

Users can create a free account at appletools.com.

Testing can be initiated for free using Test M.io.

Reach out to their teams to explore the new integration between the platforms.

New Section

In this section, the speaker transitions to answering questions from the audience regarding Jira support, dynamic data handling, headless mode compatibility, platform suitability for developers, and integrations with other tools.

Jira Support and Integrations

The Apple Tools dashboard integrates with Jira and CA Rally for ticket creation and defect tracking.

Slack integration facilitates collaboration within teams during the process.

Microsoft Teams integration is coming soon.

Dynamic Data Handling and Headless Mode

Visual validation handles dynamic data.

Layout options can handle changing content.

Ultra-fast grid runs in headless mode for rendering across different platforms.

Platform Suitability for Developers

Apple Tools is suitable for developers.

Users can author tests quickly, allowing more time to focus on building features.

Custom steps with JavaScript code provide flexibility for developers.

Encourages collaboration between team members, including developers, automation engineers, and manual testers.

New Section

In this section, the speaker continues answering questions from the audience regarding visual validation of dynamic data and the suitability of Apple Tools for developers.

Visual Validation of Dynamic Data

Visual validation handles dynamic content that changes over time.

Test M can be used to validate specific dynamic data elements like dates.

Suitability of Apple Tools for Developers

Many users of Apple Tools are developers.

The platform allows fast test authoring and maintenance, freeing up time to focus on building more features.

Custom steps with JavaScript code offer flexibility and encourage collaboration among team members.

Collaboration between Developers and UI/UX Teams

This section discusses the collaboration between developers, testers, and UI/UX teams in the context of functional test automation. It also highlights how Apple Tools' dashboard facilitates cross-team collaboration.

Collaboration with UI/UX Teams

The automation process allows collaboration between developers and testers.

UI/UX teams can also collaborate using the Apple Tools dashboard.

Business experts who are not technical can easily participate in functional test automation.

The dashboard enables cross-talk collaboration with UI/UX teams.

Visual validation can be done by comparing a sketch file or any image uploaded to create a baseline.

The visual AI in the dashboard helps identify correct implementations per design specifications.

Streamlining the Process

The dashboard provides a collaborative environment for reviewing and approving UI changes.

Baselines can be updated with new versions of the UI after sign-off, allowing ongoing monitoring of UI/UX elements.

Localization support is available, allowing localization experts to review specific text without having to review the entire web page.

Data-driven Testing

This section addresses data-driven testing and its support within Apple Tools.

Support for Data-driven Testing

Parameters can be defined in tests and used in any step.

Multiple ordered data sets can be defined, and tests can run against each set separately.

Data-driven testing can be configured as part of the test or passed through CLI configuration.

Testing Graphs and Charts

This section explores how Apple Tools supports testing graphs and charts.

Validating Renderings

Apple Tools allows validation of graph and chart renderings.

Different matching algorithms can be applied based on requirements (e.g., strict match or content-related algorithm).

Specific use cases may require reaching out to the Apple Tools team for further guidance.

Mobile Testing Support

This section discusses the support for mobile testing within Apple Tools.

Platform Agnostic

Apple Tools' screenshot analysis is platform agnostic.

Images from desktop, mobile devices, native apps, web pages, or PDF files can be tested.

Pharmaceutical industries use Apple Tools to analyze PDF content.

Integration with AWS and Azure

This section addresses integration with AWS and Azure in the context of testing.

Transparent Execution

The location of test execution does not matter when using Apple Tools.

Tests can be run on grids transparently without worrying about backend infrastructure.

Integration details should be clarified with more specific information about requirements.

Training Data for AI

This section discusses the training data required for leveraging AI capabilities in testing.

Building AI Capabilities

Two approaches: building your own AI or utilizing existing AI capabilities like Apple Tools.

Building your own AI requires developers experienced in working with AI techniques.

Training data requirements depend on the specific implementation and techniques used.

New Section

This section discusses the training and accuracy of Apple Tools' algorithm.

Training and Accuracy

Apple Tools has been trained on over a billion images, resulting in 99.9999% accuracy.

The algorithm does not require additional training, as it is already highly accurate.

New Section

This section addresses the steps for integrating with Applitools and the support for accessibility.

Integration Steps

The integration with Applitools involves three main steps: visual validation for elements, viewport, and the whole page.

For more detailed information on integration steps, it is recommended to reach out to the Applitools team.

Accessibility Support

Applitools supports accessibility testing through its recent release of Amplitude's Eyes, which includes a VPAT (Voluntary Product Accessibility Template) designed for accessibility.

Additionally, Apple Tools Eyes provides Contrast Advisor, which analyzes images and flags potential contrast issues within a page to meet WCAG compliance requirements.

Users can also check the accessibility of specific elements or entire pages using code that ensures standards are met. Different accessibility standards can be chosen for testing purposes.

New Section

This section briefly mentions upcoming follow-ups and concludes the discussion.

Wrap-up

There are remaining questions that will be followed up on separately after the session ends.

The discussion covers all topics on the list provided during the session.

Participants are encouraged to contact the teams for further information and will have access to free accounts.

The session concludes with gratitude for everyone's time and the hope of future conversations.