Back to blog
Flaky tests are the wild cards in the software testing world. Flaky tests happen when a test often changes its status from passed to failed and vice versa under the same conditions. Flaky tests can cause a lot of delays, questions, and chaos for software developers and QA teams.
Flaky test cases table widget
ReportPortal allows to find flaky tests in your runs. For that, create a Flaky test cases table widget in our test automation results dashboard: specify the Launch name and the Launches count for comparison.
The widget is built by the name of the launch, and not by the filter. To obtain all the data in this widget, make sure that the launches in which you want to address flaky tests have the same name.
The Flaky test cases table widget shows the 50 most unstable test cases in the selected launches. This includes not only Failed tests but also tests that change their status from Passed to Failed, from Failed to Passed.
This widget is dynamic, it only needs to be built once, and it will update automatically.
When creating the widget, we can also specify whether to include Before and After methods. Before and After methods are preconditions/postconditions, for example, creating test data, and then, after running the tests, cleaning them up. The reason for flakiness is not always within the test itself – the methods can also be flaky. For instance, the test itself might fail if generating test data fails. In the Launch, the reporting may be fine, and the test case would have passed if the Before or After method was functioning correctly.
Therefore, when a certain number of runs have accumulated, especially in the case of large launches, it is challenging to analyze failed automated tests. You can create this widget, investigate the unstable tests, and understand if there is an issue with the Before/After methods. The problem may not be in functionality, but in automation.
Additionally, with the flaky test widget, you can identify issues with the environment or with a specific version or branch. For example, if a test runs on different environments and passes in some, fails in others, you can temporarily remove these tests from the scope until the environment-related issue is resolved, saving time on running and analyzing flaky tests.
How to avoid flaky tests?
For those seeking ways to prevent flaky tests, consider the following best practices:
Stable Test Environment
Check the availability of critical resources for test execution. Control external factors (e.g., system resources, network connectivity).
Test Isolation
Decrease the number of connections between tests. It can significantly minimize the risk of instability.
Optimal Test Timing
Properly plan your tests to ensure optimal performance for your testing environment. Remember about network congestion and system load.
Test Data Management
Do not use mutable or shared data that can cause flakiness.
Overall, if you know what causes flaky tests and apply best practices to prevent them, you can reduce the impact of flaky tests.