Back to blog
A thousand failed tests does not always mean a thousand separate problems. Many tests may contain the same timeout, status error, or exception.
The number of failed tests tells you how many tests did not pass. ReportPortal Unique Error Analysis shows which error logs repeat and which tests contain them. Instead of opening every failed test separately, you can review related failures as a group.
This reduces repeated test failure triage while keeping each affected test available for review.
An error group is a starting point for investigation. It does not confirm that every test in the group has the same root cause.
When to use Unique Error Analysis
Use Unique Error Analysis when:
a launch contains many failed tests;
the same exception or error message appears in several tests;
reviewing failures individually creates repeated work;
you need to apply the same defect type to related failures;
you want to compare new and known errors between launches.
See Unique Errors in one place
A launch can contain failures across many suites, tests, and steps. Finding repeated errors in the launch tree may require opening the same type of failure several times.
Unique Error Analysis groups tests by matching error logs and presents the groups in the Unique errors tab. You can open this tab from the launch or from an item level, such as a suite, test, or step.
Each group contains:
a unique error log;
the number of matched tests;
the tests included in the group.
For example, if 48 failed tests contain the same timeout message, ReportPortal can display them as one group with a MATCHED TESTS count of 48. Expand the unique error to see the tests where it occurred.
Groups are formed from unique error logs. ReportPortal may merge some small error logs and display them as one error. Review the tests and their context before assigning the same defect type to the group.
The video shows how to open the Unique errors tab, review matched tests, apply a decision, configure number handling, and compare new and known errors.
Run Unique Error Analysis
Unique Error Analysis can run automatically or manually. Automatic Unique Error Analysis is enabled by default.
After a launch finishes, open an item level and select the Unique errors tab.
If automatic analysis is disabled, you can start Unique Error Analysis:
from any item level in the launch;
from the menu next to a launch.
See the Unique Error Analysis documentation for configuration instructions and interface details.
Review groups by the number of matched tests
The MATCHED TESTS column shows the number of failed tests related to each unique error.
By default, ReportPortal displays the unique error with the highest number of failed tests first. Select the MATCHED TESTS column header to change the sorting.
Starting with a large group can reduce repeated triage work because one review may cover many affected tests. However, the group with the most matched tests is not necessarily the most serious problem.
Consider the affected product area, defect severity, logs, attachments, and test context when setting priorities.
Expand a Unique Error to review its tests
Expand a unique error to see its error log and the tests where it occurred.
For example, the same search timeout may appear in tests for several locations. The group shows that the error log repeats across those tests while keeping each test available for review.
Review the matched tests to determine whether they should receive the same defect type. This reduces the need to open the same error log separately in every failed test.
Make a Decision for selected tests
After reviewing a group, select the failed tests that require the same defect type and open the Make decision modal.
Check the available Analyzer suggestions. These suggestions are based on previously analyzed test results. If a suggestion is relevant, it can help you reuse:
a defect type;
a bug tracking system link;
a comment from a previous investigation.
Review the suggestion before applying it to the selected tests. You still decide whether the suggestion is correct.
Unique Error Analysis and ML Suggestions reduce repeated analysis and defect assignment, but they do not replace investigation.
How Unique Error Analysis works with Auto-Analysis
Unique Error Analysis, ML Suggestions, and Auto-Analysis support related but different tasks:
Unique Error Analysis: groups tests by matching error logs.
ML Suggestions: provides suggestions based on similar test items from previous investigations.
Auto-Analysis: uses previous investigations to assign defect types, comments, and bug tracking system links to matching failures.
Use Unique Error Analysis to see which failures can be reviewed together. Use ML Suggestions or Auto-Analysis to help classify failures based on previous investigations.
Configure how numbers are handled in error logs
ReportPortal provides two settings that control whether numbers in error logs are used during Unique Error Analysis. The appropriate setting depends on whether the numeric value is important for analysis.
Include numbers to analyzed logs | Exclude numbers from analyzed logs |
|---|---|
Result: Messages with different numeric values remain in separate groups. | Result: Messages that differ only by numeric values can appear in the same group. |
Example: Received status 500 and Received status 503 appear in two groups. | Example: Push timeout messages with different bookingId values appear in one group. |
Use when: The numeric value changes the meaning of the error. | Use when: The number is an ID, timestamp, duration, or another changing value that does not change the meaning of the error. |
Including numbers can be useful when a status code or count affects how the failure should be investigated. Excluding numbers can prevent messages with changing IDs or timestamps from being divided into several groups.
The Exclude numbers from analyzed logs setting would also combine messages such as status 500 and status 503. Select the setting that best fits the error logs in your project.
Review new and Known Errors with Quality Gates
Unique Error Analysis works without Quality Gates. Quality Gates is a ReportPortal premium feature that extends the analysis by comparing unique errors between launches.
If Quality Gates is available in your ReportPortal instance, you can create a Quality Gate with the New Errors rule. The rule identifies new unique errors in the current launch by comparing them with another specified launch. For example, you can compare the current launch with yesterday's launch.
When a Quality Gate with the New Errors rule is configured, use the ALL UNIQUE ERRORS dropdown to review:
All
New Errors
Known Errors
For example, timeouts and refund errors that appeared in the comparison launch can be shown as known errors. A date-validation error that appears only in the current launch can be shown as a new error.
This comparison helps teams see which errors were already present and which errors first appeared in the current launch.
See Quality Rules Configuration for setup instructions.
Review repeated failures as groups
Unique Error Analysis turns a long list of failed tests into a smaller set of error groups. Teams still need to review the tests, investigate the problem, and select the correct defect type. The feature reduces how often they need to review the same error log.
A practical test failure triage process is:
Open the Unique errors tab.
Review the MATCHED TESTS column.
Expand a relevant unique error.
Review its log and matched tests.
Select the tests that require the same defect type.
Check Analyzer suggestions.
Apply the confirmed decision.
Open a completed launch, select the Unique errors tab, and compare the number of failed tests with the number of unique error groups.