Back to blog
A failed test does not always require a new investigation.
In mature test suites, many failures are variations of problems the team has already seen: the same product defect appears in another run, a known environment issue affects several tests, or an unstable automation component fails again.
An LLM-based assistant can investigate these failures, inspect logs, correlate test results, and suggest likely root causes. But asking an LLM to reason about every failure independently has an obvious downside: the same analysis may be repeated again and again.
A more efficient approach is to combine two different capabilities:
ReportPortal Auto-analyzer reuses historical defect classification knowledge for failures similar to previously analyzed results.
An AI assistant connected through ReportPortal MCP investigates failures that require deeper reasoning or do not yet have enough historical context.
The result is a more efficient triage workflow: reuse existing knowledge, apply LLM reasoning when needed, and save the outcome so similar failures are easier to resolve in the future.
How the combined workflow works
The two components solve different parts of the same problem.
ReportPortal Auto-analyzer: reuse known decisions
ReportPortal already stores test execution history, logs, defect classifications, and previous triage decisions. Its Auto-analyzer can use that history to identify failures similar to previously analyzed test results and apply existing defect information automatically.
This is particularly useful for:
recurring product defects;
known automation issues;
repeated system or environment failures;
failures similar to previously investigated test results;
large launches where many failures represent already known problems.
Instead of starting a new investigation every time, the system reuses information the team has already produced.
AI assistant through MCP: investigate what is new
An LLM-based assistant becomes more valuable when the answer is not already present in historical triage data.
Through ReportPortal MCP, an AI assistant can work with ReportPortal data as part of its investigation. Through the available MCP capabilities, it can analyze failure context, logs, related test results, historical executions, and other relevant information exposed by ReportPortal.
This is useful for:
new or unfamiliar failures;
unclear error messages;
failures requiring several pieces of evidence to be correlated;
clusters of tests that may share a root cause;
situations where an engineer wants an explanation, not only a classification.
The key difference is that the LLM helps analyze and explain failures, while ReportPortal stores and reuses test and triage knowledge over time.
The workflow looks like this:
The key is not to choose between Auto-analyzer and an LLM. It is to route each type of failure to the mechanism best suited to it.
Prerequisites
You don't need to build a complete test environment from scratch to follow this tutorial.
You need:
A ReportPortal instance. You can use your existing deployment or the ReportPortal Demo Instance.
Automated tests reporting results to ReportPortal. If you don't have a suitable project, you can use the Playwright integration example from the ReportPortal examples repository.
ReportPortal Auto-analysis enabled and configured for the project. See Auto-analysis of launches for details.
ReportPortal MCP Server connected to an MCP-compatible AI client, such as GitHub Copilot, Claude, Cursor, or another MCP-capable assistant. If you're using the ReportPortal Demo Instance, you can connect directly to its remote MCP Server. See Connecting to a remote MCP Server for configuration instructions.
For this tutorial, prepare a test execution with at least one new failure that has not been triaged before. We will investigate this failure with an AI assistant through MCP, save the resulting triage decision in ReportPortal, and then reproduce the same or a similar failure in a subsequent execution.
This lets us demonstrate the complete feedback loop from scratch:
new failure → AI-assisted investigation → saved triage decision → historical knowledge → automatic reuse by Auto-analyzer
Step-by-step: build the triage loop
Step 1 — Run the tests and capture 5 new failures
Start by running the example’s “basic” test suite and reporting the results to ReportPortal.
For this tutorial, the launch contains:
10 tests in total
5 passed
5 failed
The important condition is that the five failures have not been triaged in ReportPortal before.
At this point, ReportPortal has the execution results, failed test items, and their logs, but it does not yet have relevant historical triage decisions for these failures.
This gives us a clean starting point.
Expected result: The launch clearly shows five failures that require analysis.
Step 2 — Investigate the new failures with AI through MCP
Connect your MCP-compatible AI client to ReportPortal and ask it to investigate the five failed tests from the launch.
Instead of opening every failed test individually and manually comparing logs, you can start with the complete failure set and let the assistant reason across the available execution context.
For example:
Analyze the five failed tests from the latest launch. Group failures that appear to share the same root cause, explain the evidence, and suggest an appropriate defect classification for each group. Use ReportPortal MCP server to retrieve the data.
You can also focus the investigation on defect categories:
Investigate the five failed tests from the latest launch and determine whether each failure is most likely caused by the product, test automation, or the system/environment. Explain the evidence behind each conclusion. Use ReportPortal MCP server to retrieve the data.
Or use a more exploratory prompt:
Analyze the five failures from the latest launch. Look for common patterns in their logs and execution context, identify likely root causes, and tell me which failures appear to be related. Use ReportPortal MCP server to retrieve the data.
Investigating the failures together is useful because apparently different errors may still originate from the same underlying problem.
Expected result: The AI assistant retrieves relevant ReportPortal context and produces an initial analysis of the five failures.
Step 3 — Review the AI-assisted analysis
The AI response should help turn raw failure data into a structured triage hypothesis. A useful analysis may include:
groups of related failed tests;
likely root cause for each group;
evidence from logs or execution context;
relationships between failures;
suggested defect classifications;
potentially related historical results or defects, where relevant.
For example, three of the five tests could show different failure messages but have the same root cause.
The AI assistant accelerates the investigation, but its conclusions should remain reviewable. An engineer can inspect the evidence, ask follow-up questions, and correct the proposed classification where necessary.
As a result, the analysis suggests that all five failures are automation issues caused by two distinct root causes.
Expected result: The engineer has a clear, evidence-based triage proposal for all five failures.
Step 4 — Save the reviewed triage decisions in ReportPortal
Once the AI-assisted analysis has been reviewed, apply the final defect classifications to the failed test items in ReportPortal.
Use the next prompt:
Apply suggestions regarding defect types to ReportPortal, along with comments explaining the causes of the failures.
This is the step that turns a one-time AI investigation into reusable project knowledge.
The AI conversation helps explain the current problem, but ReportPortal is where the reviewed decision becomes part of the persistent test execution history.
After this step, the five failures from the first launch are no longer unknown: they now have validated triage information associated with them.
Expected result: The previously new failures now have persistent triage decisions in ReportPortal.
Step 5 — Run the tests again and let Auto-analyzer reuse the new history
Run the test suite one more time to reproduce the same failures.
For the demonstration, the second launch can again contain 10 tests, including failures that correspond to those analyzed in the first execution.
This time, ReportPortal already has historical triage data created during the previous analysis cycle.
As the new launch is processed, Auto-analyzer automatically compares the current failures with previously analyzed test results. Where it finds an appropriate historical match, it can reuse the existing defect classification without requiring another AI-assisted investigation.
What was a new problem in the first launch has now become reusable triage knowledge.
Expected result: Failures that required AI-assisted investigation in the first launch are automatically classified using ReportPortal's accumulated analysis history.
This completes the feedback loop: the first execution creates the knowledge, the next execution benefits from it.
Why combine Auto-analyzer with an LLM?
LLMs and Auto-analyzer are effective at different stages of defect triage.
An LLM is most valuable when a failure requires new reasoning: correlating logs, comparing test results, identifying patterns, and proposing a likely root cause. Auto-analyzer is most valuable once that reasoning has already produced a reviewed decision that can be reused.
Together, they complement each other: LLMs help investigate new or unclear failures, while ReportPortal automatically applies existing triage knowledge when similar failures happen again.
This also avoids repeatedly sending the same failure context through an LLM, reducing unnecessary AI calls and duplicated token usage.
Use AI when new reasoning is required. Reuse what has already been learned.
Make every investigation reusable
The practical benefit is that triage knowledge does not disappear when an AI conversation ends. Reviewed decisions remain in ReportPortal and can continue to provide value across future test executions.
For QA teams, this means:
less repetitive manual triage;
fewer unnecessary AI investigations;
lower duplicated LLM context and token usage;
reuse of previous engineering decisions;
more time to focus on new failures;
more consistent defect classification over time.
The objective is not to make an LLM analyze every failure.
It is to make each new investigation useful beyond the execution where it first happened.
Explore more ReportPortal MCP use cases
Defect triage is only one way to combine ReportPortal data with an AI assistant. For more practical examples, see ReportPortal MCP Server: Practical Use Cases for AI-Powered QA Teams.