Good news, everyone! Today, we are going to save a lot of time on sorting out test run results. Handling failing tests has always been a huge problem in large-scale projects. As soon as there are "hundreds of tests" suites that are launched dozens of times (in different environments or on different conditions) test management takes time.
The usual case is multiple tests failing in multiple runs due to one single issue. This is most annoying, especially when you need to investigate the reason for each red test: is it a bug? is it an infrastructure issue? is it timing or a connection timeout?
To solve this problem Allure TestOps introduces the Defect feature.
What is the Defect Feature?
The defect feature allows you to group tests by the reason of failure. How does it work?
- Filter red tests in a launch. The Launch must be in the “Open” state. Choose one of the failed tests. The test contains the stack trace of the error that caused it to fail.
- Select the test, and then click the “Link defect” button. Create a new defect, assign it a name, and an automation rule.
Create an Automation Rule
- Save the error text pattern using copy-paste and some regex spice
- Allure TestOps will suggest link all the tests with a similar error message to the defect
- The defect will then be available in the Launch.
- Once a couple of defects are created, Allure TestOps will start suggesting the defects to failed tests. These suggestions will be based on special heuristics, so you will get some smart suggestions even if there is no complete error message match
As you can see in the image, four tests have failed as a result of only two issues. Allure TestOps has been able to detect two failed tests for the Defect that has been created.
When the basic steps are done, Allure TestOps will link all the compatible failed tests to existing defects in all subsequent runs, be it pull-requests, stable branch merges, or some specific test-run. Therefore, as soon as you set up the defects for your service up, you and your fellow colleagues will only need to browse through several defects instead of dozens of failed tests.
Further Information About Defects
You can link a defect to a JIRA issue, and track the issue status and resolution either in Allure TestOps or in the JIRA interface. You will therefore only need one page to view the progress of all failed tests grouped by defects.
As soon as the JIRA issue is closed the defect will move to a “Closed” state and stop collecting red tests for its automation rule. There is two-way integration between Allure Defect and JIRA issues, so if there are more failing tests that fit the automation rule, you can reopen the JIRA issue to get the defect back. If there is no issue with which to link, you can create one from the defect interface.
Assigning the tests’ results to defects in an automated way will separate the problems in your tests into two big groups:
- Known issues i.e. defects that are already processed, analyzed, checked, and being fixed
- New issues i.e. those that still need to be analyzed
These results will allow your team to save a considerable amount of time as they will have a clear picture of what actions to perform for new issues and what steps to take to control known issues.
Learn more about Allure tools
Learn more about Allure Report, our open-source testing reporting tool, or Allure TestOps, the all-in-one quality management platform.
Subscribe to our Twitter feed!
