Great news, everyone! Today we are taking a deep dive into the latest Allure Report release: 2.16.1. We have been working hard to deliver our open-source users some neat features: UI improvements, new filters for unstable and new tests detection, and a 'flaky' annotation. Each of the features will make automation QA engineers easier!
Let's look at how it works!
💣 Flaky Annotation a 'Status changed after retry' marker
Flaky and New failed filter in the test results list
Handling flaky tests is a non-trivial task. Tests tend to be flaky for a whole bunch of various reasons: infrastructure instability, performance and timeout issues, environment configurations, or Moon-Mercury conjunction.
There are three hard steps in fixing the flaky test:
- Detect such a test and mark it as flaky. Detecting one may be hard in case automation QA engineers don't run tests frequently and get persistent tests history. This is why we introduced the 'Status changed after retry' marker: under the hood it analyzes retries. In case, there were failed retries followed by passed ones, the test is marked as Flaky-suspicious. This is why we added the 'Flaky' annotation. Stay tuned for automated Flaky detection from frameworks.
- Investigate the reason for flakiness. As we mentioned before, this step takes the most features. At this point, having several stack traces for each fail would help. That's the part of the job Allure does for you.
- Resolve the source of flakiness. Further by yourself! The steps to fix the test depend on a specific reason for flakiness: fix the environment, tune some performance, or double-check your infrastructure.
Get some insights about handling flaky e2e tests from the talk by Andrei Solntsev from Codeborne.
✨ New filters for your tests: 'New passed' and 'New broken'
Markers and filters for New passed and New failed
There are never enough filters! But some of them are crucial while others are nice-to-have. This time, we added two brand-new filters aimed at saving testers time.
The new ones are among the crucial ones as managers usually ask if tests have passed after deploying a fix to some environment. With the 'New passed' filter, you'll find out if previously failed tests turned green without digging into previous results.
These filters also allow checking how newly implemented tests passed in CI, and which result they have.
📋 Tables in test result description
That is quite a straightforward feature.
Many automation QA engineers need to provide some explicit data for a test case: several login-password pairs, a detailed list of elements to check, or additional context for steps or transitions. For such cases, we have added tables support for the test result description field.
By now, the table will be displayed in the test result after the run.
Table view in the Test Result description
Learn more about Allure tools
Learn more about Allure Framework and Allure TestOps, the all-in-one DevOps-ready testing platform.
Subscribe to our Twitter feed, GitHub Discussions, or Telegram community (ru-only). It is a wholesome place to get help and stay up-to-date with the news.
