One of the most important features of Allure TestOps is Smart Test Cases (what used to be called Live Documentation). In a nutshell, it means that test cases can be generated automatically based on the code of automated tests. This is not something that any other tool will do for you, and there are some very obvious benefits for automated QA here (such as saving time writing documentation). Going further than that, the feature can be used to restructure your testing workflow and have automated and manual testing under the same roof. Let's take a look at what alternatives to Smart Test Cases there are in other tools, why Smart Test Cases are so important, and finally, examine step-by-step how the feature works.
The problem and alternative solutions
The purpose of Smart Test Cases is to create documentation for your automated tests. Usually, this is done by hand, which means the process is time-consuming and error-prone. Also, as the tests are updated, the documentation becomes obsolete, so maintaining it always requires additional work. A related issue is that the documentation and the tests themselves reside in different spaces and might be written by different people. There is no single source of truth, and reading the documentation is not equivalent to looking at the code.
There have been several attempts to generate test documentation automatically based on the code of the tests - mostly we're talking about small projects such as docs-from-tests or test-docs. However, those are not integrated into a framework, and the solution we're offering is somewhat different: the form of documentation we're using is test cases.
On its own, automated generation of test cases is done by many other tools - but usually, the test cases are created from recorded tester actions, or from an API with the help of an AI. We're offering something different: connecting manual test cases to automated tests and having them in the same space.
The obvious benefits
To begin with, Smart Test Cases save time. Instead of first writing test cases, then automated tests, and finally binding the former to the latter, you can just write the automated tests and have the system do the rest.
And this is not a one-time thing: you reduce the ongoing maintenance, too. Normally, the documentation for your automated tests needs to be updated every time the code of those tests changes. Having Smart Test Cases means that those updates will happen automatically.
Another important advantage is that now your test cases have stored history and versioning. Branching is a technique that can be applied to anything digital, not just code - for instance, this very article that I'm writing for you is stored at GitHub and goes through the exact same steps of deployment as the code that our developers write. With Smart Test Cases, versioning and branching are easy to apply to tests as well.
Abstracting away the code
All of these things are going to be useful no matter what you think of our general approach to testing. But the Smart Test Cases feature is deeper than that. A basic idea of TestOps is having manual and automated tests in a single space, as well as storing both tests and code in a single repository. With Smart Test Cases, you can do this seamlessly. What this achieves is a level of abstraction.
A manager who needs to review the test no longer needs access to Jenkins, they can just read the generated test case. This means less time will be spent managing access. Similarly, a manual tester no longer needs to dig into the code of the automated test to review it - after all, the code might be written in a language that the tester isn't familiar with. And even if they are, reading the code will probably take more effort. Both the manager and the manual tester can stay at their preferred level of abstraction while still having a meaningful conversation with the QA engineer.
A prerequisite of Smart Test Cases: annotations
For Smart Test Cases to work, the code of the automated test needs to have proper annotations (for more on annotations, please refer to the page of the test framework you're using in the official documentation - for instance, JUnit 5). You use them to input the basic information about your tests: @Owner, @Suite, @Feature, @Story, @Tags, etc. In addition to these default annotations, you'll be able to write custom ones:
For a use case, take a look at how annotations and TestOps in general were used at Wrike.
On the surface, adding annotations looks like extra work, especially if you've got to do it to tons of existing tests at once. But the thing is, you're going to write documentation for the tests anyway, and code is the best place to store data for the documentation because then it is going to be automatically updated and versioned. The time that you've spent on adding annotations will be saved many times over when the system is up and running. And Smart Test Cases are far from the only feature that benefits from the annotations. If all your test cases are stored this way, you'll be able to display both manual and automated tests in a single tree, which can then be filtered by owner, tag, or anything else you need.
Smart Test Cases step-by-step
Let's see how this all works up close. We start with an example Jenkins pipeline:
basic pipeline for Java tests
=== clean ===
Jenkins.instance.getItemByFullName('demo-gogi/demo-pipeline-java-junit5').builds.findAll { it.number > 1 && it.number < 100}.each { it.delete() }
===schedule ===
H/5 * * * *
We connect it to a clean Allure TestOps project with no test cases. Then we build the pipeline, so that our tests are executed and uploaded into Allure TestOps:
This is what is happening at the TestOps side in the meantime:
We've got to wait for the launch to finish, and when it's done, we close it.
Once the close action is performed, the Smart Test Cases feature kicks in. It is going to update the test cases based on the test results of the launch we've just run. In our case, the test cases are going to be created from scratch:
As you can see, the test case is filled out based on the results of running the automated tests. And on the right side, you can see the insides of the test case, filled out automatically based on code, with steps, tags, owner, and more. By the way, we're planning to introduce a new layout for the test cases, this is coming real soon.
The data for the test cases are taken from the code of the automated test which was used to create the test case:
Specifically, TestOps reads the annotations to create tags, fields, etc.; and the steps functions are used to fill in the test case scenario. Note that it says "from test result" next to the scenario; had the test scenario been created manually, it would have been able to compare the automated version and the manual one. Currently, fixtures (setup and teardown blocks) are not included when building the scenario, but in the future, we're planning to add them as well.
So this is how Smart Test Cases work. We hope they will make your project much easier to maintain, and keep both manual and automation testers happy!
