With 3.5 million users, Allure Report is a rather popular tool. You might be using it and asking yourself - that same team made something called TestOps, what is it? Should I use it instead of Report? Well, let's try to look at them both and compare!
In a nutshell, TestOps takes the basic idea of Report to a new level. Report is making test results accessible to people who can't necessarily read the underlying code. TestOps builds upon Report and adds functionality that makes it useful for an enterprise, not just a single user. Let's take a closer look at the differences between the two tools.
Disposable vs Persistent
An Allure Report is disposable, with very limited capability for comparing results of multiple test runs or storing them long-term. Allure TestOps, on the other hand, provides centralized data storage for test results. This has several very important advantages.
Easy access to results
With Report, if you want to share the report with someone, you're going to have to give them the test result files and ask them to generate the report on their own. If you're running Report on a CI server, then you'll have the headache of managing access to the pipeline. With TestOps, sharing test results is as easy as dropping a link to them.
This also means that you don't have to buy the full version of TestOps for every person that needs to view the results, only for the engineers that actually work with the tests. The rest of the team can view all the data with the trial version of TestOps which is available for free. This is in contrast with e. g. Jira, in which you've got to pay for every single user in the system.
Historical data
Centralized storage in TestOps means access to historical data. Every test run generates data based on the annotations in the tests, and you can present data from multiple test runs in a dashboard with customizable widgets:
As you can see, the widgets come in all shapes and forms (pie charts, trends etc.). They can show cross-sections of data based on specific queries in the RQL language. For more on dashboards, see this video or read the official documentation.
Test case history and LiveDoc
In TestOps, historical data is also available for each individual test case. Each test run automatically updates all executed test cases and generates the test case documentation - which means that documentation is always up-to-date (this is called LiveDoc). The results of previous test runs for each test case are available in the Test Cases tab, with detailed reports for failures:
Allure Report does show the most up-to-date version of your automated tests, but getting it to store and display even a little historical data for test cases isn't easy.
Defects
Thanks to centralized storage, Allure TestOps also provides the defect feature. When a single issue causes multiple failures over several test runs, figuring out the problem over and over from scratch can be a headache. Instead, these failures can all be linked to a single defect. The system allows you to search for similar failures with a regex, thus saving you lots of time. The defects have a separate tab dedicated to them:
If you're interested, we've written an article on defects.
Centralized storage for Allure Report
In theory, it's possible to provide some of these features for Allure Report as well. However, it takes much more effort: we've talked to a team that has successfully implemented centralized data storage, and it has taken them several thousand hours of work. There are other great projects in the Allure community, such as the Allure Docker service or Allure Server. Take a look at the code base that is necessary for providing persistent storage and aggregating several reports, and drop the authors a star, they’ve done some impressive work!
Manual Tests
Allure Report is only designed to work with automated tests, whereas in Allure TestOps, automated and manual tests have the same structure and are treated the same way. Naturally, manual tests will require an operator to click through the steps (this is done inside TestOps). But all the functionality available for automated tests is also available for manual tests. Both are run within a single launch, and their results are stored together.
Manual tests can be created in Allure TestOps directly:
They can then be exported to an IDE to be automated. Once the automated test is created, it can be compared to the manual one, so that the manual tester can ensure the fidelity of the automated test. This is extremely important if you want to build trust in automation. For a quick introduction to manual QA with Allure TestOps, you can watch this video.
CI/CD Integration
Report integrates with several CI/CDs like a charm. But its integration is rather limited: it just allows some of Report's functionality to be launched from the CI side. TestOps can be triggered this way too. But in addition, it is capable of controlling the CI flow from its own side, so the integration is two-way.
More specifically, TestOps can manage runs and jobs. This means that:
-
You can track the progress of CI jobs (Jenkins, GitLab, etc.) in real time from within Allure TestOps.
Doing this with large test runs is possible because TestOps is capable of transferring test results asynchronously. Virtually no other TMS does that - they are simply not aimed at automated testing and so they are not designed to transfer results from tens of thousands of test cases in a single test run. Implementing asynchronous batch transfer has been one of the most difficult hurdles for our team, and we're proud that we've overcome it. -
Each launch connected to a Jenkins job displays its tags, as well as additional information about the environment where it is run.
-
Most importantly, you can do selective reruns of tests if you suspect that some failures were just hiccups. Specifically, you can initiate a Jenkins launch with specific tests. Without TestOps, doing this is a huge hassle - you are probably going to have to dig into your framework and write some code to launch the tests that you want. With TestOps, all of this is done with just a few clicks:
Migrating
If you're thinking about migrating from Report to TestOps, the good news is that it can be done with very little effort. Allure TestOps uses the same annotations as Allure Report, so you won't have to change anything in your code. All the functionality provided by Report is present in TestOps, too. Of course, if you had e. g. a Jenkins plugin for Report, you will need to install a similar plugin for TestOps. But that's it!
Summary
Of the two tools, Allure Report is the simpler one, and it's mainly intended to be run on a single machine. Its job is to visualize tests, and it does that job really well. Allure TestOps builds upon the functionality of Report, but it is more comprehensive and aimed at the enterprise level. TestOps allows several teams to manage and automate testing, share data and integrate their workflow, eliminating as many manual steps as possible.
