Running automated tests and sifting through their results involves a lot of repetition. Allure Testops optimizes the process as much as possible in terms of your time and resources. One of its main advantages here is how well it integrates with different CI tools. These integrations allow you, the automation engineer, to control the testing flow from a single space. How is that achieved? Let's see how automated tests are executed in Allure Testops step-by-step.
Writing the tests, setting up the system
First, you write the tests, or maybe you hook up Testops to an existing test base. Doing this wouldn't require you to change anything in the code. Thanks to its integrations with all popular programming languages, Testops can run the code as is. The same goes for integrations with CI servers: once they are installed, you don't need to fiddle with anything to make them run.
Running the tests
Once everything is set up, you run the tests. Often, you won't want to run everything at once - that could take hours. We encourage developing incrementally: do a change to one feature or microservice, then do a localized test run. To that end, Testops allows you to run tests selectively, which is possible because it can control the execution of tests on CI servers. So how would we do a selective test run?
You could simply select the test cases you need in "Test cases":
But selecting a part of your test suite is more convenient when the test cases are sorted in a hierarchical tree. To set that view, you need to click the "Test cases" heading in the upper left corner. You'll get a dropdown menu:
There, you can choose between multiple trees to present your test cases. These trees can be pre-configured in advance - you can read about it in the documentation. Let's say you want to test a particular story of some feature - so you select "Features." Now, everything is sorted into features and stories:
Let's select the test cases from the "Create new issue" story and click the "Bulk actions" button in the upper right corner, below the Total test cases count:
Click "Run," and you'll get to a form where you can prepare your launch. There's a whole bunch of stuff you can do here, but let's look at the "Jobs" tab for now. There, you specify the job from your CI that will run the test cases you've selected:
Once that's done, click "Submit," and the launch is on!
Monitoring execution
You don't need to wait until the entire job is done to see the results of already finished tests. In "Launches," all the info on your particular launch is updated in real-time as the results come in:
This is extremely useful if, for instance, some external service is unavailable and all your tests begin failing. In this case, you'll start seeing red all over because Allure Testops delivers results immediately from the CI server. You can stop the execution, fix the problem, and re-run everything properly.
By the way, this feature, tracking results in real-time, was not easy to set up. It is possible because Allure Testops can transfer test results asynchronously. Most other TMS do synchronized HTTP requests, which puts a high resource cost on every test executed. It's not a huge problem for manual tests, but with automated tests, asynchronous data transfer is the way to go.
Re-running the tests
Once the first run is done, you've got your proper test results. You sift through them, and if you suspect that some failures were just hiccups, you'll want to re-run them. There are testing frameworks that automate this for you (like TestNG, Jest, or SpecFlow). But often, automated framework-side re-runs might not be enough. As said, a service you depend on might be down, the server could be under too much load, or there might be issues with the network. Eventually, a human will have to re-run the tests by hand, and they'll want to only run the failed tests, not the entire suite. In Testops, you can filter the test results to only select failed ones:
From there, you can click "Bulk actions" and do a selective test run, just as we've done before.
Automation as a service
As you can see, we've gone to great lengths to make the cycle of running tests as smooth and automated as possible. Of course, there are very obvious advantages for you as an automation engineer - it saves your time and allows you to control the entire process from one application: you can do the continuous testing part of your CI/CD without logging in to your CI server. But there is an indirect advantage, too.
In addition to making the execution of automated tests easy, we've also covered them with a very familiar interface: in Allure Testops, they look and feel just like manual tests. Because of that, people who might not be directly familiar with automated testing (manual testers, managers, etc.) can still work with those tests. This is automation as a service - minimal effort on the user's part.
Why should you care? Well, for one - instead of asking you to run the tests every single time, your teammates can now do it on their own, saving both their and your time. Also, because it's much easier for them to solve any problems they have on their own. This means more of your communication will be about important problems, not routine. And you get more time to write the tests.
