Testing automation is quickly becoming the baseline standard in software development. However, if you’re still working on making the switch towards it, you may still be wondering what the most important issues are. Here are five must-ask questions that can help you decide.
What functionality is tested by now?
Kind of an obvious question. You need to know what the testing process looks like for your automation team right now, and what their future plans are. The tricky part here is that getting the answer might not be as trivial as it seems to be. In Qameta Software, we recommend reviewing test case storage patterns: the good old "folder" way lacks flexibility in creating coverage reports by feature, story, suite, or branch.
Ideally, you'd also want to make sure the team is using the latest technology and applying the most up-to-date best practices.
What is the automation / manual rate?
There is no ultimate answer to this question. The principle is that the higher the percentage of automated testing is, the more flexibility and scalability you will have. Yet most teams do testing of new features and exploratory testing manually. At this point, keep your attention on the pace of stable and frequently run manual tests being automated.
This exact way to expand automation provides greater testing productivity at lower costs. Be sure to ask for updates on this number regularly – and the answer should always be trending towards more automation.
Do the tests run frequently?
Again, the higher the testing frequency, the better.
Testing is a long, repetitive, time-consuming process, so you want tests to be run as often as possible to get as much data as possible. You’ll periodically catch flaky tests that fail from time to time depending on the instability of the tests themselves, unstable testing process, time and day issues, and infrastructure issues.
To minimize these factors’ influence, run the test suites on warmed-up infrastructure. Try to keep your testing machine working and run tests on different environments and branches, at different times of day and with varying infrastructure.
Do you have automated fail resolution?
Test failure should be a speed bump – not a brick wall.
As soon as you start running your tests a lot, another challenge pops up: testers will spend 80% of their time sorting failed tests. Just imagine you have 100 tests (which is not a high number) and a fail rate of 5%. If you run your tests 10 times a day it means the testers will have to sort 50 red tests a day. With 5 minutes per test, this results in 4+ hours of work.
Automated fail resolution is a must for a stable automated testing infrastructure. Find a tool like Allure Report that notes a problem, analyzes the failure, puts it in a group of similar fails, and moves on without getting stuck or providing dozens of failures for a single error. This also allows the testing process to finish faster.
Who actually uses the test results?
Remember, you're not the one who needs the testing reports. Do you have all the necessary information in error messages and stack traces? Well, most of your colleagues don’t. So try to find a way to make the testing results actionable for the rest of the team.
- Export automated test run results to the TMS that manual testers in your team are using. Sometimes it’s useful to confirm a test with manual expertise or double-check a run result. In this case, good reporting will give you more freedom.
- Export bugs and defects to the developer’s issue tracker. We need our fellow developers to see and accept the testing results provided. Don’t expect any enthusiasm from the receiving side — they are overwhelmed with complex tooling, so provide insights into a tool they’re already using, be it JIRA, GitHub Issues, or something else.
- Be ready to create custom dashboards and reports quickly. Managers usually don’t drill down into technical details, but they may need some specific metrics slice. Fast. When a manager asks a question they expect the answer to be already on the table. This becomes possible when your tests are all tagged and marked, so you just need to build a view or a filter to show the dashboard.
Keep your testing documentation up-to-date and share detailed and actionable reports. Allure TestOps has all the documentation, exporting, and reporting functionality out-of-the-box, so doing each of these steps takes a couple of clicks.
A good testing team is always looking for new ways to automate, integrate and innovate.
Learn more about Allure tools
Qameta Software focuses on developing amazing tools that help software testers. Learn more about Allure Report, a lightweight automation reporting tool, and Allure TestOps, the all-in-one DevOps-ready testing platform.
Subscribe to our Monthly Newsletter (below) or follow us on Twitter, or ask for assistance on GitHub Discussions.
