Testing automation is fast becoming the baseline standard in software development. However, if you’re still working on making the switch, you may still be wondering what really matters. Here are five must ask questions that can help you decide.
What functionality is tested by now?
Kind of an obvious question. You need to know how your testing automation team is testing now, and what their future plans are. The tricky part here is that getting the answer might be not as trivial as it seems to be. In Qameta Software, we recommend reviewing test cases storage patterns: the good old "folder" way lacks flexibility in creating coverage reports by feature, story, suite, or a branch.
Ideally, you'd also want to make sure the team is using the latest technology and applying the most up to date best practices.
What is the automation / manual rate?
There is no ultimate answer for this question. The principle is that the higher the percentage of automated testing is, the more flexibility and scalability you will have. Yet most teams keep new features and exploratory testing manual. At this point, keep your attention on the pace of stable and frequently run manual tests being automated.
This exact way to expand automation provides greater testing productivity at lower costs. Be sure to ask for updates on this number regularly – and the answer should always be trending towards more automation.
Do the tests run frequently?
Again, the higher the number for testing frequency, the better.
Testing is a long, repetitive, time-consuming process, so you want tests to be run as often as possible to get as much data as possible. You’ll periodically catch flaky tests that fail from time to time depending on the instability of the tests themselves, unstable testing, time and day issues, and infrastructure issues.
To minimize these factors’ influence, run the test suites on warmed-up infrastructure. Try to keep your testing machine working, run tests on different environments and branches, at a different time and with various infrastructure.
Do you have automated fail resolution?
Test failure should be a speed bump – not a brick wall.
As soon as you start running your tests a lot, another challenge pops up: testers will spend 80% of their time sorting failed tests. Just imagine you have 100 tests, not much, and a fail rate of 5%. That means if you run your tests 10 times a day, the testers will have to sort 50 red tests a day. 5 minutes each makes 4+ hours of work.
Automated fail resolution is a must for a stable automated testing infrastructure. Find a tool like Allure Report that notes a problem, analyzes the failure, puts it in a group of similar fails, and moves on rather than getting stuck or providing dozens of failures for a single error. This also allows the testing process to finish faster.
Who actually use testing results?
Remember, it’s not you the one who needs testing reports. Do you have all the necessary information in error messages and stack traces? Well, most of your colleagues don’t. So, try to find a way to make the testing results actionable for the rest of the team.
Export automated test run results to a TMS manual testers use in your team. Sometimes it’s useful to reassure a test with manual expertise or double-check a run result. In this case, good reporting will give you more freedom.
Export bugs and defects to developers’ issue-tracker. We need our fellow developers to see and accept the testing results provided. Don’t expect any enthusiasm from the receiving side — they are overwhelmed with complex tooling, so provide insights in a tool they already use, be it JIRA, GitHub Issues, or anything else.
Be ready to create custom dashboards and reports fast. Managers usually don’t drill down in to technical details, but they may need some specific metrics slice. Fast. When a manager asks a question they expect the answer to be already on the table. This becomes possible when your tests are all tagged and marked, so you just need to build a view or a filter to show the dashboard.
Keep your testing documentation up to date and share detailed and actionable reports. Allure TestOps has all the documentation, exporting, and reporting functionality out-of-the-box, so doing each of these steps takes a couple of clicks.
A good testing team is always looking for new ways to automate, integrate and innovate.
Learn more about Allure tools
Qameta Software focuses on developing amazing tools that help software testers. Learn more about AllureFramework, lightweight automation reporting tool, and Allure TestOps, the all-in-one DevOps-ready testing platform.
Subscribe to our Monthly Newsletter (below) or follow us on Twitter, or ask for assistance on GitHub Discussions.