Good news, everyone! Today we are going to discuss the tooling tips for testing in the DevOps age. It’s often said that “T” in “DevOps” stands for “Testing”. No wonder the term “TestOps” had to appear. So what is it, exactly? Wikipedia provides a view on that:
TestOps is often considered a subset of DevOps, focusing on accelerating the practice of software testing within agile development methodologies. It includes the operations of test planning, managing test data, controlling changes to tests, organizing tests, managing test status, and gaining insights from testing activities to inform status and process improvements.
So, it’s all about process and standards, both of which require suitable tooling. Let’s take a look at how testing should work in the DevOps-ready pipelines.
What do testers usually do, and what is testing automation?
Testers ensure that the product has the best possible quality. How do they achieve that? The obvious answer is “testers write and run tests”. Though it's an obvious answer, it's not quite full.
As a starting point, run through this checklist:
- Are your tests trusted?
- Are your runs stable and persistent?
- Do your test suites run fast?
- Do people easily understand the results of your test runs?
If you are not sure every answer is “yes”, then you've come to the right place. So what is testing automation and how does it work with TestOps? Let's take a look at the main steps.
- We write tests. No tests — no testing, it’s that simple.
- We run tests. The way you deploy and run your tests matters — the more reliability you want, the more diverse runs you want to do.
- We analyze the results. Working with results from numerous runs and large test-suites may take a lot of time. Try to optimize it.
- We provide reports. If testing results stay within the QA team, you’re doing it wrong. If nobody gets the point of using your reports, you are doing it wrong.
In this post, you’ll find some recipes to boost your testing productivity with Allure Report, Allure TestOps, and some other useful tools.
Write tests
Choose convenient tools. A convenient tool is not always the right one, and here is the difference:
- A convenient tool solves specific tasks. You don’t want to use a bulky Swiss army knife for just one specific feature.
- A convenient tool is extendable. Your service is specific, with no exceptions. That means that the tool will need some adjustments with time: corner cases, exceptions, etc.
- A convenient tool is easy to explain. It lets you scale the team easily. Be sure to introduce your colleagues to new tools and approaches to free up some of your time from routine tasks. The only thing important here is what not to do. Avoid introductions like “Okay, it’s simple: update helm here, then deploy to canary and monitor logs in Elastic”, — that way you likely just leave the impression of a know-it-all person who does not help and stays inside their routine.
- A convenient tool is easy to unplug. With time, new, more convenient tools will show up on the horizon. Always remember the cost of jumping off the tool so that you're not left with a huge amount of legacy dependencies. Allure Report is a good example: it works on a set of annotations in your code and to abandon using it you need to just delete one dependency which will not break anything.
Don’t introduce complexity. If your project is new, stay sure that the amount of new stuff is under control. We have seen projects with 200 tests that include web-, API-, screenshots-, A/B- tests kept in a shiny brand-new database. And while we were looking at it, some Kotlin tests popped up. Each of the above tools is good, but being brought together all at once they become an intransparent unmaintainable mess for your colleagues and newcomers.
Always do code reviews. Senior people will be able to show the right patterns and practices and sort errors out on real code samples. Junior engineers may look through pull requests to find good code or learn from others' mistakes (for closed pull requests).
Run tests
The first piece of advice is to get used to Docker. Docker's entry threshold is not high, but you’ll get the fruits from it very quickly: the ability to run specific environments for your test suites without calling your Ops colleagues saves a lot of time and grants you an opportunity to compare different non-production tools. Imagine that you need some special Jenkins plugin or you want to compare Selenoid with Selenium Grid. With Docker, you may do it by yourself.
Run tests on Pull Requests. New code may break the project, it’s okay for new projects. The structure and architecture may change as new features appear or old ones are being deeply refactored. On the other hand, production failure is always painful. So how do we test new code to keep production alive?
- Let the developer create a new branch.
- Create a PR and run the tests on a new branch.
- Check how it works and fix the tests and errors.
- Merge the branch to the stable, the sprint branch. That means that the code gets to master only after the stable build is all-green. Run the tests on each merge, of course. Just do it.
Run the tests as frequently as possible. You’ll periodically catch flaky tests that fail from time to time depending on the instability of the tests themselves, unstable testing, time and day issues, and infrastructure issues.
To minimize these factors’ influence, run the test suites on warmed-up infrastructure. Don’t leave your testing server idle. Run tests on different environments and branches, at different times of day and with varying infrastructure.
So, with plenty of run results and a history of test runs, you’ll be able to figure out more issues that cause tests to go red.
Analyze results
Don’t be afraid to rerun tests. Tests may fail for a million various reasons, starting from issues in recently deployed code to cloud provider infrastructure outbreaks. Tracing and fixing these factors is not always possible, so if your tests are flaky, run them, run, like Forrest.
For Java code, you may choose Gradle with the official Test Retry Gradle plugin or Maven Rerun Failing Tests.
But if you don’t want to keep and maintain a zoo of different tools for automated and persisted runs, Allure TestOps does conditional and automated runs for multiple languages (JVM, Python, PHP, JavaScript) and frameworks (Cucumber, Mocha, PyTest, JUnit, etc.) on different CI’s (Jenkins, Gitlab, Bamboo).
As soon as you start running your tests a lot, another challenge pops up: testers will spend 80% of their time sorting failed tests. Just imagine you have 100 tests (which isn't much) and a fail rate of 5%. That means if you run your tests 10 times a day, the testers will have to sort 50 red tests a day. With 5 minutes for each test that makes 4+ hours of work.
The first step here is to start grouping tests by status. Allure Report has four of them: passed, failed, broken, and skipped, — and a JSON-list registry that sorts all the tests by error messages. In the end, you have a clean list of 4-7 issues that cause all these dozens of tests to fail. Allure TestOps offers a UI solution to work with Defects — fully automatic and scalable. If handling red tests is a pain for you, check it out in our previous post.
Provide reports
Remember, you're not the one who needs the testing reports. Do you have all the necessary information in error messages and stack traces? Well, most of your colleagues don’t. So try to find a way to make the testing results actionable for colleagues and managers.
- Export automated test run results to the TMS which the manual testers use in your team. Some say that automated and manual testing are two different universes, but both of them work towards one goal — providing the quality of software in production. Sometimes it’s useful to confirm a test with manual expertise or double-check a run result. In this case, good reporting will give you more freedom.
- Export bugs and defects to the developer’s issue tracker. You need your fellow developers to see and accept the testing results you provide. Don’t expect any enthusiasm from the receiving side — they are overwhelmed with complex tooling, so provide insights into a tool they're already using, be it JIRA, GitHub Issues, or something else.
- Be ready to create custom dashboards and reports fast. Managers usually don’t drill down to technical details, but they may need some specific data slice. This becomes possible when your tests are all tagged and marked, so you just need to build a view for a filter.
Keep your testing documentation up to date and share detailed and actionable reports. Allure TestOps has all the documentation, exporting, and reporting functionality out of the box, so doing each of these steps takes just a couple of clicks.
As a conclusion
All the tools and approaches might seem obvious, but if you stick with them all the time, you’ll see how the efficiency of testing efforts boosts. Let’s look through all the steps to set up a TestOps’ish pattern in your testing:
- Write tests. Choose new tools wisely. Run from complexity and remember that your tooling maintenance overhead should be lower than the time it takes to do your main job.
- Run tests. In the morning and afternoon, before PR and on night releases. The more frequently — the better. Build and automate the infrastructure for persistent launches in various environments.
- Use the test results. Many launches mean many failed tests. Automating the sorting of red and flaky tests is vital for insight generation.
- Share testing results. Make all the results and insights you get from testing available and actionable for the team.
If you now look at the Wikipedia definition of TestOps at the beginning of this article, you'll understand it much better, as you already know what to do. And Allure TestOps is ready to help you on this long journey to software quality.
Learn more about Allure tools
Learn more about Allure Framework, our open-source testing reporting tool, or Allure TestOps, the all-in-one quality management platform.
Subscribe to our Twitter feed or Gitter community. It is a wholesome place to get help and stay up to date with news.
