Quality Gates for Testers

Nowadays, there is a lot of talk about quality gates — milestones between the stages of the project delivery cycle. Quality gates are essential for creating top-quality software. But they are often thought of as meetings where people just sit together and manually go through checklists.

In reality, a quality gate is any check that can stop product delivery if not passed. What checks can do that?

There's all kinds of quality gates out there

• An IDE warning/error. Even something that simple can prevent you from running your code — which means it's a quality gate.
• Style guides. Those are often recommendations rather than delivery-breaking rules - Python's style guide warns us that sometimes style guide recommendations just aren’t applicable. And yet, it is possible (and recommended) to get a PEP 8 linter for your code editor.
• Code reviews. Having an experienced pair of eyes (or two) go over your code is always good - at the very least, it ensures that knowledge about code will be spread around the team, making it more robust.
• Pull requests. Whoever does the merging will review the changes, another check that can stop the process.
• Building. In a way, this is another quality gate - if a build fails, the whole process stops.
• Unit tests.
• Integration tests.
• Staging and manual tests.

Many of these checks suffer from an inherent problem: humans are too smart. The dev, the tester, the manager - everyone has their head in a different place regarding the system and what they do for it. When it comes to QA, everyone does it their own way (if at all). When the developer writes an awesome solution for a problem, writing unit tests is a chore; usually, they just confirm that the awesome solution works, testing the happy path. They might ignore yellow warnings from an IDE because, quite frankly, they've got their hands full with the red ones. Testers are the bedrock of QA, but they do make mistakes. For instance, they might mute a flaky test and then forget about it, not realizing that the test has become permanently red.

Automation has an important advantage over humans: it is dumb. Which means it's reliable and transparent.

Automation will never replace humans completely, and manual testers aren't going anywhere. But automated tests provide checks that don't depend on our laziness, lack of attention span, or our brains being busy with more burning issues. As quality gates, automated tests with pipelines are among the most efficient ones.

The Advantages of Using Automated Tests and Pipelines

By placing automated tests into build and deployment pipelines, you get automated quality gates that provide fast and continuous feedback. You can specify whether a deployment fails when it doesn't pass through all the quality gates or just one of them, depending on how strict you want to be with your pipelines.

Due to the automated nature of the tests, once you have a successful pipeline run, you’ll have the confidence that some of your headaches have been taken care of without manually tracking your progress. This is more efficient than manual quality gates and saves lots of time. You can add numerous gates into many pipelines until you’ve automated almost 100% of your testing life cycle.

Difficulties with automated quality gates

Automating pays off in the end, but it takes work. Don't get stuck at automating the individual pieces of your workflow; returns on your efforts increase once you integrate those pieces. Running your automated test suite on each pull request is one such example.

Consider a simple workflow in GitHub that runs a test suite on each pull request and sends results to Allure Testops. Writing tests is a lot of work, and on their own, only the automators can use them. With GitHub, the system becomes more controllable, it becomes possible to automate the checks. Finally, with Allure Testops, the results of those checks become much more readable, now everyone in the team can access them. Each new step makes the system more usable and more accessible; more and more people become involved, and return on invested effort increases.

Which brings us to another difficulty. If you want proper automation, you'll have to manage a zoo of programming languages, frameworks, issue trackers, CI/CD systems, etc.

We've calculated that just integrating testing will take two people per month - and we're talking about a rather specialized skillset, devs with a deep knowledge of testing and integrating different tools. It's not easy to find such people - though not impossible. Often, companies prefer to train these specialists. Either way, costs are high.

And there is another problem: automated checks are dumb (as we've already established). It means they're easier to conduct - but it also means that if you've moved a button 10 pixels to the left, your screenshot test for that button will fail. You might not want it to, but this is exactly what it is supposed to do.

Of course, automation doesn't happen overnight. It's a process that takes time and effort, and if you want to know more about it, you can read our guide on integrating automated testing into DevOps. But despite the effort, automation is worth the expenditure. Humans hurry, get tired, and make mistakes. Code doesn't.

The Best Tools for Quality Gates

1. Google Sheets and Microsoft Excel: For simple quality gate checklists, Google Sheets and Microsoft Excel are excellent.
2. Trello: For automated updates between teams, use Trello to integrate messaging tools such as Slack.
3. Atlassian JIRA: With Atlassian JIRA, you can put your quality gates into a dynamic dashboard to provide broader visibility for your team.
4. Atlassian Confluence: You can create a shared space with checklists and documented quality gates for your team with Atlassian Confluence.
5. GitHub: GitHub allows you to run whichever checks you wish on actions such as pull request.

In Summary

Having automated tests as quality gates doesn't mean you need to get rid of all your manual tests. But you can trigger the manual test runs automatically and rely on the automated tests to give the go-ahead if everything is in order. Automated tests don't require gathering meetings and going through checklists by hand. In other words, they are automated quality gates.