Wrike shows us what truly automated testing looks like

When Wrike adopted TestOps, they took a significant risk going for a tool that had not seen much usage at that point. So what made them take the risk and what setup were they using before TestOps? Their test management system had several iterations:

1. Google Spreadsheets. At first, the system consisted of just documents with test case descriptions and checklists with linked spreadsheets for autotests. In the end, they ended up with a huge number of spreadsheets, and working with them ate up too much time. Because of this, the Wrike team started looking for a TMS.
2. PractiTest. This one was used for 2.5 years. As the team grew, they became dissatisfied with this solution too. Test management had to get faster, and they wanted to get the results of running tests in real-time. There were also performance issues when receiving test results.
3. Allure Report plugin for TeamCity. Here, the advantage was getting and analyzing test results in one place, while automated test descriptions were stored in another place. However, a unified solution was needed for manual and automated tests.

In the end, Wrike adopted Allure TestOps as their test reporting solution. The Allure plugin UI was already familiar, so this made the transition somewhat easier.

Some cool features of TestOps

At Wrike, the most important features of TestOps turned out to be having manual and automated tests in the same space, automatically updating all tests based on their code, generating reports in real time, creating custom annotations and muting flaky tests.

Real-time report generation

This feature cut down the waiting time considerably. Recently, the Wrike team did an experiment with the TeamCity plugin for Report - it was used to run all the 52k+ tests that the team is maintaining. The size of the Allure results file ended up to be 3 GB, and generating that report took about 15 minutes. This is time during which other activities are halted; but with TestOps, you don't need to wait all 15 minutes if the data you need has already been generated.

Test markup

In addition to the epic/feature/story annotations already present in Allure Report, TestOps allows you to create custom annotations:

These can then be used as epics and features. You can then mark technical attributes of tests (e. g. smoke test, a test that can be run on Safari, screenshot test). At Wrike, a specific annotation is created for each team so that it's easier to determine test ownership.

What's even better is that these custom annotations can be used as filters to select all tests with a specific annotation. This can be done in the tree view:

Here, the custom annotation for the Opportunity team is used to select tests made by that team.

Test muting

Selenium tests are not stable and are easily broken. With TestOps, it is possible to mute these flaky tests. There is a problem, however: even though a muted test is not counted in the result, it is still being run. This lengthens the duration of test runs considerably.

To solve this issue, the Wrike team used the '@TestCaseID' annotation provided by TestOps. The ID is the same for the automated test and the test case, and knowing what test cases are muted the respective tests can be excluded from being run.

Some in-house solutions based on TestOps

Allure TestOps provides a rich API for creating custom solutions. This was used at Wrike for automating flaky test handling and gathering metrics.

Automating test muting

Once the system with muting / unmuting tests was up and running, the next step was to automate it. Several services were written that perform the muting automatically based on success rate. Once a test is muted a special task is created to fix the test. The custom annotations for teams are then used by the system to find whoever wrote the test, then assigns them the new task.

Suppose the test gets fixed - what then? Once per day, a special launch is performed for all muted tests, to find those that can be unmuted. Every muted test stores the id of its task. The system checks that both

1. The test passes
2. The task is completed

If that is the case, or if the test passes 10 times in a row without any info on the task, then the test is unmuted by the system. Thus, other than dealing with the code of the test, the whole process is automated. Here's a chart of tests being muted and unmuted:

As you can see, an average day might see 96 tests newly muted and 97 unmuted (with the total number of muted tests hovering around 1.4k right now). If you'd like to know more about how Wrike manages flaky tests, take a look at this presentation by Ivan Varivoda.

Gathering metrics at Wrike

Allure TestOps does have the Analytics section for presenting metrics, but our friends from Wrike honestly told us they're not using that, preferring an in-house solution. They use TestOps as a test data source (along with TeamCity). The data is then saved by the Allure client into internal databases, and then charts are generated in Grafana. As an example, here is a chart that compares the number of tests that failed on first run and the number of those that were then successfully retried:

This shows how well the retry mechanism is working. Another metric being tracked at Wrike is the run time of deploy builds:

As you can see, in the 3rd and 4th quarter the company set a goal to reduce the run time; they used this metric to measure the results of their efforts.

1. The setup allows running extreme volume of tests every day
2. Manual and automated tests are being run and analyzed all in one place
3. Real-time access to report data cuts down waiting time
4. Muting flaky tests removes noise from test results and speeds up test runs
5. TestOps' rich API allows building complex in-house solutions

Long-term, there are 3 most important goals for QA at Wrike.

1. Improve QAA processes. Right now, there exists a very clear vision of what can be improved in this regard. There is constant feedback and metrics are used to communicate clearly within and between the teams. Thanks to a well-thought-out process, it has become possible to predict and avoid many issues beforehand, and thus stop wasting time.
2. Improve QAA technologies. Here, the plan is to simplify the test framework and test processes. This would create opportunities for other engineers in teams who don't have enough technological power to develop a testing framework. Wrike is planning to teach them based on their own rich experience. Also, there is always work needed to update tools, libraries, and frameworks with fresh and testable versions.
3. Foster the QAA team. Finally, an important goal is to keep building a team of engineers that is based on trust and respect for each other. Wrike is investing a lot in personal development and growing the soft skills of its people. It has been possible to achieve great synergy so that 1 + 1 is at least 3.

Wrike is currently looking to expand its team, and they need great QA Engineers. This might be an actual opportunity for you! At Wrike, you would develop internal tools and services in Java. You'd be able to work in a full-stack QA team and develop your automation skills at the Wrike QA automation school. If you enjoy working in a fast-paced environment, have web testing experience and a proactive attitude, follow the link and give it a go!