Wrike was one of the first companies to adopt Allure TestOps, back when the tool was still brand new. So, what kind of company is it and where are they at today?
Wrike is a versatile collaborative work management platform of choice for 20k+ companies in more than 140 countries. It boosts productivity by 50% and helps cut down routine by 90%. There are more than 200 engineers at Wrike, with 60+ in QA and 16 in QAA.
The intensity of the testing process at Wrike is amazing. They run approximately 130k-180k tests every day; and they do 3k+ runs of their E2E builds every month. Their Main Project in Allure has 55k stored launches over the past 6 months. Now, this is test automation taken to a whole new level!
We've talked to Alex Shurov, head of QAA at Wrike, and Ivan Varivoda, QA Automation Tech Lead there. Here's how they sum up their experience with our tool:
It might look like advertising, but honestly, Allure TestOps is a significant tool that has boosted our process to a qualitatively new level.
You can watch our conversation with them on our YouTube channel:
Or, if you'd like the text version of our talk - dig into this article!
The setup before TestOps
When Wrike adopted TestOps, they took a significant risk going for a tool that had not seen much usage at that point. So what made them take the risk and what setup were they using before TestOps? Their test management system had several iterations:
Google Spreadsheets. At first, the system consisted of just documents with test case descriptions and checklists with linked spreadsheets for autotests. In the end, they ended up with a huge number of spreadsheets, and working with them ate up too much time. Because of this, the Wrike team started looking for a TMS.
PractiTest. This one was used for 2.5 years. As the team grew, they became dissatisfied with this solution too. Test management had to get faster, and they wanted to get the results of running tests in real-time. There were also performance issues when receiving test results.
Allure Report plugin for TeamCity. Here, the advantage was getting and analyzing test results in one place, while automated test descriptions were stored in another place. However, a unified solution was needed for manual and automated tests.
Allure Testops. This is what Wrike is still using. The Allure plugin UI was already familiar, so this made the transition somewhat easier. In the end, a system was developed that utilized some very convenient features of Allure TestOps.
Some cool features of TestOps
Allure TestOps presented a systemic solution to the problems we've just discussed. At Wrike, the most important features of TestOps turned out to be having manual and automated tests in the same space, automatically updating all tests based on their code, generating reports in real time, creating custom annotations and muting flaky tests.
Real-time report generation
This feature cut down the waiting time considerably. Recently, the Wrike team did an experiment with the TeamCity plugin for Report - it was used to run all the 52k+ tests that the team is maintaining. The size of the Allure results file ended up to be 3 GB, and generating that report took about 15 minutes. This is time during which other activities are halted; but with TestOps, you don't need to wait all 15 minutes if the data you need has already been generated.
Test markup
In addition to the epic/feature/story annotations already present in Allure Report, TestOps allows you to create custom annotations:
These can then be used as epics and features. You can then mark technical attributes of tests (e. g. smoke test, a test that can be run on Safari, screenshot test). At Wrike, a specific annotation is created for each team so that it's easier to determine test ownership.
What's even better is that these custom annotations can be used as filters to select all tests with a specific annotation. This can be done in the tree view:
Here, the custom annotation for the Opportunity team is used to select tests made by that team.
Test muting
Selenium tests are not stable and are easily broken. With TestOps, it is possible to mute these flaky tests. There is a problem, however: even though a muted test is not counted in the result, it is still being run. This lengthens the duration of test runs considerably.
To solve this issue, the Wrike team used the '@TestCaseID' annotation provided by TestOps. The ID is the same for the automated test and the test case, and knowing what test cases are muted the respective tests can be excluded from being run.
Some in-house solutions based on TestOps
Allure TestOps provides a rich API that can be used to create custom solutions. This was used at Wrike for automating flaky test handling and gathering metrics.
Automating test muting
Once the system with muting / unmuting tests was up and running, the next step was to automate it. Several services were written that perform the muting automatically based on success rate. Once a test is muted a special task is created to fix the test. The custom annotations for teams are then used by the system to find whoever wrote the test, then assigns them the new task.
Suppose the test gets fixed - what then? Once per day, a special launch is performed for all muted tests, to find those that can be unmuted. Every muted test stores the id of its task. The system checks that both
- The test passes
- The task is completed If that is the case, or if the test passes 10 times in a row without any info on the task, then the test is unmuted by the system. Thus, other than dealing with the code of the test, the whole process is automated. Here's a chart of tests being muted and unmuted:
As you can see, an average day might see 96 tests newly muted and 97 unmuted (with the total number of muted tests hovering around 1.4k right now).
All in all, if you'd like to know more about how Wrike manages flaky tests, take a look at this presentation by Ivan Varivoda.
Gathering metrics at Wrike
Allure TestOps does have the Analytics section for presenting metrics, but our friends from Wrike honestly told us they're not using that, preferring an in-house solution. They use TestOps as a test data source (along with TeamCity). The data is then saved by the Allure client into internal databases, and then charts are generated in Grafana.
As an example, here is a chart that compares the number of tests that failed on first run and the number of those that were then successfully retried:
This shows how well the retry mechanism is working. Another metric being tracked at Wrike is the run time of deploy builds:
As you can see, in the 3rd and 4th quarter the company set a goal to reduce the run time; they used this metric to measure the results of their efforts.
Challenges with Allure TestOps
Wrike was a trailblazer, and because of that, it's very interesting to look at what challenges were solved there when adopting and using TestOps - there is a lot to learn from them!
- Firstly, there were process challenges and general problems with adopting a test management system into a workflow.
- Secondly, there were infrastructure challenges. When TestOps was first adopted at Wrike, nobody knew how the system would function under load, what the most effective architecture would be, what the proper settings were, etc.
One problem was the speed of queue resolution with RabbitMQ. There were huge queues, and the reporter pods took a long time to handle them:
This would impact users because it would increase the time it took to wait for report results. In the end, it was solved by increasing the count of reporter pods.
Keep in mind, however, that one of the things we at Qameta Software plan to implement in the first quarter of 2023 is immensely increasing the speed of queue resolution - so this problem might go away on its own.
Another issue Wrike faced was the huge volume of data that they had to store. It meant that great care had to be taken with resource allocation - disk size, memory, CPU, etc. For instance, the current database size is 500 GB, and another 200 GB are used for backups for the last 3 days; the S3 size is around 9 TB. All of this takes time to support and maintain, but it allows Wrike to develop, deploy and do QA quickly and effectively.
Generally, though, Alex Shurov feels that all the difficulties when adopting Allure were very limited. Wrike has a brilliant team of engineers, and nothing they faced here was in any way exceptional. And, as Alex is telling us, Allure technical support worked remarkably well. They've got high expertise, and even though the support team isn't large, they work like hell and resolve new tickets really quickly.
Plans
Long-term, there are 3 most important goals for QA at Wrike.
- Improve QAA processes. Right now, there exists a very clear vision of what can be improved in this regard. There is constant feedback and metrics are used to communicate clearly within and between the teams. Thanks to a well-thought-out process, it has become possible to predict and avoid many issues beforehand, and thus stop wasting time.
- Improve QAA technologies. Here, the plan is to simplify the test framework and test processes. This would create opportunities for other engineers in teams who don't have enough technological power to develop a testing framework. Wrike is planning to teach based on their own rich experience. Also, there is always work needed to update tools, libraries, and frameworks with fresh and testable versions.
- Foster the QAA team. Finally, an important goal is to keep building a team of engineers that is based on trust and respect for each other. Wrike is investing a lot in personal development and growing the soft skills of its people. It has been possible to achieve great synergy so that 1 + 1 is at least 3.
Wrike is currently looking to expand its team, and they need great QA Engineers. This might be an actual opportunity for you! At Wrike, you would develop internal tools and services in Java. You'd be able to work in a full-stack QA team and develop your automation skills at the Wrike QA automation school. If you enjoy working in a fast-paced environment, have web testing experience and a proactive attitude, scan the QR code and give it a go!
