“I have been working with ALM (HP) and other tools (like SAP) to administer test results and I can say from experience that Allure TestOps is the best from a usability perspective and also in terms of integration with CI pipelines (all others seem to be left untouched since 1990).”
Balazs GalambosSDET at Miro
We've recently talked to some folks from Miro, who set an example of how to use Allure TestOps to its full potential. They are Balazs Galambos, an SDET at Miro, and Zhaopeng (Robin) Xuan, an engineering manager:
Miro is an online multi-user whiteboard. The company champions a vision of an agile, hybrid (remote and in-office) workspace that allows people to collaborate for maximum productivity.
Miro's approach to testing is an unusual one: they want their developers to write not only the unit tests but also the higher-level ones. All their tests reside in the same space and use the same tooling. This setup has been achieved with Allure TestOps, and with it, they are able to do a huge amount of tests per day. Let's look at their story in more detail.
In the initial setup at Miro, there were separate repos for production code (written in TypeScript) and test code (in Java). Test results would simply be stored in folders, and in the end, an Allure Report would be generated based on that XML. The entire system resided on a single machine, an Amazon server that ran 20-30 UI test cases in parallel in Java. It was the most powerful machine available, and it still wasn't enough.
There was no management tool for tests and no integration with any CI/CD tool. The system wasn't very controllable. For instance, rerunning failed tests was a fairly complicated process. The data necessary for that wasn't contained in the Allure Report, and one had to extract it from another place. Also, there was no real process to detect flakiness or trends in tests. The instruments weren't integrated at all. All in all, the solution lacked scalability and required lots of manual work.
The new system that Miro was looking for had to centralize all test results in one place and integrate them with CI/CD and JIRA completely. It didn't take long to choose Allure TestOps as the platform for this integration: Miro had already been using Allure Report, and their team was really impressed by the feature set of Allure TestOps.
At first, Allure TestOps was introduced alongside the old system. Both were still residing on a single machine, and it kept running out of disk space, so test cases older than a week ago had to be constantly deleted. This wasn't sustainable, so Miro decided to get rid of the old approach completely, and also make the system scalable so that it wasn't tied to just one machine. To that end, they first started writing tests in TypeScript instead of Java. Since the language of the tests was now the same as the language of the code, both could be stored in the same repo. This meant that developers had easy access to the tooling necessary to write end-to-end tests.
The process wasn't entirely smooth. Understanding certain design structures proved to be a challenge, but direct contact with support did help out. In certain cases, workarounds were required, like with the TypeScript integration. Although Allure TestOps has more than 50 integrations, there existed none with TypeScript. So Miro created one on their own, using a tool called Spectator. All in all, the challenges proved surmountable, in no small part thanks to direct help from Allure.
The end result was a system where Allure TestOps collects all existing tests, and a plugin runs them every time a new pull request is created. A merge can't happen until all those tests are green. In case they aren't, there exists an automated communication channel with the owners of the tests. In other words, Miro has created an automated quality gate that allows processing a truly stupendous amount of tests every day.
There are several long-term goals at Miro right now. One is to achieve tool stability and provide observability about what works and what doesn't. Another goal is tool scalability. The company underwent huge growth in the past years, so now they encourage all their developers to make new tools scalable, and to refactor existing ones as well.
Finally, a concern closer to the present is migrating from Allure TestOps 3 to 4, maybe even to 4.7. It will have to be done very carefully because the system already has thousands of tests, and because version 4 has some major changes compared to version 3. Still, Allure does have tools for this migration, and the benefits are well worth it.
