Today, we bring you a fascinating discussion we've had with Alfonzo Vasquez - or, as he prefers it, Fonzi. He's an experienced SDET, he has a blog, he is championing automation, and he guides startups in setting up their testing pipelines. Oh, and he is advocating for Allure TestOps. We just had to talk to him! The conversation took us from our tool to general problems of automation (both in QA and human history), to AI, to who we are as testers. Ruslan Akhmetzianov talked to Fonzi Vasquez, and you can watch the recording on YouTube:
Here, we'll sum up what they've talked about, so read on!
Automation, over and over
Fonzi has been fascinated by automation and robots ever since he was little, and what he does right now brings this old dream to fruition: he's helping startups set up their own automated testing pipelines. He worked at Code42 where he had the opportunity to observe how management was handling things; then moved to a small company where he led several automation engineers in creating the infrastructure from scratch. The changes were complex, ranging from new tools to new attitudes - testing had to be integrated into the structure and the culture of work, and stop being the last thing everyone thinks of, a box to be crossed off. The results were impressive: the company went from releasing every two weeks to releasing 10 times a day. Now, Fonzi is doing the same at yet another company, Suralink.
To achieve such a result, the monolithic application being developed by the startup had to be broken up into small parts, so that tests would be written for those parts, and KPI measurements would be taken at that level too. And this was not just a technical issue, because it required trust in developers, trust that they had sliced everything properly, that they'd achieved loose coupling. So they had to think about quality too, and instead of a point, QA had to become a process.
TestOps
Of course, that process has a name: TestOps. And it starts with asking the right questions. At the previous startup Fonzi worked on, he found that capacity was pushed to the limit once they hit 10 releases per day with 500-600 tests per run. Getting more throughput required more fundamental changes. This is the point where we've often seen a change in perspective in people: instead of asking - how much time will it take to run all our tests, they start asking - how many of our tests can we run in half an hour? And then (when they inevitably realize they can't run them all) - what can we do to fit them all in that time frame?
Fonzi started looking into DevOps, and soon enough, he found Allure Report. This was a step up at first - at the very least it removed the need for custom reporting. Once the scale has grown sufficiently and a full CI/CD process was set up, experiments began with setting up Allure in a Docker container so that there would be a permanent location for results. This is a process that many went through (and one that we've touched upon in a previous article). Once Fonzi learned about Allure TestOps, he knew that it solved precisely the problems that were bugging him. In his words:
To get to that point from something homegrown - that, in my opinion, is like magic
Going from nothing to Report to TestOps and comparing it to other tools like TestRail showed some strengths of TestOps.
It is centralized, persistent, and comprehensive
TestOps provides much more than just a reporting tool, it allows you to manage the whole testing process. This has been a big selling point for Allure TestOps at Fonzi's current company (which thanks to his efforts is currently at the vendor assessment stage). Unlike TestRail, which makes you update the test cases manually, TestOps can generate those test cases, which goes a long way toward automating the process.
It lets the developer into the testing process
People sometimes say that their developers can just look at slack reports from CI/CD systems, so why bother with Allure TestOps? Turns out, it's at its most useful when things go wrong. Since you can attach anything to a test report (a screenshot, an archive, a video), you get a very detailed view of the state of the application at the time of failure - especially if you're also using Playwright. All information is centralized, and this means that a developer usually doesn't even need to bother QA to get an explanation, so less time is wasted.
And it's not just developers who appreciate this. For Fonzi, the impression has been that the adoption of Allure TestOps is very smooth:
As for any opposition, I really haven't found it. I think as soon as you show it to a QA team, actually show the power of it, they all give the thumbs up, and they're like - yeah, let's go with this. And I think that's just what more QA teams need, they just need the exposure to it.
It is tool-agnostic
The reason why Allure TestOps can manage the testing process effectively is that it's tool-agnostic, and doesn't allow any functionality to become vendor-locked. In hardware, if you've got one port standard, like PCI Express, everyone abides by it. The whole thing would turn into a horror story if everyone started inventing a hundred different standards for GPU ports. We hope to bring that same approach to software.
So our talk about TestOps the process turned into talking about TestOps the tool. And there is probably a reason for that:
Developers have the voice of DevOps behind them, which is constantly asking - hey, how do we scale this? But QA doesn't necessarily have that, and that's where I think Qameta actually cornered the market in your product called TestOps. I think that's a role that should exist. You guys called it a product, but I see it more as a role that should exist, and that should ask - how do we scale testing? How do we scale reporting? How do we run 10 000 tests?
A call for automation engineers
What we've discussed so far about automation - the process, the tech, the tool - still misses the force that binds it all together, the testers and automation engineers themselves. And this is important because QA is all too often thought of not as something in its own right, but as a stepping stone to a more promising career. We vigorously oppose such a view.
If you look at it at a fundamental level, automation is not that different from development. You're just writing code for a somewhat different application. But all the tools and libraries you could use as a developer are still at your fingertips when you're doing test automation. In the end, it's all code. Except, it's the kind of code that not many people can write today, code that is in high demand - which is illustrated, for instance, by the 2 million euro grant given by the European Research Council to the EvoMaster project, an AI-powered test generation tool.
Will testers be replaced?
This, of course, brings us to another burning question. In the past, machines have automated the functions of the human hand. Today, AI is automating the functions of the human mind. As QA specialists, should we expect to become obsolete? Will we automate ourselves out of our jobs, as Fonzi jokes?
Probably not. Our workflow will be greatly enhanced though, that's for sure. We're all already using ChatGPT for various tasks in our daily workflow - Fonzi usually feeds it code and asks it to comment on it. GitHub Copilot helps here, too - sometimes you can just hit tab four or five times and have a test case ready for the code. Or you could give an API to EvoMaster and watch it generate test cases. Allure TestOps will join the ranks of those tools in the future - we're planning to include AI-driven features to push test automation even further.
Naturally, you still need to write the code or the API for these things to work, you need to figure out what you're doing with the code, and you need to review what your AI buddy has churned out. People keep saying that you can't automate everything 100%, but saying that is missing the point - we're not trying to do it. For instance, there is no point in automating exploratory testing - it requires a deep knowledge of the system being tested, of test design in general, of chronic corner cases, etc. We believe that automated testing, even when pushed to the limit, doesn't make manual testing obsolete. Google, despite running 4 billion tests per day, still very much employs manual testers. They do high-quality work and know their product in and out. It's the same with AI: we should be asking ourselves how to make it our ally, and not a competitor on the market for our jobs.
This, we believe, is the main takeaway here. And we would like to thank Fonzi for his kind words:
I think there are many good things on the horizon - with you guys building in some type of AI, with Playwright doing some of its code generation, there's just a lot of opportunity to grow within QA, and you can go in as deep as you want. It's still code at the end of the day, and it's all craft, in my opinion. And you guys have done a great job of representing that craft.
