We've had a fascinating discussion with Alfonzo Vasquez - or, as he prefers, Fonzi. He's an experienced SDET, he has a blog, he is championing automation, and he guides startups in setting up their testing pipelines. Oh, and he is advocating for Allure TestOps. We just had to talk to him!
The conversation took us from our tool to general automation problems (both in QA and human history), AI, and who we are as testers. Ruslan Akhmetzianov talked to Fonzi Vasquez, and you can watch the recording on YouTube:
Here, we'll sum up what they've discussed, so read on!
Automation, over and over
Fonzi has been fascinated by automation and robots since he was little, and what he does right now brings this old dream to fruition: he's helping startups set up their automated testing pipelines. He worked at Code42, where he had the opportunity to observe how management handled things. Then, he moved to a small company, where he led several automation engineers in creating the infrastructure from scratch.
The changes were complex, ranging from new tools to new attitudes. Testing had to be integrated into work structure and culture; also, people had to stop thinking about QA as just a box to be crossed off. The results were impressive: the company went from releasing every two weeks to releasing ten times a day. Now, Fonzi is doing the same at yet another company, Suralink.
To achieve such a result, the team had to break up their monolithic application into small parts so that tests would be written for those parts, and KPI measurements would be taken at that level, too.
This modularization was not just a technical issue because it required trust in developers, trust that they had sliced everything properly and achieved loose coupling. For that, they had to think about quality, too. Which meant QA had to become a process, not a point.
TestOps
Of course, that process has a name: TestOps. And it starts with asking the right questions. At the previous startup Fonzi worked on, he found that capacity was pushed to the limit once they hit ten releases per day with 500-600 tests per run. Getting more throughput required more fundamental changes.
At this stage, we often see a change in perspective: instead of asking how much time it will take to run all tests, people start asking - how many of our tests can we run in half an hour? And then (when they inevitably realize they can't run them all) - what can we do to fit them all in that time frame?
Fonzi started looking into DevOps and soon found Allure Report. At first, this was a step up - at the very least, it removed the need for custom reporting.
Once the scale had grown sufficiently and a full CI/CD process was set up, experiments began with setting up Allure in a Docker container so that there would be a permanent location for results (actually, this is something many people went through, as we've told in a previous article). Once Fonzi learned about Allure TestOps, he knew it solved precisely the problems that were bugging him. In his words:
To get to that point from something homegrown - that, in my opinion, is like magic
Going from nothing to Report to TestOps and comparing it to other tools like TestRail showed some strengths of TestOps.
It is centralized, persistent, and comprehensive
TestOps provides much more than just a reporting tool; it allows you to manage the whole testing process. This has been a big selling point for Allure TestOps at Fonzi's current company (which, thanks to his efforts, is currently at the vendor assessment stage). Unlike TestRail, which makes you update the test cases manually, TestOps generates and updates test cases automatically based on the test code.
It lets the developer into the testing process
People sometimes say their developers can just look at Slack reports from CI/CD systems, so why bother with Allure TestOps? It turns out it's most useful when things go wrong.
Since you can attach anything to a test report (a screenshot, an archive, a video), you get a detailed view of the application's state at the time of failure - especially if you're also using Playwright. All information is centralized, which means that a developer usually doesn't need to bother QA to get an explanation, so less time is wasted.
And it's not just developers who appreciate this. For Fonzi, the impression has been that the adoption of Allure TestOps went very smoothly:
As for any opposition, I really haven't found it. I think as soon as you show it to a QA team, actually show the power of it, they all give the thumbs up, and they're like - yeah, let's go with this. And I think that's just what more QA teams need: they just need the exposure to it.
It is tool-agnostic
Allure Testops has a great benefit when you manage the testing process: the platform is tool-agnostic, and it doesn't allow any functionality to become vendor-locked.
In hardware, if you've got one port standard, like PCI Express, everyone abides by it. The whole thing would turn into a horror story if everyone started inventing a hundred different standards for GPU ports. We hope to bring that same approach to software.
As you can see, our talk about TestOps, the process, turned into talking about TestOps, the tool. And there is probably a reason for that:
Developers have the voice of DevOps behind them, constantly asking - hey, how do we scale this? But QA doesn't necessarily have that, and that's where I think Qameta actually cornered the market in your product called TestOps. I think that's a role that should exist. You guys called it a product, but I see it more as a role that should exist and that should ask - how do we scale testing? How do we scale reporting? How do we run 10,000 tests?
A call for automation engineers
What we've discussed so far about automation - the process, the tech, the tool - still misses the force that binds it all together: the testers and automation engineers themselves. This is important because QA is all too often thought of not as something in its own right but as a stepping stone to a more promising career. We vigorously oppose such a view.
If you look at it at a fundamental level, automation is not that different from development. You're just writing code for a somewhat different application. But all the tools and libraries you could use as a developer are still at your fingertips when doing test automation.
In the end, automation is code. Except, it's the kind of code that not many people can write today, code that is in high demand - which is illustrated, for instance, by the 2 million euro grant given by the European Research Council to the EvoMaster project, an AI-powered test generation tool.
Will testers be replaced?
AI brings us, of course, to another burning question. In the past, machines have automated the functions of the human hand. Today, AI is automating the functions of the human mind. As QA specialists, should we expect to become obsolete? Will we automate ourselves out of our jobs, as Fonzi jokes?
Probably not. Our workflow will be greatly enhanced, though, that's for sure. We're all already using ChatGPT for various tasks in our daily workflow - Fonzi usually feeds it code and asks it to comment on it.
GitHub Copilot helps here, too - sometimes you can just hit tab four or five times and have a test case ready for the code. Or you could give an API to EvoMaster and watch it generate test cases. Allure TestOps will join the ranks of those tools in the future - we plan to include AI-driven features to push test automation even further.
Naturally, you still need to write the code or the API for these things to work, figure out what you're doing with the code, and review what your AI buddy has churned out. People keep saying that you can't automate everything 100%, but saying that is missing the point - we're not trying to do it. For instance, there is no point in automating exploratory testing - it requires a deep knowledge of the system under test, test design in general, chronic corner cases, etc.
We believe automated testing doesn't make manual testing obsolete even when pushed to the limit. Despite running 4 billion tests per day, Google still employs manual testers. They do high-quality work and know their product inside out. It's the same with AI: we should ask ourselves how to make it our ally, not a competitor on the market for our jobs.
This, we believe, is the main takeaway here. And we would like to thank Fonzi for his kind words:
I think there are many good things on the horizon - with you guys building in some type of AI, with Playwright doing some of its code generation, there's just a lot of opportunity to grow within QA, and you can go in as deep as you want. It's still code at the end of the day, and it's all craft, in my opinion. And you guys have done a great job of representing that craft.
