Testing 2.0

Testing is diverse. Despite the fact that the main goals are the same everywhere, there are many differences from project to project: different stacks, technologies, processes, and approaches. Analyzing the processes in many teams, Qameta Software has found one problem. That problem is inefficiency. Currently, testing is one of the most under-optimized processes in the development life cycle.

Today, we will try to understand how it happened and what we can do about it.

Where did it all begin?

Usually, at the beginning of the project, there is no tester, and the developer and the tentative manager are responsible for the quality of the product and the bugs found. Everything is fine in this world because if the manager finds a bug, he sends it to the developer for correction; the developer then fixes the bug, and everyone is happy. However, moving forward, the number of developers increases. This means that the amount of code increases. More code means more bugs! At this moment, the manager begins to lack strength, but he tries to cope, since who else should test the system if not the customer? He knows how everything should work, checks that everything is done as it should be, and at the same time he also brings in bugs.

Excellent! The problem arises when the manager continues working like that and eventually misses a critical error. At this moment, members of the project decide that the manager (as the bearer of sacred knowledge about the product) should test new functionality and not waste time on boring regression tests. So we got our first manual tester. Why manual? Because development doesn’t have any expertise in creating the correct automation of regression yet, and the manager is much more at ease dealing with a person rather than with code. Now we have a team, and there are no difficulties: the manager tests new functionality and features, and the tester drives the regression. Simple and clear. But the moment a critical bug sneaks into production, a muddle starts. The team will go to the tester and ask:

— Why do we have a bug in production, how did it happen? How did you miss it? To which our specialist will honestly answer — But it wasn’t there. I definitely checked this place! Most likely, the developers meddled with the code after my testing! Obviously, having analyzed the situation in retrospect, the manager decides to formalize testing and introduce checklists in testing. Whether the checklists are simple or complex, based on plain specialized software or Excel, they will clearly show what is being tested, at what time, and what was the result. Using this tool, the team will be able to tell with certainty if the tester missed a bug or if careless developers rolled out a feature without testing after the smoke run. And that’s how we live.

The product keeps growing and acquiring features; the number of checklists keeps growing; our tester is gaining experience and authority, but he is forced to spend more and more time testing.

It’s time to take on another tester to back him up! And now a team of testers appears in our development cycle. Now there are two people testing the regression.

The first Test Management System (or TMS for short) appears in the team, because now, in addition to the standard fields “step, status, time”, we need to document a bunch of data from the tester’s head: links, logins, passwords, some additional steps or how to bypass some old little familiar problems. So we have test cases that allow us to:

• easily recruit new people
• not worry too much about onboarding and transferring product knowledge
• distribute tests among team members
• monitor the performance and quality of work of testers

However, there are also disadvantages: unlike checklists, test cases require constant updating and proper organization. And if updating is just a matter of scrupulousness and process, then the organization of test cases is a complex issue. The problem arises at the moment when the test server or staging appears in the process of natural growth of the testing infrastructure. Up to this point, the test documentation includes all the details necessary for the run: links, logins and passwords, caches and cookies, etc. But with the introduction of the pre-production test environment, all these details will have to be locked and parameterized, because they won’t match in production and on the testing server. And then the testing team will have to invest twice as much effort in supporting all this stuff.

The product keeps growing and getting more features; the number of test cases keeps growing; our tester becomes a lead.

The team is also growing, more testers are joining in. However, productivity does not increase linearly, but tends to be a logarithmic function: each new engineer gives a smaller increase in team productivity. There comes a moment when the deadlines begin to grow like an avalanche, and the manager realizes that the team needs…

Automation!

As the first automation engineer joins the team, they will likely use frameworks and approaches in which the team has no expertise. This means that the team is forced to rely entirely on the experience and knowledge of a new employee, and if there was a mistake at the hiring stage, it would be difficult to detect, let alone to correct.

A man’s meat is a robot’s poison! An automated test is not a human. Automated tests are dumb and they don’t know how to think through steps and step out from scenarios. Therefore, the classic approach to “translating” test cases into a framework language is an anti-pattern that will lead to false fails or missed errors.

Imagine that in 100 tests there is a piece where you need to log in, generate a doc and download it in PDF. If anything has slightly changed, a human will move on, noting that it would be good to update the documentation. And if it is a robot, you will get 100 red tests.

What do testers do when they see 100 red tests? That’s right, they think that the tests are broken and wrong!

Nobody wants to sort out a hundred identical fails, so let’s make an automation engineer deal with them. At this point, the team comes to an evolutionary dead end: there is an automation engineer who writes automated tests himself and then sorts out the reports himself. He becomes a thing in itself, detached from the team.

Documentation won’t write itself

To somehow control testing automation, the QA lead decides to combine automated and manual tests into a single process. And here are three options on how to do so:

• Integrate testing automation with TMS. Almost all test management systems have an API so that teams can push test results into the systems to get aggregated reports. The problem here is simple: automation results are usually collected without metadata and details, and all that the testing team can learn from the report is that N automated tests failed. Why did they fail? You have to dig into the guts of the test run and figure it out by yourself.
• Separated automated test run followed by manual regression. This option, despite being an “old school” solution, is more efficient in terms of detecting bugs. The difficulty here is that the testing team will have to work in two interfaces: dealing with automation reports in one tool while doing manual runs in another. Let’s say testers create a run and run it from their interface; after that, the automators run their tests and give an Allure Report to manual testers. If there are errors in reports, the team has to work with Allure, and then they will also try to transfer the detected errors to manual TMS. And then to JIRA. That’s a lot of effort.
• Automation becomes a completely independent service. Once a test is automated, it is removed from the manual testing registry. Why link manual testing with automation if you can single out automators into a separate department and get a “green pipeline certificate” from them? This is perhaps the most dangerous path since it makes one unit completely dependent on another, and such connections often end with conflicts and confusion in retrospect if something goes wrong at the crossings.

Despite all the difficulties, the automation engineer starts working more efficiently after a while: running automated tests does not require the time and attention of an engineer. It means that the more tests are written and run, the more time we save on runs compared to manual processes. When there are ten automated tests, the effect is insignificant; when there are a hundred, it is noticeable. And every day, engineers will expand coverage and make automated tests more and more convenient for work: more atomic, more stable, and more informative. Isn’t that what we expected? At this moment, one may think that automation has come, and it will only get better. As if!

Growth problems

Once automated testing starts to grow, a bunch of new problems pops up, and the problems will be much closer to development than to existing testing. And if your developers and testers have not established close cooperation and sharing of experience, the solution to each of these complications will take time and effort. So what are we in for?

Test server stability

A test environment will definitely appear. It will end up in the operation project support team (Ops) pipeline and testing will likely not have managing access to it.

Typically, a test server isn't a particularly productive machine, and when there are more tests or runs, it starts choking. This often leads to delays in shipping the results of the run and test flakiness (in this case, it means tests that fail due to broken parallelization or timeouts), adding fuel to the fire of already weak faith in automation from both manual testing and development/operations.

Automation drops out of processes

As a result of such progress, even the most competent automator closes in on his own tasks:

• Manual testing actively communicates with the team, talking about test results and bugs. Due to the difficulties mentioned above, automated testing remains an internal artifact in the QA team, which does not allow the automation engineer to “get a sniff of fresh air” from the production team.
• It is usually hard to add anything to the development team's automated tests with metadata (IDs, for example) due to the fact that the automation engineers are considered to be less qualified in terms of coding than the average developer, so they get a firm dismissal.
• A gap in competencies also stands in the way of going to the ops team; as a result, the admins dismiss the offered help saying “we don’t have time to teach you, we’ll set up everything up ourselves, and you use it at home.” Or “do you want a Selenoid cluster? Slow down, we don’t have Docker in our pipelines!”

As a result, we get a situation where engineers have to break a lot of walls and silos to “legalize” automation, and without the support of a good manager, such things fail. In the end, automators give up, and they just write their tests in their test environment and show beautiful useless reports. If things go this badly, automated testing gives us more questions than answers:

1. We don’t know how many automated tests are written and running.
2. It is not clear which functionality is covered by automated testing and which is not.
3. There is no understanding of where the tests are relevant and where they are not.
4. There is no clarity on how these tests can be run.
5. It is necessary to find out where and how to store test results. In the end, we have manual testing, which deals with regression and does not scale. And the automation team, designed to solve scaling problems, is disempowered and left without the support of colleagues, so it just breeds infrastructure and process complexities. Testing becomes ineffective.

What is to be done? Testing 2.0

If you have a large project and set the goal of figuring out how much it costs you to manually test each release, you will realize that automation is indispensable. That is why any large IT business builds testing around automation.

Some say “testing is not necessary — the development itself will write the tests, and the canary deployment will check everything else”, but this is not entirely true. Here’s the difference between developers and testers:

• Developers write optimistic tests 90% of the time. Tests that check the happy path: when you enter the right data, you get the correct result; the database accurately stores and updates data; data structures and algorithms are executed correctly.
• The testers, on the contrary, test with a destructive paradigm. Their task is to find vulnerabilities in both basic and unexpected scenarios. They look for them despite the fact that everything usually works fine.

Let’s rewind time a bit to understand when we took the wrong turn. This is the moment when an automation engineer joined the team… No, the decision itself was correct! However, let’s try again to think about where he should have been put so we won’t end up in a situation when automation is off the grid. The following text may seem to you either a story by Captain Obvious, or utter nonsense. In any case, reach out to us on Twitter! One article can't cover everything, so how about we have a discussion there?

Automation in charge!

Let’s flip the paradigm! Today the trust in development teams in almost any business is phenomenal (glory to DevOps!):

• Development has competencies for evaluating the code and the way it is tested.
• Everyone trusts development immensely.
• Development has a close relationship with the management and operations teams.

If you carefully look at this list, you can point out that each item allows you to fix the corresponding source of test automation problems. Let’s imagine that the moment a QA automator arrives he becomes not just a cog in the manual testing department, but leads the movement of the entire team towards automation. This will slightly change the entire vector of his responsibility and tasks from the very beginning of the journey. There are at least three big stories on our radars when a “change of power” takes place in a revolutionary way because the team is so great and because a strong-willed decision on the primacy of automation is made at the managerial level.

The second option is suitable for teams where there is no super-qualified team of testers, and the developers themselves take care of testing. In this case, the decision to put the automator in the Dev team is a no-brainer. Such developers have a culture of experience sharing and a high testing culture. Because of that our engineers have something to learn and someone to show the code for review.

• In this scenario, you don’t have to worry about competence. These developers will be happy to help the automation engineer figure out a bunch of new technologies that we didn’t get our hands on in the previous scenario: Docker, git, CI/CD pipelines.
• Developers don’t like to write automated tests. For them, writing tests for a feature is a chore that is needed only for the pull request to pass the code review. But it is a great task for the automation engineer. He can both deal with the feature, and write more interesting tests. Just imagine how great it would be to free up to 20-25 % of a developer’s time to develop new functionality.
• Most often, the development team also communicates better with the Ops team. If all of a sudden the tests start to fail like crazy, the admins will help you figure it out or allocate more CPU/RAM. Why? Because if testing in development slows down releases, this is a problem for everyone in the pipeline.

Of course, in the long run, this will entail major organizational changes. We put automation at the forefront of testing, rather than making it a manual testing tool. On the contrary, manual testing becomes a tool for automated testing!

Have you ever seen anything like that? But think, do we really need manual testers to run regression manually? They should create new scenarios and pass them to automation by default, being responsible for exploratory testing and expertise in fundamental testing to ensure maximum quality and coverage. FTW!

In the next article, we will try to define the specific stages of testing evolution in a team. We will try to think of problems for each stage and offer solutions and tools that will help overcome these problems.