Software testing: do we really need more tests?

What is software testing?

I have to admit that at the beginning, searching for the title of the article, strong words came to mind – mostly because the issue of limiting the number of tests gives rise to lots of negative emotions. After analyzing various comments on industry websites, I came to the conclusion that this happens for many reasons. But let me start with what software testing is. Let’s refer to the ISTQB definition:

“A process consisting of all life cycle activities, both static and dynamic, focused on the planning, preparation and evaluation of software and related products to determine whether they meet specified requirements, to demonstrate that they are fit for purpose, and to detect faults.”

The process itself indicates that the natural form of developing tests in a given project is to increase the number of test cases, as well as other test-related activities.

Tests – do figures count?

Let me start with the figures. Having worked for several years on numerous projects, including foreign ones, where there were 15 test programmers (not to mention a whole army of manual testers, programmers, etc.), I have noticed one trend: figures count. The test manager becomes a person who “protects” the tests from the decision-makers. Most often, management pays attention to costs, presenting the approach known as: „Why do we need tests? After all, programmers should write flawless code”.

Such situations still happen, and it is not just a tale told on the Internet. This is an approach that I have encountered many times over the last decade of testing. And which – I am not being sarcastic here – doesn’t surprise me at all. What is a test manager today? It is a person with a certain amount of technical knowledge, but who makes use of most of his skills in other ways.

Let me give you an example that I believe many of us have experienced: the classical conflict of interest between the student and the teacher. The student distributes their efforts and manages their time in such a way as to switch between many different subjects and effectively achieve their goal (to pass the year and move onto the next). For the teacher, in turn, their subject is the most important (I know what I am talking about – I have been both a student and a teacher in my time).

The manager manages projects the way a student manages an exam session, so tests are just one of many elements. And they are low in the hierarchy because the most important thing is to deliver the product. To prove that hiring a certain number of testers makes sense, the test manager must provide the project manager with the data: the number of test cases, automation coverage, as well as the number of errors found, time saved thanks to automation, risks, potential error costs, etc.

Why do we need test automation?

Today, test automation is an important complement to manual testing. Automated tests allow you to replace a large number of manual tests and relieve software testers, thus saving time. From the test manager’s perspective, when it comes to automatic tests the most important thing is often what manual tests cover, and especially how many hours of manual testers’ work are replaced by automated tests.

Everything seems fine. The number of test cases is growing, automation continues to expand. There are about 4000-5000 test cases, and automation covers 1600 of them. Other people join the project. The statistics are fine; management is satisfied.

Test maintenance

But at this point, a new problem arises – the growing number of tests that need maintenance. The time pressure and increasing complexity of the project mean that everyone works more every day. One-day releases turn into weekly tests and the need to check why automatic tests won’t work arises. When correcting them, it turns out they are outdated, and the manual test case has been outdated for 3 months. The test used to pass but suddenly it’s stopped working. Nobody knows how the function should work, because there are thousands of tasks in Jira, and searching for answers would take 2 days. Of course, you can always get in touch with someone who has the requisite knowledge, but… it has been a while since this individual left the company (in IT projects, employee rotation is standard).

Blame testers and tools?

In such a situation, you may think that it is the fault of unsuitable tools, or testers who do not do their job diligently, as well as test managers. For sure, it is (to some extent), but most importantly, it is all a matter of scale.

There are two things that are not limited by resources: assuming the instant growth of the economy, and an increase in the number of tests. Managers, test managers, testers and automation testers all share the ambition to perform as many tests as possible. It’s completely natural. Unfortunately, they are limited by reality. At some point during testing, the number of existing test cases causes significant problems.

Tests simply have to be maintained. Covering manual test cases with automatic tests is supposed to unburden manual testers in theory. However, this cannot be achieved if manual cases are not regularly updated in the context of new functionalities, and automatic tests along with them.

False positive tests

What might the scale of the problem be?

This is a very dangerous question. It’s easy to identify test cases that have stopped working. Such tests simply „fail„, and they need to be verified manually, so that they can be improved while performing them. In the case of automatic tests, they also end with a negative result, so a similar path appears, though it takes a little longer. Unfortunately, I have witnessed a lot of projects that have stopped there. However, there is a much more harmful phenomenon. What if the test passes despite being outdated? The number of false-positive tests in a project that is focused on the number of tests, not the quality, is something that simply cannot be controlled.

Software testing – quantity or quality?

Another consequence is the decreasing quality of tests, both when it comes to the code and the test cases themselves. This can be observed over time. The older test cases are detailed, containing everything you need. The new ones are just vague descriptions created quickly, and updating them is problematic. The mess in the code gets bigger, and the test code review turns into a quick look at the code. Testers don’t have enough time and it all starts to resemble a slippery slope: the more tests there are, the more problems there are… and the management expects a constant increase in test cases.

Of course, everybody is aware of this and tries to solve the problem in many ways. But which of them is effective?

Too many tests – possible solutions

Escape to BDD

Sometimes Behavior-Driven Development is offered as a solution. While it seems to be one of the most interesting directions, in many projects it brings the opposite effect. I say „yes” to BDD if it is implemented for everyone but not if only for automation testers. In many projects, I have seen an approach in which manual testers have written scenarios using the classic method and automating testers with BDD. Wasting time on another layer of abstraction is the straw that breaks the camel’s back.

In BDD, test managers often see another type of solution: it might seem that if the team automates a big part of the code and and there is a group of steps that already work, then anyone in a project can automate as well. In fact, steps become a new pseudo language (which becomes unnecessary if the code is well-written).

In order to solve the problem of scale, new testers join the project – often with very little knowledge and experience, because the price is the key factor. Managers delude themselves that by using BDD, you can develop automation simply by reusing ready-made steps written in natural language. Often such people can do very well in copying code, but not everyone has the talent and qualities to learn automation this way. In my view, it is an incorrect assumption. I even think that if a tester, looking at the name of the tests and the methods used written in Selenium, cannot understand what a given test is for, then he has no chance of development.

Because of the above solution, the project now includes a person who needs to be taken care of, who other team members need to spend time with – and the conditions of the project do not create opportunities for them to develop.

Experienced testers can join the project to improve the automation code or support the team in manual tests. This is an important remedy, but still does not solve the problem of the continuous increase of number of tests (which, as we already know, will also need improvement).

Responsibility for tests

A solution that I am in favor of, and which can be observed increasingly rarely, is an increase in the part played by the test manager. A person whose role is quite often limited to protecting tests from interference by management and fighting for more resources must get back to basics. Unfortunately, elements such as creating test strategies, planning and coordinating tests and reporting the results are becoming relics of a bygone era. It’s not that only one person in the project should think of everything. Everyone involved in testing should act as a responsible test manager. However, in order for this approach to work, the initial assumption must be that „it is not about the number of tests but about the quality and proper management”.

For many people, limiting tests is tantamount to heresy – it seems like the whole world is saying “We need more tests!”. But no, that’s not the point. It is rather about

better planning of work, as well as making the software tester take responsibility for the test plan.

As a tester working on a given area, I know which tests are relevant and what will bring value to the customer. I take responsibility for what test cases I create, as well as what I automate. Certain areas of testing may be irrelevant at a given stage of the project. It seems reasonable to dedicate our efforts not only to performing tests, but also to analyzing the justification behind it.

For example, in many cases, automating each path makes no sense. Many paths cover similar areas, so there is no need to run separate cases. Let’s go back to the basics of test methods and techniques, which most often do not really answer the question of „how to best test”, but rather „how to perform tests effectively, with the least amount of work, bearing in mind the risks resulting from this method”. But let’s not ask the test manager to tell us what to do. It is the testers who should provide information to the test manager on how many and what tests they perform at a given time in a specific release. The test manager does not work with a given functionality and does not have all this knowledge.

Impact analysis can be very useful – areas that are already very well tested and stable often do not need to be tested again. This can be determined on the basis of the abovementioned analysis. As testers, you have to do it with every new Sprint. Such an analysis should answer the following questions:

• how new features will affect old ones,
• whether we need to update tests,
• what tests should be performed to make sure that everything works properly, and in what areas,
• which automatic tests will help, which tests should be automated, which should be performed manually, and which are exploratory, only leaving evidence in the form of checklists

Read also: End to end testing with Cypress

Risk management

When introducing a new approach, there will certainly be a fear of the „butterfly effect” associated with regression, but the tester should think of testing as a kind of risk management, because that’s what testing is. Of course, areas of higher risk must be covered by appropriate sets of tests, and we have room to act in the areas of lower risk. Do not be afraid to test less, betting on the better quality of the tests performed, as well as stability. Mistakes will always occur; the question is how important and harmful it is to focus on all areas to the same extent.

Conclusion

Every tester should act like a test manager. Everyone should take responsibility for decisions and managing their capacity in a given project. So who should be the test manager in an Agile team? A person who aggregates information, presents the results to management, and supports testers in making decisions about taking responsibility. The test manager must also be able to explain to decision-makers that if there are 1600 automatic test scripts, 600 of which are worthless, then this is not the basis for adding another 600. Rather, slowing down, improving quality and thinking about which tests are right. To sum up, both the test manager and the tester must not be afraid to ask the question: “Do we really need so many tests?”