Continuous testing: get started with shift left testing approach

Traditional vs continuous testing

In traditional software development methodologies, such as Waterfall, the software testing process follows the design and development phases. The tested object is the finished, final product. The main problems caused by this are: long time taken to repair bugs, the high cost, and the risk that the changes introduced will affect other functionalities (which in turn results in a need to run all tests once again, and thereby leads to extending the entire process).

In recent years, so-called Agile methodologies (Scrum, Kanban, etc.) are the state-of-the-art in software development. Thanks to the iterative approach, it is not the entire finished product that is tested, but its fragments earlier in the development life cycle: previously developed versions, albeit not the final ones.

Agile: Scrum or… Scrumfall? 

Unfortunately, the incorrect implementation of Agile methodologies also carries the risk of the abovementioned issues. Let me give an example of one of my earlier projects, where phases were clearly divided during the Sprint into discussing tasks, development, and then testing. What did such a Scrumfall result in at the end of the Sprint?

• The testers had little time to run tests (because the development of the task took longer than expected). 
• The developers had little time to make any corrections (because the testers needed time for testing),  
• The Product Owner suggested that maybe from time to time we could turn a blind eye to the tests (because we have to deliver the product, and we can test it later),  
• The client wondered if there is a need for full-time testers (because they don’t do anything in the first week of the Sprint).

What is more, there were situations during the testing stage when it turned out that the developer understood the task one way, the tester another way, and the client meant something completely different. Which is why the task that was originally considered “almost finished” became a task “to be discussed” (which extended the process of delivery). 

The shift-left testing approach – what is it? 

It is not only a misunderstanding of Agile in an organization that can increase the risk of project failure. The problem of planning time for software tests occurs in many projects, regardless of the methodology used. It turns out, however, that there is an approach which might be, if not a panacea, certainly a medicine for alleviating the symptoms of inadequately planned project work: shift-left testing. In this approach, the start of the testing process takes place at the earliest stage of software development.

Now let’s go through all the stages of software development and answer the questions:

”Is it early enough for testing? and if not – why?”

Testing in a production environment 

The question regarding testing in a production environment is always debatable and often leads to the answer: “It depends.” So is it worth testing at this stage? Yes, in my view. However, we are talking about so-called smoke tests here. We use them to check whether the key paths are correct. Detailed tests should be run at earlier stages, if possible. When is it not possible? For example: the service belongs to an external provider and it is mocked on lower environments.

Another example: we send a query to another service, the production base of which is 1000 times bigger than in the test environment (several million objects vs several thousand), which significantly affects the size of the received data. . Yet another example of testing in production can be e.g. configuration testing. In one of the projects, we were tasked with changing the way in which an application is developed. Nothing had changed on the user level. But everything had changed for the development team, as a new tool and a new configuration were implemented. Moreover, it turned out that its configurations were slightly different in each of the environments.

Testing in a test environment 

The test environment tries to reflect the production environment as much as possible. We perform regression tests on it (in the form of automated tests launched every night), which are not only to check the widest possible range of functionalities of our service, but also its integration with other services.

This approach allows us to make sure that the latest version of the application can be released. However, running tests is still not the earliest stage of the software development process. We consider it some kind of “double check” of the applied change. This approach results from the fact that the repair process takes a lot of time due to the necessity of creating a new version of the code with the fix, performing a code review, merging and releasing a new version of the application.

Testing in a development environment

This environment is used by programmers for software development. It is characterized by the fact that the version of the application implemented on it may change very often (even several times a day) and it does not have to work properly (it does not have to be stable). However, due to the very short implementation time, it is also an excellent environment for conducting functional tests (both initial and comprehensive) and regression tests. Thanks to the tests in this environment, as a tester, I am able to tell in a very short time whether a given version can be moved to the test environment or if it needs to be changed.

Read also: All you need to know about Quality Assurance (QA) and Quality Control (QC)

Sometimes these changes are introduced almost in real time – the developer observes the behavior of the application during tests, analyzes the cause of the problem, introduces a fix, and then a retest is performed to check if everything works. This approach significantly shortens the time needed to receive feedback from the tester. It also shortens the code review process, because the delivered code works as it should, and we do not need to modify it to fix defects (which would only extend the review phase).

Testing in the business analysis and requirements collection phase 

Can you test documentation or business requirements? Of course! Very often I encounter situations in which goals and the scope of tasks are set, everyone says that they understand and know what needs to be done, and then questions come up such as:

• “Would that affect another area of the application?”,
• “Is this logic compliant with the requirements of other services?”,
• “Do we know where to download the data from?”.

And suddenly it turns out that a task that would already be included in the scope of the Sprint is removed from it or is blocked from further explanation. If not for this, it is highly likely that these questions would arise during the development phase, which would affect the time of software development or even a failure to provide functionality.

Is earlier testing possible?  

It would seem that business analysis is the earliest stage of the software development process in which it is possible to test software. And therefore – testing cannot be shifted further left. Well… that’s not entirely true. At the TestWarez 2019 conference, in his speech “Shift Left, Shift Right – Why These These Buzzwords Matter”, Marcus Merrell said that the next shift left to an earlier stage in testing is… the production environment. How come? Testing in production? Actually, not testing, but monitoring. And then drawing conclusions and creating new tasks based on them that can improve system operation. 

For example, one of the metrics we collect and analyze in our project is the type of business errors in the application. When an error occurs more frequently, we investigate its cause and consider how to reduce the frequency with which it occurs.

Benefits of shift left testing

The most obvious and widely emphasized advantage of shift-left testing is detecting bugs at the early stage of software development. Nevertheless, this approach brings other advantages in a project we are carrying out for an e-commerce client.

These advantages include:

Early automation

In discussions about test automation, the question often arises: “When to start automating?”. I always answer: “As soon as possible.” For example – the task is to add a new field and validations to the form. Do I have to wait for the developers to prepare the backend and frontend so that I can start manual testing and then create automated tests? No. Based on the description of the task, I create test cases to be undertaken, choose those that will be automated, and then prepare the test code. As a result, when the developer gives me a task for testing, it is very often sufficient to run such a code to quickly get information on whether the functionality works as intended or not. Sometimes I do not even have to perform manual tests, because automated ones will ensure coverage of all cases, and in addition, they will be faster and more precise.

Double-check

Early automation brings yet another benefit: functionality is double-checked. This means that the change is tested in the development environment, the code is merged, the new version of the application is released and uploaded to the test environment. The prepared and automated tests should give the same results in both environments. Is this really the case? Usually it is. Unfortunately, situations have arisen in which several different changes were merged one day, which resulted in the incorrect functioning of already tested functionalities. Fortunately, running tests every day allowed us to quickly detect and solve this problem.

Greater awareness of the importance of tests

Nowadays, the importance of tests in software development process is often emphasized. But is this really the case in reality? It turns out that it can differ. I would say that each team has a different approach. In my current project, initially, the work model was… typical, which means that the developer would start working on the task, prepare the change, the code would undergo the code review process, a new version of the application would be released, and the task moved to the testing stage. After a few months of work, this process has changed significantly. We have worked out an approach in which the code is checked by a QA engineer before the code review.

Shift-left testing – why it works for us

Will shifting left always work for every team? In my view: no. So why does the shift-left approach work for us? There are several reasons: 

1. Everyone is a member of a development team. There are no situations in which, for example, a team gets tasks to be carried out without prior consultation or in which testers and developers are in separate teams and have no contact with each other. Everyone has an insight into the task and can make comments at any stage. 
2. Suitable infrastructure. As a development team, we do not have to devote time to creating infrastructure, environments, etc. All this is prepared by a separate, dedicated team. We make a request for resources and we are provided with working solutions. 
3. A highly competent team. Team members have experience of working in both well-organized projects and those which cannot be described as such. This way all our experiences (“We were doing this and that, and it caused problems with this and that”) allows us to set up a much better development and testing process. Also, we know that our freedom is accepted as long as we deliver a high-quality product. Therefore, everyone wants to detect errors as soon as possible.