What is test automation? How does it work?
Before I discuss the test automation process further, a quick reminder: automated testing is software testing by executing a set of test cases using dedicated tools. Now, let’s see how such tests are created.
Firstly, we specify areas requiring automated testing, focusing on relevant issues from a business perspective. We set up the goal and terms depending on the tested system, bearing in mind one of the principles of testing: “Testing is context-dependent”.
Once the goal is set up, we proceed to the technical aspects. We choose a testing tool, and create the testing environment and testing scripts. Having run tests, we analyze the results and verify whether we have managed to obtain business value as assumed earlier on.
Deterministic automated tests work in a quite simple way. First of all, we put the application in the initial state to test a desired action, for example how the application acts while being used by, say, 1000 users. The aim of the verification is to compare the assumed results with the actual state.
Read also: Software testing: do we really need more tests?
Types of automated tests
There are several different types of automated tests. To get the most out of it, you need to know the key differences between the two. Tests are divided into functional and non-functional.
Functional testing
Is a quality assurance process and a type of black box testing that bases its test cases on the specifications of the software component under test.
- Unit testing: Unit tests are the first level of testing and form the basis for other more extensive types of testing. These are usually done manually by the developers.
- Integration testing: This type allows for layered testing. This type allows for layered testing. Generally, it’s about integrating 1 component with 2. For example, API with a database. This is often at the code level. 1st class with 2nd class (if developers do it).
- System testing: It’s a type of functional test that only covers the most crucial features of the software to ensure that it could be further tested.
- Regression testing: A type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software.
- User Acceptance Testing: Software test, the purpose of which is not to detect errors, but only to obtain a formal confirmation of the implementation of the software of appropriate quality.
Non-functional testing
Is testing an application or system against non-functional requirements: how the system works, not the specific behaviors of that system.
- Documentation testing: includes the plans, results, and testing of a system or system component. It includes test case specifications, test plans, test procedures, test reports, and test logs.
- Installation testing: Also known as Implementation testing and it is done to check whether the software is properly installed in the new system with its prerequisites and necessary data and is working as expected or not.
- Performance testing: The practice of evaluating how a system performs in terms of responsiveness and stability under a particular workload. Performance tests are typically executed to examine speed, robustness, reliability, and application size.
- Reliability testing: An important software testing technique that is performed by the team to ensure that the software is performing and functioning consistently in each environmental condition as well as in a specified period.
- Security testing: Type of Software Testing that uncovers vulnerabilities, threats, risks in a software application and prevents malicious attacks from intruders.
Automation testing and myths
1. Each test can be automated
According to one of the widespread myths, each test can be automated. Let’s think it over: is it really possible? As per the International Software Testing Qualifications Board – the software testing certification organization – the testing process is infinite, which is why we cannot forecast and test everything.
Test automation is still incomparable to efficiency tests, volume testing, and stress testing. Automated testing has its limitations, though. For example, it cannot be used for non-functional tests. These are related to our personal perception of esthetics, legibility and transparency.
In the abovementioned area we are unable to define the initial requirements and the expected final results. Some of the aspects, for example the appearance and transparency of the application, are perceptible only to the human eye. The same thing applies to accessibility testing, which aims to determine whether users with disabilities can use the system with ease.
All in all, knowing that some of the tests cannot be automated, we understand that it is impossible to entirely replace manual testing with automated testing – as per the second common myth. There will always be areas requiring manual testing.
2. Automated tests can replace manual testing
This misunderstanding brings to mind a common myth from the past, that one day television would entirely replace radio. Nowadays, as we can see, radio, television and the Internet coexist and complement each other. It works similarly when it comes to automated and manual tests.
The first ones are advantageous due to their repeatability. With appropriately implemented tests we can be sure that every test will be run in the same, replicable manner.
Additionally, contrary to human nature, automatons do not need coffee, the weekend or sleep, so they can work perpetually. On the other hand, humans are better when it comes to intelligence, intuition and imagination, which are essential to running an effective test of the application.
The majority of tests related to functional and non-functional requirements can be performed manually. There will always be a need for manual tests, calling for the engagement and knowledge of a tester – namely usability tests, accessibility tests and security tests.
Test automation supports manual testing, but cannot replace exploratory testing or the technique of error guessing, wherein the tester’s knowledge in the context of the product and tools is essential.
3. Automated tests help to catch a lot of defects
Contrary to popular opinion, automated tests will not help to detect a lot of defects. One can ask: “How can that be? Isn’t that what we run them for?” Of course, it is one of the goals we are trying to attain; however, automated tests are quality-oriented, not quantity-oriented. To give an example, automated tests are useful in catching functionalities’ change-related defects in the event that software code has been amended. These are called regression tests and are used to ensure that changes implemented to either software, configuration or the environment will not impact the developed product itself.
Automated regression tests can successfully replace manual testing carried out by entire development teams, which – depending on the software’s complexity – would normally take hours, days or even weeks. Our trust increases with every single automated test. Even though they do not catch a lot of defects, they allow us to save time and money, enabling us to focus on the quality of the product instead.
4. Each automatable case should be automated
Knowing that not all tests can be automated and that there are areas in which we will still need manual testing – is it worth automating every automatable testing case?
There are situations in which swift software testing is required and engaging additional resources, tools or funds (which is often related to test automation) is not an option. Let’s paraphrase a quote by Roger Needham, Microsoft’s Research Laboratory Director, to clear up doubts: automation is a process of replacing something that works with something that almost works, but is quicker and cheaper.
This reminds us of the two principal ways of defining whether we should automate tests or not:
- automated tests should be faster than manual verification
- the cost of programming tests cannot exceed the income gained thanks to test automation
From a business perspective, the cost of tools and time needed for the implementation of automated tests in comparison with manual testing should be the decisive factor. This should facilitate the decision-making process in terms of automating testing cases.
Read more: The test pyramid – what, when and how to test software to get profits
Benefits of automation testing
Automated tests have many benefits. Application testing is a process. Every time the code is modified, it needs to be changed and retested in different modifications or systems. In such a situation, manual testing would be extremely time-consuming and thereby costly. With test automation, you can run tests multiple times at no extra cost.
Test automation also allows for continuous testing and a quick check of the app. This allows you to avoid bottlenecks at the end of the software development process. Test also automation helps make your life easier. Reporting in manual tests is uncomfortable and thanks to test automation, reporting is smooth.
Test automation tools
So how to run the tests in the most effective way? Here is a list of a few best practices for test automation, as well as the tools and automation frameworks that will help you achieve success.
To start with, determine which test cases should be automated first. Then select an automation tool for testing. If you have already defined your needs, you can proceed and choose the appropriate test solution, for example:
- Selenium
- Cypress
- Cucumber
- LambdaTest
Summary
Despite many myths related to test automation, automated tests are gaining greater understanding and popularity. Myths related to automated tests keep coming up, which proves that a lot still needs to be done to clarify doubts. I hope that I have managed to clear up some of them and that the article will help to define further discussion on automated tests. This is priceless help for testers in the process of delivering a high-quality product.
Technical leader and software tester, specializes in automated testing. Her adventure with testing began quite unexpectedly – on the production line of the Slovak car factory. Then she entered the world of embedded software testing and fell in love with the diverse world of mobile and web applications.