AWS Lambda at a glance

Let’s start with the fact that serverless services are currently some of the most advanced features of cloud computing. Today, every cloud provider has function-as-a-service solutions in their offer. In the project I am involved in, we use AWS Lambda (read more about its capabilities in this article: AWS Lambda Functions). 

In a nutshell, AWS Lambda is a serverless service provided by Amazon Web Services that allows developers to run code in the cloud in response to various events. The serverless model means that infrastructure management is on the cloud provider’s side. This makes it just perfect for developing applications requiring rapid scaling and efficient resource management. 

Also read: 

• Automation testing tools: Selenium + Python vs Selenium + Java  
• Python libraries – Pandas  

The testing challenge. Why do you need to test serverless applications?

What do standard applications and serverless applications have in common? Both should pass unit tests. Testing serverless Lambda function code is quite a challenge. Using my project as an example, I would like to present the challenges we face as a team in the context of the unit testing of our application. 
 
Firstly, AWS Lambda functions are often tightly integrated with other cloud resources, such as databases or message queues. This can cause problems if developers want to isolate unit tests, as we need to create tests independent of other components. In any scenario, we need to ensure that our unit tests do not affect the state of other services (such as changing records in a database). 

Another challenge is handling asynchronous communication. In many scenarios, our application responds to asynchronous events such as processing an incoming message from a queue. Testing such cases requires appropriate strategies to keep control of the data flow and ensure that the application behaves as we expect.  

Why is test coverage within a Lambda function so important?

Before getting into the practical aspects of unit testing, let’s see why it is so important, especially in the context of Lambda functions. 

Keeping functions working correctly  

Lambda functions are often small pieces of code invoked by specific events. Unit tests help ensure that these functions behave as expected in different scenarios.  

Quick error detection  

AWS Lambda can respond to various events. If your unit tests are well-written, it can quickly detect errors in the code while the product is still being developed. Unit testing is crucial, as it allows you to avoid introducing errors into the production environment and thus ensures that the application is reliable at this stage. 

Shortening the development cycle  

Unit testing enables developers to iterate quickly when making changes to code. By shortening the development cycle, the application is developed faster and the value is delivered quicker.  
 
Now that we know why unit testing is important, let’s consider how to do it effectively with the PyTest framework. 

What exactly is PyTest?

PyTest is a tool for testing code in Python which allows you to write both simple and complex tests. Among the available tools, Selenium, Cypress, Playright, or Robot Framework are popular (we have already discussed them for you in other articles and on webinars). It’s time to take a closer look at PyTest, which is particularly useful in an AWS Lambda function environment, as it allows local testing before deployment. 

What is more, thanks to its integration with mocking libraries such as Moto, you can simulate (“mock”) AWS services directly in your local environment. This will allow you to test your code thoroughly before deploying to the AWS cloud. 

Elevate Your Application Development

Our tailored Application Development services meet your unique business needs. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

Practical applications of unit testing

1. Handling expected errors 

Let’s assume we are developing a Lambda function to process online orders. The unit tests may include error-handling scenarios, such as no connection to the database or a problem with a third-party payment provider. By ensuring that the function handles these situations correctly, we can increase the system’s resilience to failures. 

# An example of code for error-handling 

 

def test_error_handling():  

 

 # We simulate a database communication error  

 

 event = {"order_id": "123"}  

 

 result = process_order(event, None)  

 

  

 

 assert result == {"statusCode": 500, "message": "Internal server error"}  

2. Validation of input data  

Lambda functions are often called by events, and the input data is crucial for the function to work correctly. Unit tests can cover different instances of input data. You gain confidence that the function validates them correctly and responds as expected. 

 #Example of a unit test for validating input data
def test_input_validation():  

 

 event = {"customer_id": "123", "order_total": 50.0}  

 

 result = process_order(event, None)  

 

 assert result == {"statusCode": 200, "message": "Order processed successfully"}  

# We test invalid input data
 
 event_invalid = {"customer_id": "abc", "order_total": "invalid"}  

 

 result_invalid = process_order(event_invalid, None)  

 

 assert result_invalid == {"statusCode": 400, "message": "Invalid input data"}  

3. Integrations with AWS services  

Lambda functions often use other AWS services, such as the storage service S3 or the NoSQL database service DynamoDB. We can use mocking techniques to avoid actual calls to these services while testing. PyTest offers the pytest-mock module, which makes it easy to create mocks for selected services. 

import pytest  

 

from my_module import process_s3_event  

# An example of a unit test using pytest-mock to mock an AWS service 

def test_s3_upload(mocker):  
 # We create a mock of the S3 client 

 

 s3_client_mock = mocker.Mock()  

 

 # We update the (“patch”) "boto3.client" function in such a way that it returns our mock  

 

 mocker.patch("boto3.client", return_value=s3_client_mock)  

 

  

 

 # We create an example of an event  

 

 event = {"data": "example"}  

 

 # We call the function that processes the event  

 

 result = process_s3_event(event, None)  

 

  

 

 # We check if the function has returned the expected result  

 

 assert result == {"statusCode": 200, "message": "S3 event processed successfully"}  

 

 # We check that the "upload_file" method on the S3 client's mockup was called exactly once 

 

 s3_client_mock.upload_file.assert_called_once_with("source_file", "bucket", "destination_key")  

4. Multi-layer applications  

In a microservice architecture, where Lambda functions act as microservices, unit testing becomes even more complex. A unit testing framework like PyTest allows you to create layered tests that check integrations between various microservices. 

import pytest  

 

from my_module import process_microservices  

 

  

 

# An example of a multi-layer unit test 

 

def test_microservices_integration(mocker):  

 

 # We create a mock for Lambda A function  

 

 lambda_a_mock = mocker.Mock()  

 

 # We create a mock for Lambda B function  

 

 lambda_b_mock = mocker.Mock()  

 

  

 

 # We define what data will be passed between microservices 

 

 event_a_to_b = {"data": "example_data_from_A_to_B"}  

 

 event_b_to_a = {"data": "example_data_from_B_to_A"}  

 

  

 

 # We determine what should be returned by mocks of the Lambda function  

 

 lambda_a_mock.return_value = {"statusCode": 200, "message": "Response from Lambda A"}  

 

 lambda_b_mock.return_value = {"statusCode": 200, "message": "Response from Lambda B"}  

 

  

 

 # We patch the imports in the process_microservices function to use our mocks 

 

 mocker.patch("your_module.lambda_a_function", lambda_a_mock)  

 

 mocker.patch("your_module.lambda_b_function", lambda_b_mock)  

 

  

 

 # We call the process_microservices function with sample data 

 

 result = process_microservices(event_a_to_b, None)  

 

  

 

 # We check if the function has returned the expected result after the integration of microservices 

 

 assert result == {"statusCode": 200, "message": "Microservices processed successfully"}  

 

  

 

 # We check that Lambda functions A and B were called with the right data 

 

 lambda_a_mock.assert_called_once_with(event_a_to_b, None)  

 

 lambda_b_mock.assert_called_once_with(event_b_to_a, None)  
 

Techniques of testing AWS Lambda functions

1. Fault tolerance testing  

This is a key component of AWS Lambda function testing. Fault tolerance testing covers scenarios in which one of the AWS services working with the function is unavailable or returns errors. An example of the test may look like this: 

import pytest  

 

from unittest.mock import patch, Mock  

 

  

 

  

 

# Example of a fault-tolerance test  

 

def test_error_resilience(mocker):  

 

 # We mock a service that returns an error  

 

 external_service_mock = mocker.Mock()  

 

 external_service_mock.side_effect = Exception("Service unavailable")  

 

  

 

 mocker.patch("boto3.client", return_value=external_service_mock)  

 

  

 

 event = {"data": "example"}  

 

 result = process_event(event, None)  

 

 assert result == {"statusCode": 500, "message": "Internal server error"}  

 

  

 

  

 

2. Test-Driven Development (TDD) tests 

The practice of TDD means writing unit tests before implementing functions. This allows for a more informed code design and ensures that tests cover every line of code. 

Below is an example of TDD for a Lambda function: 

import pytest  

 

  

 

# Example of unit testing with TDD  

 

def test_tdd():  

 

 event = {"data": "example"}  

 

 result = process_event(event, None)  

 

 assert result == {"statusCode": 200, "message": "Event processed successfully"}  

 

  

 

 # We add a new functionality and write a test for it first 

 

 new_feature_result = new_feature(event)  

 

 assert new_feature_result == {  

 

 "statusCode": 200,  

 

 "message": "New feature processed successfully",  

 

 }  
  

 

3. Using Fixture in PyTest  

PyTest Fixtures are functions that run before tests (and optionally after them) that can prepare the necessary resources. For example, a fixture can create a fake input or configure a mock of any AWS resource before running tests. This way, each test runs in an isolated and controlled environment.

import boto3  

 

import pytest  

 

from moto import mock_s3  

 

  

 

@pytest.fixture()  

 

def mock_s3_resource():  

 

 with mock_s3():  

 

 # We use Moto library to simulate the S3 service  

 

 import boto3  

 

  

 

 s3 = boto3.resource("s3", region_name="us-east-1")  

 

 # Crearing mock – AWS S3 resource 

 

 s3.create_bucket(Bucket="mocked-bucket")  

 

 yield s3  

 

 # Teardown (resource cleansing) is not required, Moto performs automatic cleansing  

 

  

 

# We use a decorator to turn on S3 simulation for the test  

 

@mock_s3  

 

def test_s3_operations(mock_s3_resource):  

 

 s3 = mock_s3_resource  

 

  

 

 # Perform requested operations on S3  

 

 s3.Object("mocked-bucket", "test.txt").put(Body="Hello, World!")  

 

  

 

 # Check if the performed operations were successful  

 

 obj = s3.Object("mocked-bucket", "test.txt").get()  

 

 assert obj["Body"].read().decode("utf-8") == "Hello, World!"  

4. Parameterization of tests 

Parameterization allows you to run the same test with different input data sets. Using the @pytest.mark.parametrize decorator, you can specify different data sets for a single test function. This way, you can easily verify the performance of Lambda functions in different scenarios without writing multiple similar tests. 

@pytest.mark.parametrize(  

 

 "input, expected",  

 

 [  

 

 ({"data": "example1"}, {"statusCode": 200, "message": "Success"}),  

 

 ({"data": "example2"}, {"statusCode": 400, "message": "Error"}),  

 

 ],  

 

)  

 

def test_lambda_function(input, expected):  

 

 result = lambda_handler(input, None)  

 

 assert result == expected  

How to start unit testing with the PyTest framework

PyTest installation 

1. Make sure you have Python installed on your system.  
2. Open a terminal or command line.  
3. To install PyTest, type the following command: 

pip install pytest  

3. If you plan to use libraries like Moto for more advanced testing, of course, you will need to install them as well – for example: 

pip install moto  

Test preparation  

4. Prepare your unit tests in Python (.py) files that contain test functions. Your tests should be organized following the PyTest convention, in which test function names start with “test_”.  

Running tests  

5. In the terminal, navigate to the directory where your test files are located, or, if they are in the same directory as the terminal, you can skip this step. 

6. To execute all defined tests from a given file, use the following command, specifying the name of the test file: 

pytest file_name.py  

For example, if your tests are in the my_tests.py file, the command will look like this:  

pytest my_tests.py  

If you want to test all files located in the current directory, use the command: 

pytest .  

Test results analysis 

8. Once the tests are executed, PyTest will provide detailed information about the test run and its results. You will see which tests succeeded, which failed, and why. 

Practical application of more advanced unit testing and mocking of AWS services for Lambda 

Now that you know what the PyTest framework is, what unit tests are for, and how they can be used with various techniques, it’s time to move on to a more advanced, practical application of unit testing and AWS service mocking for Lambda functions. When developing serverless event-driven applications using AWS Lambda, the best practice is to validate individual components and services.  

Now let’s get into an example of a function for handling notifications in the cloud. Imagine that we have an application that needs to quickly notify users about various events – from a meeting reminder to a sale notification.  

That’s where AWS SNS, a service that allows you to send notifications in the form of messages to a wide range of subscribers, comes in. We can think of it as a cloud-based instant messaging system. Here, the AWS Lambda function acts as an intermediary – it accepts specific requests and forwards them to SNS, which then handles the distribution of messages. 

How does it work? The process is simple:  

1. Message preparation: The Lambda function takes two key elements – the identifier of the SNS subject (to whom we want to send the notification) and the message content.  
2. Sending the message: The function communicates with the SNS using a special API to deliver the message to the specified subject. 

You may be wondering: how can we trust that this function works properly? The answer lies in unit testing itself. We run test scenarios to make sure our code does exactly what it’s supposed to do in different situations. We simulate (“mock”) the operation of the SNS service to see how the function behaves when everything goes successfully, and how it handles any errors. 

Before writing the first unit test, we will look at an example of the AWS Lambda function we want to test: 

main.py:  

 

import boto3  

 

import json  

 

  

 

def send_notification(sns_topic_arn, message):  

 

 if not message:  

 

 raise ValueError("Message cannot be empty")  

 

 try:  

 

 sns_client = boto3.client("sns")  

 

 response = sns_client.publish(TopicArn=sns_topic_arn, Message=message)  

 

 return response  

 

 except boto3.exceptions.Boto3Error as e:  

 

 print(f"Error sending notification: {e}")  

 

 raise  

 

  

 

def lambda_handler(event, context):  

 

 # Parsowanie zdarzenia wejściowego  

 

 sns_topic_arn = event.get("sns_topic_arn")  

 

 message = event.get("message")  

 

  

 

 # Validation of input data  

 

 if not sns_topic_arn or not message:  

 

 return {"statusCode": 400, "body": json.dumps("Missing sns_topic_arn or message")}  

 

  

 

  try:  

 

 # Calling function sending notification  

 

 response = send_notification(sns_topic_arn, message)  

 

 return {"statusCode": 200, "body": json.dumps("Notification sent successfully")}  

 

 except Exception as e:  

 

 return {"statusCode": 500, "body": json.dumps(str(e))}  

 

  

  

 

import pytest  

 

import boto3  

 

from moto import mock_sns  

 

from unittest.mock import patch  

 

from main import lambda_handler # Importing from a handler  

 

  

 

# Preparing test environment  

 

@mock_sns  

 

class TestLambdaFunction:  

 

 # Test to see if we correctly handle the situation when input is missing 

 

 def test_missing_data(self):  

 

 response = lambda_handler({"sns_topic_arn": "", "message": ""}, None)  

 

 assert response["statusCode"] == 400  

 

 assert "Missing sns_topic_arn or message" in response["body"]  

 

  

 

 # Test to see if the function sends the notification correctly 

 

 @patch("main.boto3.client")  

 

 def test_send_notification_success(self, mock_boto3_client):  

 

 mock_sns_client = boto3.client("sns")  

 

 mock_boto3_client.return_value = mock_sns_client  

 

 mock_sns_client.publish.return_value = {"MessageId": "12345"}  

 

  

 

 response = lambda_handler(  

 

 {  

 

 "sns_topic_arn": "arn:aws:sns:region:123456789012:testTopic",  

 

 "message": "Test message",  

 

 },  

 

 None,  

 

 )  

 

 assert response["statusCode"] == 200  

 

 assert "Notification sent successfully" in response["body"]  

 

  

 

 # Test to check how the functions cope with SNS errors  

 

 @patch("main.boto3.client")  

 

 def test_sns_error(self, mock_boto3_client):  

 

 mock_sns_client = boto3.client("sns")  

 

 mock_boto3_client.return_value = mock_sns_client  

 

 mock_sns_client.publish.side_effect = boto3.exceptions.Boto3Error  

 

  

 

  response = lambda_handler(  

 

 {  

 

 "sns_topic_arn": "arn:aws:sns:region:123456789012:testTopic",  

 

 "message": "Test message",  

 

 },  

 

 None,  

 

 )  

 

 

 assert response["statusCode"] == 500  

How to interpret tests

• test_missing_data: We check if the function responds correctly when we do not provide the required data. We expect a response with error code 400, meaning the query was invalid.  
• test_send_notification_success: Here we simulate a situation in which everything goes successfully (the so-called “happy path”). We use mock_boto3_client to simulate the interaction with SNS, and then check if the function returns a success code of 200 and the corresponding message.  
• test_sns_error: We test how the function will behave when the SNS reports an error. Again, we simulate interactions with the SNS and check if the function handles the error correctly, returning a status code of 500. 

Remember, these tests are just examples. You may want to test more scenarios to make sure a function works as it should in all possible situations. Unit tests can greatly improve the quality and reliability of your code, so it’s worth paying proper attention to them! 

FAQ – frequently asked questions

AWS cloud function testing in Python – summary  

Unit testing of AWS Lambda functions with PyTest is more than just a code validation process! It’s also an opportunity to develop more stable, reliable, and efficient applications. Practical usages of unit testing include error handling, input validation, integration with AWS services, performance optimization, and fault tolerance testing. Advanced techniques, such as using PyTest fixtures and test parameterization, allow you to keep even tighter control of code quality. Unit testing is not only a pre-implementation step, but also a key part of continuous application improvement in the dynamic world of the cloud.  

I hope this article has helped you understand how we can use unit testing and AWS service mocking to ensure the reliability and correctness of our AWS Lambda functions.

Functional testing – functional requirements

Functional testing focuses on the most basic requirements. This type of testing aims to check whether the functional aspects of software applications work as intended. The project team gets the requirements ready, and the client or employer knows more or less exactly what they want to get. During the process itself, these requirements may change. Nevertheless, we still have the target in front of our eyes and know where we are headed. A tester, programmer, business analyst or even the end user can test the application against the functional requirements, and there is usually no need to acquire complex theoretical knowledge about how to test. The application needs to be “clicked through,” checking whether it works according to the business requirements.

Also read:

• Test case
• Testing tools: Cypress vs Selenium

Non-functional testing

Non-functional tests are also referred to as quality tests. This aspect is more complicated than in the case of functional tests. Most often, the scope they check is defined by a short sentence: “How the system works”. Let’s start with a breakdown based on the ISTQB Test Analyst syllabus.

Non-functional tests can check non-functional aspects such as:

• usability
• security
• reliability
• performance
• maintainability
• portability
• compatibility

Importance of non-functional tests

Only those companies that are aware of the issue and the risks related to not testing decide to carry out non-functional tests and check non-functional requirements. However, not everything can be predicted or found. From time to time, even giants like LinkedIn (from which the data of 700 million users, 93% of accounts, was leaked and put up for sale) can also learn that the hard way. Facebook (and the related applications such as WhatsApp, Messenger, and Instagram), which in October 2021 experienced its biggest-ever outage and was inaccessible for almost 6 hours, has also faced similar problems. That failure generated losses of about $6 billion! The public projects case is particularly interesting. Although they seem to be subject to several legal requirements, and therefore much more difficult to implement, they are frequently subject to failure, such as:

• Security vulnerabilities – such as a vulnerability in the e-Toll system which allowed for the downloading of user data.
• Performance problems – such as those faced by government servers.
• Unsuitability for users – the low intuitiveness of the system hinders the operation of applications for tolling on highways, which has brought about criticism from many drivers. So why aren’t non-functional tests performed? In small (e.g., up to 10 people) and short-term projects, even if awareness of non-functional testing is high, the reason for overlooking them is obvious: the costs involved.

Non-functional tests

Performance tests

Among the non-functional tests mentioned earlier on, the awareness of 2 types of tests in particular is increasing, and they are worth looking into.

The first, with a lower entry level, is performance testing. The subject seems quite complicated, but after learning a few tools (e.g., JMeter, Gatling), it turns out that you can relatively quickly start running basic performance tests that will catch serious problems in an application. The following types of performance tests can be distinguished:

• Load testing – testing the performance of the system according to different load levels, falling within the assumed range,
• verloading – testing performance beyond the maximum expected load or with resource availability lower than the minimum requirements,
• Scalability testing – checking the system’s ability to increase or decrease the resources used depending on the load,
• Spikes – testing the system’s ability to process a spike in load and then return to a typical load,
• Endurance – testing the stability of the system under a given load over a period that is adequate and proportional to production realities,
• Concurrency – testing the situation in which the same action is performed simultaneously,
• Throughput – determining the maximum load the system can handle while meeting performance requirements.

Requirements for performance testing

When developing new applications, it is difficult to give exact requirements for performance tests. They are usually based on defining production profiles or on predictions or experience with similar applications that are already running. It is easier to define requirements when implementing a newer version of the software, because here one can refer to production data, such as the number of queries per second, bandwidth, or computing power. One of the risks in performing performance tests is inappropriate execution (e.g., by inexperienced people without the requisite competences). This can even lead to a lack of analysis of key system components or the misinterpretation of results. Another big risk is conducting tests on test environments, which are usually not the same as production environments due to configuration, data volume and capacity.

Safety tests

The second, much broader field is security. Security tests check whether the system sufficiently protects data and functionality from third-party attacks. The importance of such tests is constantly growing. Who hasn’t heard of various hacks, leaks, frauds or thefts resulting from insufficient security or low user awareness?

However, it isn’t easy to find a tester who deals with functional testing on a daily basis, in addition to being able to conduct security tests from time to time. That’s why security audits are most often (if not always) outsourced after a system (or part of it) has been developed. This aspect is very broad, and it’s a good idea to start learning by exploring the OWASP Top 10 – a document that lists the 10 most common security risks. Some types of security tests are as follows:

• Vulnerability scanning – performed using tools that scan a program to look for known vulnerabilities in its components,
• Security scanning – focuses on the security of application configurations, networks and other systems,
• Penetration testing – a process that simulates an actual attack,
• Security audit – an extensive, structured application inspection process that follows defined standards,
• Risk assessment – analyzing and identifying the biggest security risks,
• Security health check – a combination of security scanning, penetration testing and risk assessment to determine the level of current security measures and their effectiveness.

Usability tests

These tests check whether the user can easily learn how to use the application and whether their experience will be positive. Aspects such as aesthetics, intuitiveness, and the ability for people with disabilities to use the app also come into play here. It’s a good idea to keep this in mind and focus on these issues from the very beginning of the project, especially in agile projects, where we have constant contact with the client and a real opportunity to influence changes to the functionality being created. Indiscriminately adding functionalities according to the requirements of the client (who is not always technically-minded) is not the best approach. Conducting usability testing only after the application is completed can result in a significant increase in project costs due to additional work.

Reliability tests

Reliability testing means verifying if a system can operate correctly for a specified period under specified conditions. However, it is difficult to simulate production conditions on test environments over a long enough period, which is why, among other things, such testing can take place partially on production servers. An example of reliability testing is placing a new version of an application on one of many servers (usually a less frequently used one) and leaving the previous version on the others. After an assumed period of application monitoring, the new version is uploaded to the remaining servers. This approach must be accompanied by appropriate configuration so that if there are problems with the new version on a single machine, traffic is automatically redirected to others. The reliability characteristics are:

• Maturity – the level of fulfillment of reliability requirements,
• Reachability – the time after which the system is available,
• Fault tolerance – the ability of a system to continue operating during a failure,
• Reproducibility – the ability to recover from a disaster, measured in time and the amount of lost data.

Maintainability tests

Maintainability testing allows you to analyze the complexity of maintaining a system in the future. They are rather rarely conducted, despite the fact that the maintenance of the program will (as assumed) last longer than its development. Prevention plays an important role from the beginning of the project – systematic code review and keeping records. It is very likely that in the future the system will be maintained or developed by other people rather than those who created it. For this reason, code and documentation transparency are key.

The main objectives of the maintainability tests are:

• Minimizing the cost of application maintenance,
• Minimizing the application downtime needed for maintenance.

Portability testing

Portability testing helps determine the level of complexity involved in moving a software component or application from one environment to another. In this case, too, separating them from the testing process and focusing only on such types of testing is rare. Agile design frameworks such as Scrum naturally foster portability testing. In an iterative approach to software development, portability is tested on frequent releases of successive versions (moving the application from the development/testing environment to production), even as often as every 2 weeks. Other situations in which it is worth conducting portability testing include moving from Internet Explorer to Chrome, from one version of a database to another, or extending the program’s installability to successive versions of Windows or Mac systems. Portability can be measured by the amount of work required to move a software component from one environment to another. The characteristics of portability are:

• Installability – the capability to install the program on a new system,
• Adaptability – the capability to install applications on all target systems,
• Substitutability – the capability to replace an existing software module.

Compatibility tests

Compatibility tests verify the capability of different programs to coexist in the same environment, as well as verifying the ability of a program to work according to different parameters. A very popular case is compatibility testing, which aims to verify the operation of applications in different browsers, such as Chrome, Firefox, Internet Explorer, Safari or Opera. Another case is the testing of mobile applications on many different devices. This approach identifies problems that customers may encounter using non-standard operating systems or browsers. Compatibility can be tested according to:

• equipment
• operating system
• software
• network
• browser
• mobile device
• software version

Test cases for the non-functional type of software testing

The non-functional type of software testing focuses on the aspects of the software that are not related to performance, usability, security, and scalability. When creating test cases for non-functional testing, it is important to consider various scenarios that can impact the overall performance and user experience of the software.

Non-functional testing tools

Non-functional testing tools can help a great deal in the testing process. There are different tools available used to test non-functional aspects (stress testing, volume testing, performance, etc.). Some examples of non-functional testing tools are JMeter, Loadster, and Loadrunner.

Testing non-functional requirements – key takeaways

• Non-functional testing is a type of testing that checks non-functional requirements (performance, usability, security, and scalability).
• Non-functional tests are also referred to as quality tests.
• It is important to recognize the significance of non-functional testing and understand the potential consequences of skipping this type of testing in a project.

Non-functional software testing – summary

Functional testing is very important, and in many projects, no one questions its relevance. Functional testing verifies functional requirements. Ongoing manual testing and regression development and maintenance practically exhaust the subject. However, we need to be aware of the importance of non-functional testing as well as the risks involved in skipping this type of testing.

Non-functional aspects are important. However, even at the test planning stage, there is often a lack of thorough analysis of what non-functional tests should be carried out, with the result that only some bugs are found. The risks are high: an unresponsive online store on Black Friday, leakage of confidential customer data, blocking of the site for ransomware, non-intuitiveness of the application or an unattractive design discouraging users from using it, and payments not working after uploading a new version… All these can prove to be much more costly than investing in systematic non-functional testing.

Elevate Your Application Development

Our tailored Application Development services meet your unique business needs. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

Software development and regression testing 

Let’s imagine that we have an application up and running and available to users. The customer is satisfied because it is popular, and wants to add a new functionality to increase user satisfaction and attract new users. The project team can’t wait to start working: new functionalities, new tasks, new challenges! After deciding on the details, the developers eagerly set to work. Then the testers check the new functionality. After minor improvements, everything works fine. It’s time to make the new functionality available to users. When the day came to release the new version of the application, everyone was happy – the project was completed on time and with the requirements fulfilled. 

Unfortunately, this blissful state of affairs did not last long. A day later, the website was flooded with reports that not all the existing application functions worked properly. The team was surprised: „We didn’t even touch those functionalities, and the new one works fine”. So what went wrong? 

The miserable consequences of not conducting regression tests… 

Well, there was one very important element missing in the testing process. No one checked whether the newly added functionality affected the existing ones. In other words, no regression tests were performed. 

Because of this lack of regression tests, the application didn’t gain new users. Furthermore, the majority of the existing ones stopped using it. It tumbled down the rankings, incomes decreased, which made the client unhappy.  

Elevate Your Application Development

Our tailored Application Development services meet your unique business needs. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

What exactly is regression testing?

According to the ISTQB dictionary, regression testing is the testing of a previously tested program following modification “to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed”. 

We can illustrate it this way: during regression tests, we check that „when laying wallpaper in one room, the roughcast did not fall off the wall in the other one”. Any changes to the code in one place can cause the function to break in another, although at first glance it seems that they are not related. That is why it is so important to retest the software after implementing changes.

Red also: Functional testing in software development

How often should regression tests be performed? 

Now you can see how important it is to make sure that the changes have not affected the existing functions and to find errors early on. Changes may result from modifications to the requirements and code of the software, adding new functionalities or repairing a defect, so it is important that regression tests are carried out as often as a new version of the software is released. 

At what levels do we perform regression testing? 

Regression tests are independent of the test levels and test types. They can be carried out at the level of modular, integration, system or acceptance tests and regardless of whether we are dealing with functional or non-functional tests.

Creating test cases 

When checking the software, the tester creates test cases. However, not all of them will be performed during regression tests. What does this depend on? 

It mostly depends on the quantity and time needed to run them. If we are dealing with an uncomplicated system and it will not take too much time to perform all test cases while testing regression, then they can be performed. 

Most often, however, systems or applications are characterized by a high level of complexity and are dependent on other systems. We are then unable to use all the created test cases. So what should be the decisive factor when choosing cases for regression tests? 

Selection of regression test cases  

If we have to limit ourselves, it is worth choosing a set of test cases accordingly. Certainly, it’s a good idea to include those cases that: 

1. Check the most critical functions of an app – first of all, test cases that verify the operation of main functions or functionalities that are often used. 
2. Were created on the basis of defects – test cases that were created on the basis of reported defects are important. 
3. Verify the vulnerability to errors – it is worth considering which area of the application is most vulnerable to errors after introducing changes, and also perform test cases from a given area while running regression tests. 
4. Check the new functionality – don’t forget about test cases that check the newly added functionality. 
5. Require many steps – let’s bear in mind complex test cases requiring a longer sequence of activities. 

Right Priority 

It is also important to prioritize test cases accordingly. This helps in the selection process, especially when the scope of the test area is large and we don’t have much time to perform them.  

In such a situation, the tester – with time in mind – selects the test cases with the highest priority. This gives us confidence that the most critical areas of the application will be tested. The remaining cases (with a lower priority, which were not performed during regression) should be included in the final report and marked as non-executed. 

How to prioritize test cases? 

There are several factors to consider when prioritizing test cases: 

• The importance of a given functionality from a business point of view 
• Its availability / visibility to the end user 
• The complexity of a given functionality 
• The financial impact  
• Whether it was created under time pressure 
• Whether there were problems with similar functionalities 

Taking into account the above aspects and answering the questions, prioritizing test cases should not be a problem. We will benefit from the work put into this process when the time for testing is very limited. 

Read also: The test pyramid – what, when and how to test software to get profits

Reusable test cases in regression 

The list of test cases selected for regression tests should be verified regularly. Some of them can be used in subsequent regression tests – these are the so-called reusable cases. Others may turn out to be outdated, and thus useless in subsequent regression cycles; such an „outdated case” should be removed from the list of regression tests.  

Verification of whether they are still useful mainly consists in trying to execute a given test case. If the steps described cannot be reproduced or the functionality no longer exists, then such a test case should be removed. It often turns out that the execution path or the expected result has been slightly modified and all you need to do is update the test case. Verification of the set of test cases selected for regression tests is necessary, because leaving outdated test cases gives us a misleading picture of the state of the application in the final reports, and the entire regression process is of no value. 

What do we do when testing regression?  

We check the expected result 

When performing regression tests, we check whether the system behaves as expected. If the regression tests are prepared in the form of test cases, then after completing the steps described in the case, we check whether we get the expected results. 

Verification using a checklist  

Another way to test regression is to prepare a checklist, which is a set of operations in the application that will be tested, saved in the form of short keywords. The checklist is relatively easy to prepare and maintain and is ideal when testers know the application very well and do not need detailed steps. They know what operations should be checked in a given area and what the expected result is. 

Unfortunately, in the case of a checklist, there is a risk that not all operations that should be checked will be tested. To avoid this, it is a good idea to update the checklist while the team discusses the changes to be made. Also, during the testing process of a given functionality, the missing elements should be filled in on an ongoing basis, so that the list is complete during regression. 

What makes a perfect checklist? 

A well-prepared checklist should be concise, and at the same time it should cover all the relevant functionalities. If the system is extensive and the checklist is long, it should be divided into appropriate categories that clearly define the area to be tested. It is also worth making sure that the tested operations are set in the right order, which will speed up the tests and mean that you will not have to go back to check things that have already been checked. Don’t forget that each checklist must be expanded along with the software development. 

Regression vs. retesting – what’s the difference? 

Many people, especially those new to IT, wonder if retests and regression tests are the same thing. 

• A retest is a single test that checks whether a defect has been removed. It means re-testing of a case in which the final result was different from the expected one and it was reported as a bug and fixed by the programmer. If the system works as expected after correction, we can close the request. However, if we still do not obtain the expected results, the request should still remain open, as it requires further analysis and repair. 
• Regression is a set of tests, that gives us the answer to whether the entire module or system is free of errors after making changes to the code. Regression tests are much more time-consuming than retests, but let’s also remember that their goal is different. 

How Do You Perform Regression Testing? Automation in regression 

How regression tests are performed depends mainly on how big the system is, and often on what budget we have as well. 

In small projects, when regression tests do not contain a large number of test cases, it is often sufficient to perform tests manually. But what if we have thousands of them and manual work would take months? Or if, during regression tests, there was an error blocking further work and you would have to check everything again after removing it? Then the best solution is to use automatic tests. Regression tests are perfect for automation, mostly because we perform them regularly and automatic tests definitely shorten the time required for executing test cases. If there are thousands of them, automation allows you to save time and effort. 

Read also: Test automation – common myths

Don’t forget about regression testing tools

Of course, the introduction of automatic tests involves costs related to, for example, the purchase of licenses for tools and hiring a person with the right competences. However, these costs in the case of large projects are still much lower than those incurred each time when performing manual regression. 

What if we can’t use test automation? 

Very often, a mixed approach is used in projects, because not every test case allows for automation. Often, complex test cases are troublesome and hard to automate, which is why they are most often performed manually. Of course, it is best if there are fewer cases to be performed manually, and the others are automated – then the regression process runs smoothly and does not take up so much time. 

The Advantages of Regression Testing

The regression testing process requires work and proper groundwork. Undoubtedly, it also implies costs related to time, workload or tools. However, if wisely planned and properly carried out, it brings benefits above and beyond the investment made. Regression testing prevents the introduction of defects to production, which may result in thousands of Euro in losses to the customer. If such a situation occurs, the team must deal with repairing defects, which means that further software development must wait, and is most often postponed. 

The regression test report gives us an answer to the question of whether a given software is of the expected quality and whether it can be implemented in the production. Also, it often indicates modules or areas of the software that are particularly vulnerable to defects, which gives the team the opportunity to thoroughly investigate the problem.   

Summary 

Regression tests are a very important stage in the software development process, regardless of the methodology of the project, whether it be a cascade or agile approach. They answer the question of what condition our application is in after introducing changes, and allow us to avoid the costs of repairing defects in production. As we already know, the costs of correcting such errors significantly exceed those incurred during the regression testing process. 

FAQ

Unraveling the concept of functional testing: what is it all about?

The goal of functional testing is to verify that the application is functioning as required in specific conditions. At various levels of testing, the tester checks whether the software works in accordance with the business requirements set by the stakeholders, and whether it meets users’ expectations.

Functional tests are one of the 4 traditional types of tests. They include functional tests, non-functional tests, structural tests and change-related tests. These tests check the activities that the program is designed to perform, or put simply, they check whether the program is doing what it should.

But to check this, we first need to know what the program should do. Traditionally, the answer to this question should be sought in the specification of requirements, documentation, and ultimately, we must rely on our experience.

Functional testing vs. non-functional testing

Functional testing and non-functional testing – are they two sides of the same coin? Not quite. Functional tests focus on the most basic requirements – whether functionalities work as intended.  Non-functional tests are also referred to as quality tests – here the matter is more complicated than in the case of functional tests. Non-functional testing checks parameters such as performance, usability, and reliability. In other words, functional testing ensures that the software system functions as expected, whereas non-functional testing explores how well the system performs.

Types of functional tests and the indispensable role of functional testing in software development

What you need to bear in mind at this stage is that test types and test levels are two separate concepts.

The test types determine their purpose. In this case, it is to check the behavior of the application and verify that it complies with the specifications and user requirements.

The test levels refer to the phases of software development. The basic divisions are unit, integration, system and acceptance tests.  In this article, we will not focus on developing this division further, but we must remember that functional tests should be performed at every available level of testing.

Unit testing & how does it function as a type of functional testing?

Functional tests at the level of unit tests are usually the realm of programmers, because they are usually responsible for creating and maintaining them. At this level, they check whether a single method / function does what it should, e.g., whether the appropriate action is triggered or whether the calculations are correct. This is the lowest of the test levels, as the tests themselves should be simple and performed very quickly. Therefore, this level of testing should be the foundation for automatic verification if the software works properly.

Also read: Test Pyramid

Integration tests

Integration tests, on the other hand, can be the realm of both testers and developers. The basic factor here will be the level at which we check the integration. If we verify the integration of individual methods and functions (after checking with unit tests if they work on their own), it will tend to be the domain of programmers.

However, if we test integration at a higher level, e.g., integration between two application modules, this work can (and often should) be carried out by a software tester. This is due to the fact that a developer testing his own code tests his interpretation of the requirements, but not necessarily the correct one. This is not a shot at programmers, but rather an explanation of why you should not test your own solutions.

System tests

Most people consider functional tests at the system level as the default type of tests. System testing means checking software functionality in an integrated environment, where functionalities can be tested end-to-end. It is also the most intuitive level for beginner testers or people who don’t deal with testing.

But that’s not all – since it is end-to-end testing, it is also very important from the perspective of business and stakeholders, because end users will use the software in a very similar way. Often only at this point can we pay attention to the accuracy of the user interface (both compatibility with mockups and the simplicity of use) and the issue of presenting data to the user. At this level, we can finally check the accessibility of the product for people with disabilities, which is especially important for testers working in public institutions.

Acceptance testing

Acceptance testing is the last stage of testing, which is usually performed on the finished product, by end users, or perhaps by or with the contribution of a client. Functional tests are carried out at this level to finally confirm and approve that the resulting product meets the requirements. It is also the last chance to find defects before handing the product over to end users and the final call to check the compliance of the product with the legal requirements. What is a test pyramid? What, when and how to test?

Read also: User Acceptance Testing (UAT) – Why is it important

How do we perform functional tests?

We have discussed functional tests at different levels, but let’s answer the question of how to perform them. As is often the case with testing, the best answer is that  it depends. Tests at all levels, except for unit tests, can be performed manually more or less successfully.

In the case of integration tests, we will often need additional software to help, but this does not change the fact that well-designed tests can be performed manually.

Test Automation and functional testing: what’s the connection?

We can cover all the previously mentioned test levels with automatic tests. This allows for effective regression testing and also builds trust in the product, as it is regularly checked for correct operation (automatic tests are not for finding errors, but to make sure that functionalities that have already worked keep on doing so).

When to automate your functional tests

The most important business paths during acceptance tests are worth checking manually so we can rest assured that the product is ready to be made available to users. It is worth utilizing automated tests during acceptance testing to limit repetitive work (e.g., redirects with multiple language versions, translations, etc.).

White-box testing or black-box testing techniques?

We already know what functional tests are, what their purpose is, what levels we should perform them at and who can perform such tests. It’s time to answer the question of  what the right way to determine how to test our functional features is.

Tests based on documentation

As I mentioned, functional tests check the correctness of the software in terms of customer requirements, but also in terms of compliance with user requirements.

Compliance with customer requirements

Let’s focus on meeting customer requirements at the beginning. These requirements are usually included in the specification, mock-ups and other forms of documentation. Designing tests based on documentation means the use of black box techniques in practice. Beginner testers often do not even know that they use the popular black box techniques described above when designing tests. This is because the basic techniques are simple and intuitive.

Meeting user expectations

The second part of functional tests is to check whether they meet user expectations. Here the experience of a tester is vital. By that, I mean both testing and industry experience. In these tests, we rely on techniques based on experience – guessing errors, exploratory tests or using checklists.

To sum up, testers have a range of different techniques, some of which require intuition, and others experience. The common denominator for them is the result: well-designed tests which allow us to find defects or raise the level of confidence in the quality of the software.

Is it worth taking an interest in functional software testing?

As part of the functional testing of the application, testers are responsible for testing functions at different levels. The techniques used to find errors create significant opportunities for development and specialization.

Essential Takeaways: What Should You Remember About Functional Testing?

As we conclude this comprehensive guide, let’s recap the key points:

• Functional testing is pivotal to verify that a software system operates correctly.
• There’s a clear distinction between functional and non-functional testing.
• Numerous types of functional testing exist, each with its unique purpose and application.
• A variety of functional testing tools are available to streamline and automate testing.
• The future of functional testing lies in increased automation and the use of AI-based techniques.

Java and Python  

Each of them is well-established, platform-independent, object-oriented and part of a large, supportive community. When it comes to selecting Selenium language bindings (read Java or Python here), you should first understand some key differences between them. 

Selenium supports a wide range of programming languages for test automation. Java and Python are the top choices for binding with Selenium in this context. 

A programming language or a scripting language? 

One of the main questions that could come to mind when comparing Java and Python could be: is Python/Java a programming language or a scripting language? 

Basically, all scripting languages are considered to be programming languages. The main difference between them is that programming languages are compiled, whereas scripting languages are interpreted. Scripting languages are slower than programming languages and usually sit behind them. 

You can find a brief comparison in the link below: Interpreter vs compiler  

Python 

Python can be called a scripting language as well as a programming language since it works both as a compiler and an interpreter. Standard Python can compile Python code into bytecodes and then interpret it just like Java and C. 

“However, considering the historical relationship between the general-purpose programming language and the scripting language, it would be more appropriate to say that Python is a programming language that works nicely as a scripting language too”. 

An interpreter is a layer of software that works as a bridge between the program and the system hardware to keep the code running. The Python interpreter is an application that is responsible for running Python scripts. 

Java 

Java is generally faster and more efficient than Python because it is a compiled language. As an interpreted language, Python has simpler, more concise syntax than Java. It can perform the same function as Java in fewer lines of code. In Java, the data types of variables should be declared before using them, unlike for Python. You can then decide what appeals more to you here. 

What is easier to learn: Java or Python?  

Another question you may have, especially when you are new to programming, is: how quickly can you learn Java vs Python? 

And there is no doubt here that Python is the winner. As mentioned above, its syntax is shorter and easier to digest. Java has more complex prototyping; for example, it burdens your code with additional declarations and restrictions. 

Choose Python then if you have a short ramp-up period in which to learn. 

The list of key factors to take into consideration before making a decision: 

• Job opportunities: still more jobs for Java. 
• Job changes / migrations: thanks to scripting, it’s easier for Python if you switch to, for example, DevOps. 
• Object oriented: both Python and Java rely on object programming so if you know one of them, you have a good starting point for the other. 
• Platform independent: both Python and Java binaries (bytecodes), once generated, can be executed on any supported platforms thanks to their own virtual machines (PVM and JVM respectively). 
• Simplicity: Python is easier to use thanks to its simpler syntax.  
• Execution: as a scripting language, Python has the ability to run your script code directly using its interpreter instead of directly running a compiler (like Javac for Java). 
• Performance / execution speed: Java has the advantage here as it is faster (can be JIT-compiled to be more efficient). 
• Popularity / support: Java wins out here, as solutions based on Java are more popular and it has a very large supporting community. 
• Python is a statically typed language (performs type checking at runtime, meaning you can have a code that fails at the very last moment of execution). 
• Java is a dynamically typed language (performs type checking at compile time, meaning that you catch problems in your code earlier in the development process). 
• Extendibility: since Java is more popular, more plugins, APIs, frameworks and libraries are available on the market than for Python. 

Test automation – what options do you have? 

Selenium satisfies a specific need in the test automation world. It is only for browsers. If you want to quickly start automating your test cases, Selenium IDE is your friend here, as it allows you to rapidly start automating tests.  

The test steps you perform in a browser are directly transformed into a set of commands thanks to Selenese, which is Selenium IDE’s built-in language. It is very simple. Moreover, after storing your tests in a set of Selenese commands, you can aggregate them into test suites and finally export to your preferred Selenium bindings (e.g., Java, Python, etc.). 

Selenese and Selenium IDE is a good start, but the more you dig into the subject of web application automation, the more you see its constraints. 

To properly test a modern web application with its dynamically changed content (e.g., using AJAX or JS technologies) you are going to need to start looking for some more advanced options. That’s where Selenium WebDriver comes in, along with its Selenium-specific language bindings. Selenium WebDriver is often used as a basis for testing web applications.  

According to the A4Q Selenium 4 Tester syllabus: 

„Selenium, when used for testing web applications, needs some testing frameworks. The Selenium engine, as a standalone tool, only exposes its native APIs as methods or functions to the TAE. The TAE must then program a testing framework to use these APIs to carry out the tests required. A testing framework can be conceived from scratch or existing frameworks could be used. The Selenium project supports multiple testing frameworks in diverse languages” such as:  

• for Java: Selenide  
• for Python: Selene  

Whether you use your own testing framework or not, you need to drive your test execution somehow. Many TAEs use some unit test libraries to drive their tests (test runners). 

Some of them like TestNG or Pytest allow you to create higher-level functional tests apart from its primary unit testing use (functional, component, integration, end-to-end etc.). 

Read also: User Acceptance Testing (UAT) – Why is it important?

For Python 

The simplest case is to use Python’s standard unit test library, or alternatively you can use the highly popular Pytest.  

For Java 

You may use xUnit libraries (e.g., JUnit 4/5) or TestNG.  

Once you are familiar with running tests locally using your chosen browser, you may take a step forward and set up your grid of browsers to run tests remotely. 

This way, your tests can be run on different browsers and multiple operating systems across a range of machines in parallel mode. If your configuration is large, you could run your regression testing in a fraction of the time it would take without Selenium Grid. 

Later, you can go even further and integrate your tests into the CI/CD pipeline (e.g., running Selenium Tests in GitLab CI/CD or Jenkins). 

Please note that you could take advantage of containerization techniques like Docker to create various images for automated UI tests. 

By doing this, you could prepare a separate VM for any combination of your OS / browser environment for your SUT (System Under Test) / AUT (Application Under Test). 

CI/CD has been pivotal in setting up software development lifecycles, where new features are constantly updated at maximum levels of speed and efficiency. This allows end-users to access new features without the need to download software updates, offering them a seamless experience.  

Given the short span of the release cycle, testing has to be scaled to deliver bug-free software in the allocated timeframe. This is where Selenium plays a pivotal role by providing test automation in the CI/CD pipeline. 

Thanks to Selenium automation testing, parallel tests can be performed at the required scale, while Docker helps maintain the sanity of the test environment with its ephemeral containers. 

What is Selenium WebDriver? 

The most comprehensive source of knowledge to help answer this question can be found on the Selenium WebDriver page:  

„Selenium WebDriver refers to both the language bindings and the implementations of the individual browser controlling code. This is commonly referred to as just WebDriver.” 

or directly from the A4Q syllabus:  

„Selenium WebDriver, as the name suggests, interacts with the browser in the same way that a real user would. WebDriver is an end-to-end test compilation tool for web applications that is also frequently referred to as „Selenium WebDriver”. 

From a Selenium architectural point of view, all the components are visualized in the official documentation page. 

We can say that WebDriver helps us to mimic user / browser interactions; hence, it is extremely helpful in terms of test automation. WebDriver controls a browser using browser specific drivers which are usually created by their vendors (e.g., Google for Chrome or Mozilla for Firefox). 

All the communication is 2-way and is compliant with the following protocols: 

• for Selenium 3: JSON Wire Protocol over HTTP (translated communication) 
• for Selenium 4: W3C WebDriver Protocol (native communication). See more on the W3C site 

Thanks to W3C Protocol introduced in Selenium 4, we have a direct transfer of information between client and server. So, if you are still using Selenium 3 (WebDriver) and are hesitant to upgrade, consider checking the following page: Upgrade to Selenium 4. 

The minimum versions required are Java version 8 and Python version 3.7. 

Selenium and Python 

If you decide to go this way but don’t yet have the requisite knowledge, then try to play with Selenium IDE and generate your Python code using its export function.  

Once you are more familiar with Selenium IDE, you can then prepare your full Selenium / Python test automation development environment. 

What you need: 

• IDE (PyCharm, Visual Studio Code or Aqua) 
• Python 
• PIP (Package Installer for Python) 
• Build tool (Pipenv, PIP + Venv/Virtualenv, Poetry or Bazel) 
• Selenium Python bindings 
• Selenium browser drivers (Chrome, Firefox, Safari, Opera, Edge or webdriver-manager/SeleniumManager*)
• Test Runner (Unittest or Pytest) 

*Selenium Manager is available starting from version 4.6 (Beta) 

Selenium and Java 

If you decided to go this way but don’t yet have the requisite knowledge, then try to play with the Selenium IDE plugin for Chrome or Firefox and generate your Java code using its export function. 

Once you are more familiar with it, you can then prepare your full Selenium / Java test automation development environment. 

What you need: 

• IDE (IDEA, Visual Studio Code or Aqua) 
• Java (JDK) 
• Build tool (Maven or Gradle) 
• Selenium Java bindings 
• Selenium Browser Drivers (Chrome, Firefox, Safari, Opera, Edge or WebDriverManager/SeleniumManager*) 
• Test Runner (JUNIT 4/5 or TestNG)  

*Selenium Manager is available starting from version 4.6 (Beta) 

What is a better option for automated tests – Selenium with Python or Selenium with Java?  

If you are a complete beginner when it comes to test automation with Selenium and maybe to programming too (whether it is Java or Python), I suggest that you start by using Selenium IDE, which is provided in the form of an add-on for Chrome, Firefox and Edge (version 3.x). It allows you to generate your Java / Python code using its export function. It is very straightforward, as you simply record your actions in a browser. Code is generated for you afterwards by exporting to your favorite language function. At the time of writing (June 2023), it was even possible to download a standalone version (4 alpha) of Selenium IDE for several platforms (e.g., Windows, MacOS). Find out more about Selenium’s latest releases. 

From the point of view of automation, and especially CI/CD integration, it looks like a combination of Selenium and Java is better suited here. It blends in well with this process thanks to very good integration with tools like Jenkins/Maven/Gradle. 

This doesn’t mean that you can’t achieve this with Python, albeit it would be more tiresome and some plugins would be necessary. But once again, Python provides simple coding and readability, so if you don’t have a background in Java/Python programming, you may be tempted to use Python. 

One of the most important points in the Selenium test automation development process is related to test debugging. Many browsers provide “DevTools” – a set of tools that are integrated with the browser that developers can use to debug web apps and explore the performance of their pages. 

The first thing that comes to mind in any case is Chrome DevTools based on Chrome DevTools Protocol (CDP): 

• https://developer.chrome.com/docs/devtools/ 
• https://www.selenium.dev/documentation/webdriver/bidirectional/chrome_devtools/ 
• https://www.selenium.dev/selenium/docs/api/Java/org/openqa/selenium/devtools/package-summary.html 
• https://chromedevtools.github.io/devtools-protocol/ 

which is probably most commonly used and integrated with Selenium 4 itself. It helps you inspect, test and verify your selectors. As selectors are generic and language independent, it has universal usage. Another thing that is useful for debugging your Selenium test code is breakpoints in your favorite IDE. Set it anywhere you seem to have a problem and your test stops in the browser at that moment. Now you can continue debugging using your browser’s DevTools such as Chrome DevTools. 

This works both for Java (e.g., IDEA) and Python (e.g., PyCharm). But there is one thing for Python that is very interesting for debugging too. It is called PDB (The Python Debugger) and it is one of the best debugging techniques for Selenium. 

• https://docs.python.org/3/library/pdb.html 
• https://realpython.com/python-debugging-pdb/ 
• https://visualmodo.com/6-simple-tips-for-debugging-selenium-python-code/ 
• https://seleniumbase.com/the-ultimate-pytest-debugging-guide-2021/ 
• https://www.codementor.io/@stevek/advanced-python-debugging-with-pdb-g56gvmpfa 

This Python module gives you an interactive source code debugging option. To see how useful it could be, look at the example below: 

C:\Users\user> py 
Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32 
Type "help", "copyright", "credits" or "license" for more information. 
>>> from selenium import webdriver 
>>> from selenium.webdriver.common.by import By 
>>> import pdb 
>>> if __name__ == '__main__': 
...   print('Hello World!') 
...   browser = webdriver.Chrome() 
...   browser.get('http://selenium.dev/') 
...   web_driver_selector = "div.selenium-button-container a.selenium-webdriver" 
...   web_driver_link = browser.find_element(By.CSS_SELECTOR, web_driver_selector) 
...   pdb.set_trace() 
...   browser.quit() 
... 
Hello World! 

and now when your browser is stopped at the breakpoint set on the http://selenium.dev/ page, you can check the web_driver_link: 

(Pdb) dir 
<built-in function dir> 
(Pdb) dir(web_driver_link) 
['__abstractmethods__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_execute', '_id', '_parent', '_upload', 'accessible_name', 'aria_role', 'clear', 'click', 'find_element', 'find_elements', 'get_attribute', 'get_dom_attribute', 'get_property', 'id', 'is_displayed', 'is_enabled', 'is_selected', 'location', 'location_once_scrolled_into_view', 'parent', 'rect', 'screenshot', 'screenshot_as_base64', 'screenshot_as_png', 'send_keys', 'shadow_root', 'size', 'submit', 'tag_name', 'text', 'value_of_css_property'] 

As you can see above, pdb gives you the opportunity to inspect your object and what you can do with it using dir command, for example: 

(Pdb) web_driver_link.is_displayed() 
True 
(Pdb) web_driver_link.is_enabled() 
True 
(Pdb) web_driver_link.is_selected() 
False 
(Pdb) web_driver_link.text 
'READ MORE' 
(Pdb) web_driver_link.get_property('href') 
'https://www.selenium.dev/documentation/webdriver/' 
(Pdb) web_driver_link.get_dom_attribute('class') 
'selenium-button selenium-webdriver text-uppercase font-weight-bold' 
(Pdb) web_driver_link.get_attribute('class') 
'selenium-button selenium-webdriver text-uppercase font-weight-bold' 
(Pdb) web_driver_link.get_dom_attribute 
<bound method WebElement.get_dom_attribute of <selenium.webdriver.remote.webelement.WebElement (session="b51ad53a899388ec9293d83fd0750a98", element="E97FE72B561D59DFF64B6C8846C4CC17_element_31")>> 
(Pdb) web_driver_link.id 
'E97FE72B561D59DFF64B6C8846C4CC17_element_31' 
(Pdb) web_driver_link.__class__ 
<class 'selenium.webdriver.remote.webelement.WebElement'> 
(Pdb) web_driver_link.value_of_css_property 
<bound method WebElement.value_of_css_property of <selenium.webdriver.remote.webelement.WebElement (session="b51ad53a899388ec9293d83fd0750a98", element="E97FE72B561D59DFF64B6C8846C4CC17_element_31")>> 
(Pdb) web_driver_link.value_of_css_property('color') 
'rgba(200, 16, 46, 1)' 
(Pdb) 

This could also be one of the arguments in Python’s favor. You may even extend it to include an ipdb add-on that improves the experience of the pdb debugger:  

ipdb Debug Mode adds color-coded syntax highlighting, tab-completion, and an expanded view of the code with line numbers. ipdb must be installed separately from pytest before use: 

pip install ipdb  

What is Selenium with Python for? 

This blend of technologies is meant for testing front end web applications. It helps to leverage your tedious and repeatable GUI actions or tests and replace them with automated scripts. By doing so, you can integrate them further with CI/CD tools for nightly regression testing, for example. If I were to answer this in only one sentence I would say: 

Selenium + Python is basically for web application test automation. 

Selenium (WebDriver) is responsible for 2-way client-server communication between you and the browser. Python and its Python Selenium bindings wrap everything in executable code driven by testing frameworks like xUnit / TestNG. 

Is Selenium with Python a good choice? 

I would say that the answer to this question depends on many factors. Each of us is different and has our own experience, skills and ability to learn. The final answer to this question will be specific to you. The aim of this article is to help you find it. Read the article and analyze it along with your professional objectives, your past, present and future, as it could indeed be your best choice. 

How to execute Selenium script in Python? 

In Python, we can distinguish 2 main types of Python files: 

• .py files are human readable text files with Python code inside, called Python scripts (on Windows systems the extension can also be .pyw). 
• .pyc files are compiled bytecode files that are generated by the Python interpreter when a Python script is imported or executed. 

Python files are saved on the user machine with the file extension “.py” as Java files are saved with .java extensions. Python codes are then converted to bytecodes after being interpreted. These bytecodes are saved in files with the extension “.pyc”.  

The “__pycache__” directory is the folder created to store the “.pyc” files. 

• https://prepinsta.com/python/difference-between-py-pyc-and-pycache-files/ 
• https://stackabuse.com/differences-between-pyc-pyd-and-pyo-python-files/ 
• https://net-informations.com/python/iq/pyc.htm 

Think of a .pyc file as a compiled bytecode, similar to a .class file in the JAVA world. 

As soon as you generate it, it can be directly executed by the interpreter without recompiling a .py script file each time you use it. By doing so, you speed up the execution, which is important when you have large python scripts and/or modules. 

To sum up: 

• .pyc files are generated automatically by the Python interpreter when a Python script is imported or executed. 

• .pyc files are stored in the same directory as the corresponding .py files, and have the same name as the .py files, except with a .pyc extension instead of .py. 
• .pyc files are specific to the version of Python that was used to generate them. If you try to run a .pyc file with a different version of Python, you may encounter errors or unexpected behavior. 

If you want to run a Python script, you have 2 options: 

1.  run Python Script with the Python Interpreter using Interactive mode (in the form of a piece of code written in an interactive session – useful for debugging). 

python 

e.g., typing your code line by line: 
C:\Users\user> py 
Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32 
Type "help", "copyright", "credits" or "license" for more information. 
>>> from selenium import webdriver 
>>> if __name__ == '__main__': 
...   print('Hello World!') 
...   browser = webdriver.Chrome() 
...   browser.get('http://selenium.dev/') 
...   browser.quit() 
... 
Hello World! 
  
DevTools listening on ws://127.0.0.1:56302/devtools/browser/fdca3cc9-4266-49af-a146-41d60ed464eb 
>>> exit() 
  
C:\Users\user> 

or using exec() which provides an alternative way of running your scripts: 

C:\Users\user> py 
Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32 
Type "help", "copyright", "credits" or "license" for more information. 
>>> exec(open('your_script.py').read()) 
Hello World! 
>>> 
>>> quit() 
  
C:\Users\user> 

A Python interactive mode session allows you to write many lines of code, but once you close the session, you lose everything you’ve written.  

That’s why the usual way of writing Python programs is by using plain text files. The file with the Python code must be located in your current working directory. The file must be in the Python Module Search Path (PMSP), where Python looks for the modules and packages you import. 

2) run Python Script using Command-Line (in the form of a script or module – useful for automation) 

python your_script.py 
  
e.g.: 
C:\Users\user> py your_script.py 
Hello World! 

or even more simply, by using Windows file associations and providing just a script name: 

C:\Users\user> your_script.py 
Hello World! 

In GNU/Linux systems you can achieve a similar outcome by adding a so-called hashbang or shebang to your script, which is the character combination #! followed by the interpreter path. 

There are two ways to specify the path to the interpreter: 

• #!/usr/bin/python: writing the absolute path. 
• #!/usr/bin/env python: using the operating system env command, which locates and executes Python by searching the PATH environment variable. 

As not all Unix-like systems locate the Python interpreter at the same location, the second flavor of path given above looks more universal. 

You need to add execution permission as well: 

#assign the execution permissions 
chmod +x your_script.py 
#run script using its filename 
./your_script.py 
Hello World! 

You can run your bytecode-generated script directly as well: 

C:\Users\user> dir your_script.py 
Volume in drive C is Windows 
Volume Serial Number is 94D2-9546 
  
Directory of C:\Users\user 
  
15-Jun-23 14:02        544 your_script.py 
        1 File(s)      544 bytes 
        0 Dir(s) 194,664,701,952 bytes free 
  
C:\Users\user> cd __pycache__ 
  
C:\Users\user\__pycache__> dir your_script.cpython-311.pyc 
Volume in drive C is Windows 
Volume Serial Number is 94D2-9546 
  
Directory of C:\Users\user\__pycache__ 
  
15-Jun-23 14:06        401 your_script.cpython-311.pyc 
        1 File(s)      401 bytes 
        0 Dir(s) 194,664,546,304 bytes free 
  
C:\Users\user\__pycache__> your_script.cpython-311.pyc 
Hello World! 
  
C:\Users\user\__pycache__> 

How to install Selenium with PIP? 

• install Python (manually or using IDE) 

Go to Python.org and get the suitable version. 

py –version

Python 3.N.N

e.g. for WIN (CMD):

C:\Users\user> py --version 
Python 3.11.4 
  
C:\Users\user> 

• install or upgrade PIP (manually or using IDE) 

What is PIP? 

PIP is the package installer for Python. You can use it to install packages from the Python Package Index and other indexes. The most popular tool for installing Python packages, and the one included with modern versions of Python.  

It provides the essential core features for finding, downloading, and installing packages from other Python package indexes, and can be incorporated into a wide range of development workflows via its command-line interface (CLI). 

py -m pip –version

pip X.Y.Z from … (python 3.N.N)

e.g., for WIN (CMD): 

C:\Users\user> py -m pip --version 
pip 23.1.2 from C:\Program Files\Python311\Lib\site-packages\pip (python 3.11) 
  
C:\Users\user> 

e.g., for WIN (IDE): 

(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> py -m pip list 
 
 
setuptools 65.5.1 
wheel   0.38.4 
  
[notice] A new release of pip available: 22.3.1 -> 23.1.2 
[notice] To update, run: python.exe -m pip install --upgrade pip 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> py -m pip --version 
pip 22.3.1 from C:\Users\user\PycharmProjects\pythonSelenium\venv\Lib\site-packages\pip (python 3.11) 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> python.exe -m pip install --upgrade pip 
Requirement already satisfied: pip in c:\users\user\pycharmprojects\pythonselenium\venv\lib\site-packages (22.3.1) 
Collecting pip 
 Downloading pip-23.1.2-py3-none-any.whl (2.1 MB) 
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 2.7 MB/s eta 0:00:00 
Installing collected packages: pip 
 Attempting uninstall: pip 
  Found existing installation: pip 22.3.1 
  Uninstalling pip-22.3.1: 
   Successfully uninstalled pip-22.3.1 
Successfully installed pip-23.1.2 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium>  
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> py -m pip --version 
pip 23.1.2 from C:\Users\user\PycharmProjects\pythonSelenium\venv\Lib\site-packages\pip (python 3.11) 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium>  
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> py -m pip list 
Package  Version 
---------- ------- 
pip    23.1.2 
setuptools 65.5.1 
wheel   0.38.4 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium>  

If that worked, congratulations! You have a working PIP in your environment. 

•  install Selenium Python bindings 
   

Verify the supported Python version (as of Selenium 4.10.0): 

pip install selenium 
pip install -U selenium 

If you have PIP on your system, you can simply install or upgrade the Python bindings: 

pip install selenium==<version_number> 

If you want the specific Selenium python bindings version: 

pip install selenium==<version_number> 

Alternately, you can download the source distribution from PyPI (e.g. selenium-4.10.0.tar.gz), unarchive it, and run: 

py setup.py install

e.g. for WIN (CMD):

C:\Users\user> py -m pip list 
Package  Version 
---------- ------- 
pip    23.1.2 
setuptools 65.5.0 
  
C:\Users\user> 
C:\Users\user> py -m pip install selenium 
Defaulting to user installation because normal site-packages is not writeable 
Collecting selenium 
 Downloading selenium-4.10.0-py3-none-any.whl (6.7 MB) 
   ---------------------------------------- 6.7/6.7 MB 23.7 MB/s eta 0:00:00 
Collecting urllib3[socks]<3,>=1.26 (from selenium) 
 Downloading urllib3-2.0.3-py3-none-any.whl (123 kB) 
   ---------------------------------------- 123.6/123.6 kB ? eta 0:00:00 
Collecting trio~=0.17 (from selenium) 
 Downloading trio-0.22.0-py3-none-any.whl (384 kB) 
   ---------------------------------------- 384.9/384.9 kB 25.0 MB/s eta 0:00:00 
Collecting trio-websocket~=0.9 (from selenium) 
 Downloading trio_websocket-0.10.3-py3-none-any.whl (17 kB) 
Collecting certifi>=2021.10.8 (from selenium) 
 Downloading certifi-2023.5.7-py3-none-any.whl (156 kB) 
   ---------------------------------------- 157.0/157.0 kB 9.2 MB/s eta 0:00:00 
Collecting attrs>=19.2.0 (from trio~=0.17->selenium) 
 Downloading attrs-23.1.0-py3-none-any.whl (61 kB) 
   ---------------------------------------- 61.2/61.2 kB 3.2 MB/s eta 0:00:00 
Collecting sortedcontainers (from trio~=0.17->selenium) 
 Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) 
Collecting async-generator>=1.9 (from trio~=0.17->selenium) 
 Downloading async_generator-1.10-py3-none-any.whl (18 kB) 
Collecting idna (from trio~=0.17->selenium) 
 Downloading idna-3.4-py3-none-any.whl (61 kB) 
   ---------------------------------------- 61.5/61.5 kB 3.2 MB/s eta 0:00:00 
Collecting outcome (from trio~=0.17->selenium) 
 Downloading outcome-1.2.0-py2.py3-none-any.whl (9.7 kB) 
Collecting sniffio (from trio~=0.17->selenium) 
 Downloading sniffio-1.3.0-py3-none-any.whl (10 kB) 
Collecting cffi>=1.14 (from trio~=0.17->selenium) 
 Downloading cffi-1.15.1-cp311-cp311-win_amd64.whl (179 kB) 
   ---------------------------------------- 179.0/179.0 kB ? eta 0:00:00 
Collecting exceptiongroup (from trio-websocket~=0.9->selenium) 
 Downloading exceptiongroup-1.1.1-py3-none-any.whl (14 kB) 
Collecting wsproto>=0.14 (from trio-websocket~=0.9->selenium) 
 Downloading wsproto-1.2.0-py3-none-any.whl (24 kB) 
Collecting pysocks!=1.5.7,<2.0,>=1.5.6 (from urllib3[socks]<3,>=1.26->selenium) 
 Downloading PySocks-1.7.1-py3-none-any.whl (16 kB) 
Collecting pycparser (from cffi>=1.14->trio~=0.17->selenium) 
 Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) 
   ---------------------------------------- 118.7/118.7 kB ? eta 0:00:00 
Collecting h11<1,>=0.9.0 (from wsproto>=0.14->trio-websocket~=0.9->selenium) 
 Downloading h11-0.14.0-py3-none-any.whl (58 kB) 
   ---------------------------------------- 58.3/58.3 kB ? eta 0:00:00 
Installing collected packages: sortedcontainers, urllib3, sniffio, pysocks, pycparser, idna, h11, exceptiongroup, certifi, attrs, async-generator, wsproto, outcome, cffi, trio, trio-websocket, selenium 
Successfully installed async-generator-1.10 attrs-23.1.0 certifi-2023.5.7 cffi-1.15.1 exceptiongroup-1.1.1 h11-0.14.0 idna-3.4 outcome-1.2.0 pycparser-2.21 pysocks-1.7.1 selenium-4.10.0 sniffio-1.3.0 sortedcontainers-2.4.0 trio-0.22.0 trio-websocket-0.10.3 urllib3-2.0.3 wsproto-1.2.0 
  
C:\Users\user> 
C:\Users\user> py -m pip freeze 
async-generator==1.10 
attrs==23.1.0 
certifi==2023.5.7 
cffi==1.15.1 
exceptiongroup==1.1.1 
h11==0.14.0 
idna==3.4 
outcome==1.2.0 
pycparser==2.21 
PySocks==1.7.1 
selenium==4.10.0 
sniffio==1.3.0 
sortedcontainers==2.4.0 
trio==0.22.0 
trio-websocket==0.10.3 
urllib3==2.0.3 
wsproto==1.2.0 
  
C:\Users\user> 
C:\Users\user> py -m pip show selenium 
Name: selenium 
Version: 4.10.0 
Summary: 
Home-page: https://www.selenium.dev 
Author: 
Author-email: 
License: Apache 2.0 
Location: C:\Users\user\AppData\Roaming\Python\Python311\site-packages 
Requires: certifi, trio, trio-websocket, urllib3 
Required-by: 
  
C:\Users\user> 
  
  
or for WIN (IDE): 
  
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> py -m pip install selenium 
Collecting selenium 
 Using cached selenium-4.10.0-py3-none-any.whl (6.7 MB) 
Collecting urllib3[socks]<3,>=1.26 (from selenium) 
 Using cached urllib3-2.0.3-py3-none-any.whl (123 kB) 
Collecting trio~=0.17 (from selenium) 
 Using cached trio-0.22.0-py3-none-any.whl (384 kB) 
Collecting trio-websocket~=0.9 (from selenium) 
 Using cached trio_websocket-0.10.3-py3-none-any.whl (17 kB) 
Collecting certifi>=2021.10.8 (from selenium) 
 Using cached certifi-2023.5.7-py3-none-any.whl (156 kB) 
Collecting attrs>=19.2.0 (from trio~=0.17->selenium) 
 Using cached attrs-23.1.0-py3-none-any.whl (61 kB) 
Collecting sortedcontainers (from trio~=0.17->selenium) 
 Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) 
Collecting async-generator>=1.9 (from trio~=0.17->selenium) 
 Using cached async_generator-1.10-py3-none-any.whl (18 kB) 
Collecting idna (from trio~=0.17->selenium) 
 Using cached idna-3.4-py3-none-any.whl (61 kB) 
Collecting outcome (from trio~=0.17->selenium) 
 Using cached outcome-1.2.0-py2.py3-none-any.whl (9.7 kB) 
Collecting sniffio (from trio~=0.17->selenium) 
 Using cached sniffio-1.3.0-py3-none-any.whl (10 kB) 
Collecting cffi>=1.14 (from trio~=0.17->selenium) 
 Using cached cffi-1.15.1-cp311-cp311-win_amd64.whl (179 kB) 
Collecting exceptiongroup (from trio-websocket~=0.9->selenium) 
 Using cached exceptiongroup-1.1.1-py3-none-any.whl (14 kB) 
Collecting wsproto>=0.14 (from trio-websocket~=0.9->selenium) 
 Using cached wsproto-1.2.0-py3-none-any.whl (24 kB) 
Collecting pysocks!=1.5.7,<2.0,>=1.5.6 (from urllib3[socks]<3,>=1.26->selenium) 
 Using cached PySocks-1.7.1-py3-none-any.whl (16 kB) 
Collecting pycparser (from cffi>=1.14->trio~=0.17->selenium) 
 Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB) 
Collecting h11<1,>=0.9.0 (from wsproto>=0.14->trio-websocket~=0.9->selenium) 
 Using cached h11-0.14.0-py3-none-any.whl (58 kB) 
Installing collected packages: sortedcontainers, urllib3, sniffio, pysocks, pycparser, idna, h11, exceptiongroup, certifi, attrs, async-generator, wsproto, outco 
me, cffi, trio, trio-websocket, selenium 
Successfully installed async-generator-1.10 attrs-23.1.0 certifi-2023.5.7 cffi-1.15.1 exceptiongroup-1.1.1 h11-0.14.0 idna-3.4 outcome-1.2.0 pycparser-2.21 pysoc 
ks-1.7.1 selenium-4.10.0 sniffio-1.3.0 sortedcontainers-2.4.0 trio-0.22.0 trio-websocket-0.10.3 urllib3-2.0.3 wsproto-1.2.0 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium>  
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> py -m pip list       
Package     Version 
---------------- -------- 
async-generator 1.10 
attrs      23.1.0 
certifi     2023.5.7 
cffi       1.15.1 
exceptiongroup  1.1.1 
h11       0.14.0 
idna       3.4 
outcome     1.2.0 
pip       23.1.2 
pycparser    2.21 
PySocks     1.7.1 
selenium     4.10.0 
sniffio     1.3.0 
sortedcontainers 2.4.0 
trio       0.22.0 
trio-websocket  0.10.3 
urllib3     2.0.3 
wheel      0.38.4 
wsproto     1.2.0 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium> py -m pip show selenium 
Name: selenium 
Version: 4.10.0 
Summary: 
Home-page: https://www.selenium.dev 
Author: 
Author-email: 
License: Apache 2.0 
Location: C:\Users\user\PycharmProjects\pythonSelenium\venv\Lib\site-packages 
Requires: certifi, trio, trio-websocket, urllib3 
Required-by: 
(venv) PS C:\Users\user\PycharmProjects\pythonSelenium>  

Test automation – summary  

Selenium supports a number of programming languages for test automation. Among developers, both Java and Python have achieved record-breaking popularity. Python is easy to learn, and its syntax is simple to read, so if you are new to programming, using Selenium with Python might sound interesting in terms of automation. On the other hand, Java and Selenium are an excellent fit for the entire software development process, especially in the context of CI/CD integration, thanks to the use of the appropriate build tools. The answer to the question of which to use, Java or Python, is up to you, depending on your professional background and objectives for the future.  

Elevate Your Application Development

Our tailored Application Development services meet your unique business needs. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

FAQ – starting points for Selenium with Java or Python 

Here is your starter pack for test automation with Selenium + Java or Selenium + Python, no matter which of them you decide to use. 

1. 1. What are Acceptance Tests (UAT)?
2. 2. Purpose of UAT (User Acceptance Tests), according to ISTQB 
3. 3. The Importance of UAT
4. 4. The example of UAT testing process
5. 5. Types of UAT tests
6. 6. UAT – verification or validation?
7. 7. To sum up: Why should I conduct User Acceptance Tests?

What are Acceptance Tests (UAT)? 

Let’s start with what acceptance testing is not. The goal of UATs is not to get something on programmers who have developed software, or on QAs who have run software tests. Acceptance testing (UAT) is part of the software development process (see our article on software testing for more details). At the UAT stage, the product is ready for implementation, and we have to confirm that all requirements have been met.   

Purpose of UAT (User Acceptance Tests), according to ISTQB  

As per ISTQB, acceptance tests should check:  

• end-user requirements  
• system requirements 
• use cases 
• business processes 

Read also: The test pyramid – what, when and how to test software to get profits

The Importance of UAT  

The customer or users of the system are most often responsible for conducting acceptance tests. The product to be tested by acceptance tests has already been thoroughly checked by the previous system tests. Acceptance tests are run to confirm that the new version meets the customer’s requirements and help him learn to trust it more.   

Elevate Your Application Development

Our tailored Application Development services meet your unique business needs. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

The example of UAT testing process   

• Initiation: 
  • Defining the conditions of acceptance  
  • Defining the team responsible for carrying out the tests  
  • Preparing the environment and the appropriate version of the software 
  • Preparing suitable test data  
  • Determining the teams/people needed for full support during the tests (this includes providing data, determining who should answer potential questions and a general overview of the teams working in a given PRE-PROD environment, so that other applications are stable while acceptance tests are ongoing   
• Design 
  • Selecting appropriate test cases and creating a set of tests based on them (test suits)   
  • Preparing test management tools (here we will mark tests that have been performed with a positive/negative effect)  
  • Ensuring that test teams have the required accesses  
• Execution
  • The test team starts work on the first tests 
  • Marking revealed errors with the use of test tools  
  • Re-checking new functionalities and regressions  
• Summary / Closing: 
  • Preparing a report on the test results  
  • Deciding on implementation based on the report and the errors described therein  

Types of UAT tests 

Acceptance tests performed by users (alpha and beta tests)   

• Alpha tests – this is a form of internal acceptance tests. These are most often performed by internal Quality Assurance teams and testers.   
• Beta testing is a form of testing carried out by end users. The software version, the so-called beta version, is passed to several external test groups.  

Beta testing helps to reveal errors not captured by an internal QA team. This type of test is very cost-effective, and doesn’t require any additional investment.  

Why do companies back out of conducting beta tests? 

Confidentiality – a common argument against conducting beta tests. Companies are afraid of information about the system leaking to the media. 

Too much confidence in one’s own abilities – companies believe that Quality Assurance team can capture any error.  

Too late in the process – some people claim that beta tests are conducted too late to bring any measurable effects. 

Lack of time – doubts related to reporting errors, e.g., that users won’t be systematic or report duplicated issues, which might impact the time of implementation.  

Production acceptance tests   

Theses are tests designed for the operators and administrators of a given app. The simulated production environment is used for tests related to backup, re-installation of versions, user management, and data migration.  

• Acceptance tests for compliance with the law 
  These are tests that are based on checking whether the application complies with current laws, regulations or security standards. Such tests are often performed by users or independent testing teams.  
• Acceptance tests for compliance with the contract   
  These are tests that check the acceptance criteria included in the contract while setting the terms of the request. Like legal compliance tests, such tests are often performed by users or independent testing teams.  

Read also: Test automation tools or good practices? How to speed up testing?

UAT – verification or validation? 

These concepts are very similar. However, to correctly answer the question of whether acceptance tests constitute verification or validation, let’s start with their definition according to ISTQB.   

• Verification – checking the correctness and providing objective proof that the product of the software development process meets the defined requirements. 
• Validation – checking the correctness and providing objective proof that the product of the software development process meets the user’s needs and requirements. 

Based on these definitions and my brief introduction to what acceptance tests are, we can conclude that acceptance tests are run to validate the product and are one of the last steps in software development. 

When answering the question: „Why are acceptance tests important in the software development process?”, it’s best to present their benefits:   

• They build end-user / customer confidence in the system  
• They check the compliance of the specifications with functional and non-functional requirements 
• They check if the system is functioning well 

To sum up: Why should I conduct User Acceptance Tests?  

All of this is very important in the software development process. In order for cooperation between the customer and the software provider to bear fruit, it’s necessary to ensure that each new version of the application brings as many benefits as possible, while minimizing the risk of new errors.  

GO TO:

1. 1. RxJS observables  
2. 2. What is RxJS testing (marble testing in RxJS)? 
3. 3. How to write RxJS test cases?  
4. 4. What is a marble diagram in RxJS?  
5. 5. How to read marble diagrams in RxJS? 
6. 6. What are the types of marble tests in RxJ?
7. 7. What are the benefits of RxJS testing? 
8. 8. FAQ

RxJS observables  

RxJS is a library for creating, managing and handling asynchronous data streams in JavaScript applications. The key element of RxJS is observables, which represent the data streams emitted over time.  

Observables are based on the Observer design pattern, in which the observer “subscribes” to the observed object and waits for the data it emits. RxJS offers a variety of operators, such as map, filter, and merge, which allow you to transform, filter, and merge data streams. The concept of the RxJS library is described in more detail in the article “Reactive Programming with JavaScript in RxJS“. 

What is RxJS testing (marble testing in RxJS)? 

RxJS enables reactive programming in JavaScript, so you can support asynchronous data streams and manipulate them with various operations, such as mapping, filtering, and merging. RxJS testing is designed to ensure that the code running on these streams is correct, error-free, and meets expectations. 

Marble testing 

In the context of RxJS testing, it is worth mentioning marble testing, which is one of the most popular tools for testing data streams in this library. Marble testing is a visualization of data streams using so-called marble diagrams. In marble, data streams are represented as strings of characters, where each character symbolizes a value in the stream or a special event, such as the end of the stream or the occurrence of an error. Marble testing allows you to define the expected results and compare them with the actual test results, which is especially useful for complex asynchronous operations.  

Test types

RxJS testing, especially using marble diagrams, allows you to verify whether functions and reactive operators work correctly in the context of unit tests. Unit tests focus on the evaluation of individual functions, operators and data streams. Their goal is to isolate individual code elements and check whether they behave as expected.  

RxJS Testing Tools  

To facilitate testing with marble diagrams, RxJS provides tools such as TestScheduler, which allows you to control time virtually without affecting your actual test run time. TestScheduler allows you to define input values and expected output values using marble diagrams, and then simulates data flow in reactive streams.  

How to write RxJS test cases?   

Writing RxJS test cases requires careful planning, an understanding of the source code, and the use of suitable testing tools. Marble testing is one of the approaches that helps to create transparent tests for data streams. Here are some tips on how to write tests for code based on the RxJS library:  

1. Get to know the source code  

Before writing test cases, carefully read the source code that will be tested. Understanding the structure of the code and the purpose of individual functions and operators will allow for the creation of effective and comprehensive tests.  

2. Choose the right testing framework  

Choose a JavaScript testing framework, such as Jasmine, Mocha or Jest, that best suits your needs and preferences. Make sure that the chosen framework works with RxJS and allows you to use marble testing if you plan to take advantage of this approach.  

3. Create the test structure  

Unit tests are usually organized in “describe” blocks (or similar, depending on the framework you use) which serve grouping tests for a specific function, operator, or component. In the “describe” blocks, we place individual test cases, defined by the “it” function (or the equivalent in the selected framework). 

4. Apply marble testing  

Marble testing is an approach that allows you to visualize data streams using so-called marble diagrams. In marble testing, data streams are represented as strings of characters, where each character symbolizes a value in the stream or a special event, such as the end of the stream or the occurrence of an error.  

For RxJS testing using marble testing, use tools provided by the library, such as TestScheduler. TestScheduler allows you to control time in a virtual way and define input values and expected output values using marble diagrams.   

5. Define test cases  

Test cases should be defined in a legible and understandable manner, describing the expected behavior of a function, operator or data stream. Using marble testing, you can accurately and legibly describe the expected behavior of data streams and compare the actual results with the expected ones.  

6. Test different cases and scenarios  

When writing test cases, it is worth testing different scenarios, such as:  

• Data flow with different values  
• Data flow with delays or intervals  
• Code behavior in case of errors  
• End of data streams  
• Proper transfer of values to subsequent operators  
• Also, try to include boundary cases to make sure the tested code can resist different situations.  

7. Browse and improve test cases  

Review your test cases regularly to make sure they are up-to-date and are still working. Try to optimize your tests by eliminating repetitive or redundant pieces of code, as well as making it easier to read and understand. This will facilitate the maintenance and development of tests as the source code changes.  

Writing RxJS test cases, especially using marble testing, requires an understanding of the structure of the code, selection of the appropriate framework for testing, creation of a test structure, and definition of the test cases that accurately describe the expected behavior of data streams. By applying the above guidelines, you will be able to create clear and effective unit tests for your code based on the RxJS library.  

Elevate Your Application Development

Our tailored Application Development services meet your unique business needs. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

What is a marble diagram in RxJS?  

Marble diagrams are visual representations of data streams in the RxJS library that help you understand the flow of data and interactions between operations in streams. The introduction of marble diagrams into the RxJS software development process can improve code readability, facilitate communication between developers, and speed up the testing implementation process.  

How to read marble diagrams in RxJS?  

Marble diagrams use characters to represent the values emitted by the stream, the end of the stream, the occurrence of an error, and subscription as well as the cancellation thereof. Here are some additional tips on how to read diagrams:  

Time: Each hyphen (–) means a unit of time, also called a “frame”. In marble tests, the length of the time frame is defined in the TestScheduler tool, which allows you to control the time of a test.  

Values: alphanumeric characters (e.g., a, b, c) represent the values emitted by the data stream. These values are usually defined in a value object passed to the toBe() method, which maps these characters to actual values. 

Subscribing and unsubscribing: in Marble, we can use the symbol ^ to indicate the moment when the subscription starts, and ! to indicate the moment when it is cancelled.  

An example of a marble diagram for RxJS may look like this:

---a---b---c---|  

This diagram represents a data stream that emits three values: a, b, c, each emitted at equal time intervals. The last symbol, |, indicates that the stream was finalized after emitting c. 
 

A B C 

     map(x => x.toUpperCase())  

A B C 

In this case, we have an input stream that emits the values a, b, c. Then we used the map operator, which converts each emitted value into its equivalent with a capital letter. The resulting stream thus emits the values of A, B, C. 

Marble diagrams can be much more complicated, with many operators and events. It is vital to understand basic notation to interpret and create such diagrams.  

What are the types of marble tests in RxJS? 

There are two main types of Marble tests in RxJS: unit tests and integration tests.  

Unit tests 

Unit tests focus on isolating and testing individual components, operators or functions. Marble testing is very useful in unit testing as it allows you to precisely check the sequence of events and the effects of individual operators on data streams. In unit tests, we use TestScheduler, which allows us to control the time and verify the expected test results.  

Integration tests 

Integration tests test the interactions between different components, operators or functions in the system. Marble testing can also be used in integration tests to examine how different parts of the code work together. In integration tests, we check whether the data flow between different components is in accordance with our expectations. Marble testing in the context of integration testing is more complex than unit testing, but it can provide valuable information about the behavior of the system as a whole.  

When testing with marble diagrams, it is also important to understand what “hot” and “cold” operators are. In the context of data streams, a hot operator is one that emits values regardless of whether there is currently any subscription. In turn, the “cold” operator begins to emit values only after subscription. In marble tests, the “cold” operator is usually represented by streams marked as source. Let’s picture this using the example below:

import { TestScheduler } from 'rxjs/testing'; 

import { of, Subject } from 'rxjs';  

import { map, takeUntil } from 'rxjs/operators';  
 

it('should demonstrate hot and cold observables', () => {  

  const testScheduler = new TestScheduler((actual, expected) => {  

    expect(actual).toEqual(expected);  

  });  

  

  testScheduler.run(({ cold, hot, expectObservable }) => {  

    const sourceCold$ = cold(' --a--b--c--|');  

    const sourceHot$ = hot('  --d--e--f--|');  

    const stop$ = cold('      -----s--|');  

    const expectedMarble = '  --x--y--(z|)';  

  

    const resultCold$ = sourceCold$.pipe(  

      takeUntil(stop$),  

      map(((value) => value.toUpperCase())  

    );  

    const resultHot$ = sourceHot$.pipe(  

      takeUntil(stop$),  

      map(((value) => value.toUpperCase())  

    );  

    expectObservable(resultCold$).toBe(expectedMarble, {  

      x: 'A',  

      y: 'B',  

      from: 'C'  

    });  

    expectObservable(resultHot$).toBe(expectedMarble, {  

      x: 'D',  

8. 

      from: 'F'  

    });  

  });  

});  

In the example above, we see two data streams: “hot” and “cold”. The “cold” stream only starts emitting values after the subscription, while the “hot” stream emits values regardless of subscription. In the test, we use the “takeUntil” operator to stop observing data streams at the right time. 

To sum up, marble diagrams in RxJS are extremely helpful in understanding the data flow and behavior of individual operators in data streams. Marble testing facilitates the creation of unit and integration tests, which are necessary for the proper functioning of the application based on RxJS. Know-how on marble diagrams and the ability to write effective marble tests are important competencies for every programmer who uses the RxJS library.  

When working with marble, it is worth remembering a few additional rules and tips that can facilitate the testing process: 

• Naming conventions: When you create data streams using marble diagrams, it’s a good idea to maintain naming conventions and use names related to stream functionality. This allows us to better understand which data flows through the streams, making it easier to analyze the test results.  
• Using Helper: In many cases in marble testing, we can use support functions provided by the RxJS library, such as TestScheduler, hot, cold, time, expectObservable and expectSubscriptions. These features make it easy to create and manage marble diagrams and test the flow of data between different streams.  
• Verification of test results: When writing marble tests, it is worth paying attention to test results and errors reported by testing tools such as Jasmine or Mocha. This allows us to quickly identify and correct errors in the code.  
• Accuracy of tests: Marble testing allows for precise tracking of the sequence of events in data streams. However, it is important to remember that marble testing does not replace other types of testing, such as unit or integration testing. It is worth using different testing techniques depending on the context and needs of the application.  
• Practice: Marble testing may be difficult to understand at first, but over time and with experience it becomes much easier. Working with different examples and scenarios will help you to understand how to best use marble diagrams in RxJS tests.  

Using marble in RxJS, you can effectively test data streams and interactions between operations. Marble diagrams make it easier to understand the data flow and the expected test results, which leads to building more reliable and efficient applications. Practice and experience are key to mastering marble testing, but its benefits are priceless to any front-end developer.  

Read also: Agile tester roles and responsibilities

What are the benefits of RxJS testing?  

• The code performs as expected – testing the RxJS code is extremely important because it allows you to increase the level of certainty that the code is working as you expect. The tests enable developers to be sure that their applications meet the assumed business requirements, which translates to a lower risk of errors or inconsistencies.  
• Error detection – at the same time, testing makes it easier to detect errors, because it checks individual functions, operators and the interactions between them. This allows you to identify and fix problems in the code more quickly. It also shortens the debugging time, as the tests help developers locate the source of the error quickly.  
• It’s easier to make changes – RxJS tests also support making changes to the code, as they automatically check the functionality after each modification. Thanks to this, the programmer can quickly detect and fix any problems resulting from the introduced changes. In addition, the tests facilitate refactoring, because the programmer can be sure that his changes will not result in regression.  
• Code documentation – it is also worth mentioning that RxJS tests often serve as documentation for the code, providing information on how to use individual functions and what their expected results are. Such documentation is invaluable to other programmers who wish to understand someone’s code and familiarize themselves with it.  
• Greater trust in software – finally, testing increases trust in code among both developers and project stakeholders. Everyone can be sure that the application works properly and meets the requirements. Tests are crucial to the maintenance of high quality applications based on data streams and reactive programming.  
• Better code – as a result, testing the RxJS code contributes to better code quality, faster error detection and correction, easier changes and increased trust in the code. As an RxJS expert, I strongly recommend using tests in all projects based on this library.  

FAQ

Read also: What is regression testing

1. 1. Is it possible to test applications faster?
2. 2. Creating preconditions using APIs or using a database directly
3. 3. Creating preconditions using APIs or using a database directly
4. 4. Converting Tests to Lower Level Tests
5. 5. Breaking down tests into smaller packages and grouping
6. 6. Reducing the number of similar tests
7. 7. Changing the test structure – matching the test to the tested functionality
8. 8. Appropriate use of waits
9. 9. Dedicated database for testing
10. 10. Use of headless mode
11. 11. Suitable equipment for testing
12. 12. Summary

Is it possible to test applications faster?

The dynamically developing technology and growing user requirements make the design of efficient and reliable web applications increasingly complex, and ensuring the proper functioning of the system requires a great deal of effort not only from the development team, but also from the test team. A good set of tests helps to address these challenges. Unfortunately, a focus on quality does not always go hand-in-hand with speed.

It is not only test managers and project managers who dislike the long wait for test results. This is also problematic for developers – the lengthy application testing process may be one of the reasons for delays in software development. For the sake of the project, tests should be carried out as soon as possible (using common sense – they also should be reliable and give real information about the tested application). Below are some ways to speed up testing without compromising quality.

Parallel running of tests

Running tests in parallel is the best and most effective way to significantly reduce testing time. By default, in many tools, tests are carried out in a sequential way (only one test is performed at a time, and once it is completed, another one is launched). In general, this option depends on the test runner used and the options it provides for this purpose.

For example, XUnit uses a concept called test collections, which defines how tests can be performed in parallel. By default, each test class is a unique collection, and tests in the same class will not run in parallel – for this to happen, they must be separated into two or more classes. Additionally, this tool allows you to create custom collections where the user can apply special attributes to create them, making it easy to manage and group tests as well as to run them accordingly, depending on the context. Other test runners might have similar functionalities. For example, in the Playwright tool, you can run tests in parallel both within one test class and throughout the project using configuration parameters. This option is enabled by default.

How much time can be saved with this approach? Let’s assume (without taking into account machine performance, server latency, network bandwidth, and other factors affecting testing performance) that we have a collection of 100 tests, each performed for an average time of one minute. It will take 1 hour and 40 minutes to start them sequentially. When running them in two threads (2 tests performed at the same time), the run time needed will be reduced by half – up to 50 minutes. Adding more threads will allow for an even greater reduction in the time needed to execute them.

Potential problems

Unfortunately, running tests in this way is not the golden mean, where you just need to change the configuration and increase the number of threads. Be aware of potential problems related to running tests in parallel, for example:

• the performance of one test may (e.g., through changes to the database or resource consumption / load) affect the performance of other ones. Such tests should be identified and run sequentially (before or after a collection of tests performed in parallel). Thanks to this, we will provide some isolation for all tests, and they will not affect each other while running;
• computer / server performance – select the maximum number of tests performed in parallel to avoid performance issues.

The appropriate approach to test analysis and dividing them into two groups (those that can be run in parallel and those that should be run sequentially) and selecting the right parameters allowing you to determine the maximum number of tests running can be a challenge, but the benefits of changing the way tests are run will allow you to significantly reduce the time taken for testing. Not only will this give you extra time for other activities, but most of all it will enable you to get feedback from tests earlier and address potential errors and failures more quickly.

Read also: Test automation frameworks. Introduction to Atata

Creating preconditions using APIs or using a database directly

When testing the user interface, it is often necessary to properly prepare the configuration of the tested application or use certain data necessary to run the test (these are most often described as preconditions). These types of actions should be performed using the API. If possible, the data can be prepared directly in the database. This increases the speed of tests (direct HTTP requests or modification of data in the database are faster than „clicking” individual options in the interface) and their stability.

Potential problems

Here, too, we face potential problems, such as the need to assure compliance with the API or database. While the API is easy to maintain (in most cases it is a matter of adding/removing fields in requests), in the case of a database you need to know its structure and the relationships between tables.

Converting Tests to Lower Level Tests

Performing tests through the user interface is generally slower and less reliable than lower-level tests such as API or unit tests. According to the test pyramid, there should be fewer UI/E2E tests than tests on lower levels.

UI tests require the browser to run and interact with it, which equals more time and resources. The interface rendering also takes time; it takes longer for the page to load properly for various reasons (e.g., a loaded application server or database). Sometimes the page may behave unpredictably (the component will not load properly, the dynamic element will not appear, etc.). This can not only lengthen the time of testing, but also make it unstable.

If you have multiple UI tests, consider converting them to lower-layer tests if possible. For example, a page access test, which can only be viewed by a logged-in user (e.g., a user of the account panel), can be performed entirely on the UI layer – including logging in. However, you can modify it accordingly and implement login to the application through the API, so that the UI layer will be used only to verify a specific page. Moving the login „lower” will speed up the execution of the test, because its exact execution on the UI side will be possible to start with the already logged-in user going directly to the desired page.

Breaking down tests into smaller packages and grouping

Grouping tests does not itself affect the performance of tests which have already been created. The goal here is to enable the selection of specific tests or sets of tests due to their properties (tested area, functionality, type, priority, etc.). Thanks to the appropriate specification of test cases, we get an option to run only those tests that are needed at a given point in time. The main split can be done by isolating stability, smoke and regression tests, and providing tests with additional attributes will allow you to start them in almost any configuration (e.g., due to the above-mentioned area and type of tests).

It is not always necessary to run the entire set of tests. Often we want to test only a selected part of the system or a specific functionality. Manual searching and running them would be a very time-consuming activity, and when we have a large set of tests, there is a chance that some may be missed. The use of attributes for filtration allows you to specify all tests that are needed in a given situation. For example, when you have an online store and are working on a new product type, you can only run tests related to adding a product to your basket or wish list before you decide on a full regression.

Read more: What is regression testing

Split testing – case study

Let me give you a real-world example. While working on a project for a customer from the bookmaking industry, we launched regression tests, among others, during each release.

Challenge:

At the very beginning, they were not categorized except for a few general tags describing what type of test (regression, smoke) it was, its ID in testRail, and who the author was. All tests were performed over about 2 days. Then came the meticulous process of analyzing failures.The client decided to change his approach and make running tests and subsequent analysis easier and faster.

Solution:

Together with other teams, we developed a process on the basis of which tests were analyzed and assigned to the teams responsible for a given application area. We marked them with new tags corresponding to the tested area, among others; we also created documentation – tests were unified and described. In addition, API/UI tests were extracted. At a later stage, we analyzed whether they were stable – on this basis, we isolated stable and unstable tests (flaky tests). We moved the unstable tests from the regression tests to a separate section and updated them to subsequently add them back to the regression set. During the next stage, we analyzed UI tests and converted them to API tests whenever possible.

Result:

Thanks to these solutions, we were able to reduce the time needed to perform all tests to almost 18 hours, and run additional tests, e.g., for specific areas of the application, so that we did not have to run a full regression every time.

Reducing the number of similar tests

It may turn out that a set of test cases consists of tests verifying practically identical functionality, but using different data (e.g., adding 1 product to the basket, adding 5 products to the basket, adding a product from the „household devices” category to the basket, adding a product from the „Consoles” category to the basket). If the second test case does not verify additional criteria (e.g., add a 5% discount when the customer has at least 5 products in the cart), consider whether the test case should continue to be used. This happens especially with a large number of test cases created for large systems, where as they develop you can lose track and create a test case that already exists. Periodic analysis of test cases and updating and verifying them will help to identify such test cases, and further research will answer the question of whether to keep the analyzed test case, modify it or completely remove it.

Changing the test structure – matching the test to the tested functionality

Tests should not only be stable and fast, but also properly written – so that they test a specific area of application or functionality. In a properly designed test, the number of steps needed to fully verify functionality should be as small as possible to avoid verifying something that should not be covered by testing. For example, when testing adding a product to the basket (assuming that the user must be logged in), we should not focus on asserting whether the user has actually been logged in – this should be checked when testing the customer’s login area to access the store. In particular, you can fall into such a trap by creating tests according to the BDD process (example: specflow), where the test steps are often simply copied from the existing one and placed in a new one, without editing it to fit them into the context.

Let’s consider a test of user login to the application:

Given I navigate to the homepage
And I click the Login button
When I log in as ‘testUser’ with password ‘testPassword’
Then I assert that the My Account button is displayed
And the ‘testUser’ name is displayed

You can use its steps and add them at the beginning of the test of placing a product in the basket:

Scenario: Add the product to the basket
Given I navigate to the homepage
And I click the Login button
When I log in as ‘testUser’ with password ‘testPassword’
Then I assert that the My Account button is displayed
And the ‘testUser’ name is displayed
When I navigate to a test product page
And I add the product to the basket
Then the Basket should contain the test product

Steps regarding the correct user login are not relevant in the context of adding a product to the basket – moreover, the login test is duplicated. Its significance is questioned, because logging in will be checked in every test that will be written on that basis.

Instead, it should be modified accordingly to include only the function of adding a product to the basket:

Scenario: Add a product to the basket
When I navigate to a test product page
When I log in as ‘testUser’ with password ‘testPassword’
And I add the product to the basket
Then the Basket should contain the test product

Thanks to this, the test was optimized in terms of speed and tested functionality.
Referring to the aspects related to creating preconditions, the login can be moved to this section and done on the API page, meaning that the test could start directly from navigation to the product page, where the user would already be logged in:

 Background: User is Logged In
Given I login as a test user through API
Scenario: Add the product to the basket
 When I navigate to a test product page
And I add the product to the basket 
Then the Basket should contain the test product

Going a step further referring to the test conversion, if it is not necessary to check something on the UI side, you can try to convert the test to the API level and speed up its execution even more.

Appropriate use of waits

The content of the application can be displayed statically or dynamically. AJAX technology, in which communication with the server takes place in an asynchronous manner without the need to reload the entire document, is used for the dynamic display of content. Thanks to this, the elements are loaded on the page at different intervals.
Unfortunately, this behavior forces the use of the right approach to interact with dynamically loaded elements. Using the static Thread.Sleep() method, which suspends code execution for a specified period of time, is not recommended. This results in an unnecessary extension of test time, and if the desired item is loaded faster than the declared pause, the program will wait idle until the specified time. If the item is not loaded before the specified time, the test will fail, as there are many factors affecting the load of the item (server load, network bandwidth, etc.), and sometimes it may take several seconds. Excessive wait time can also affect the stability of tests, which can sometimes fail.

Multiple automation frameworks (e.g., Selenium, Cypress, Playwright or Spock) contain built-in solutions. It is also worth mentioning external libraries such as Awaitability, which can extend the functionality of other tools. For this purpose, you can implement methods that will wait for further code execution until a specific condition is met (conditional waits). An additional advantage of this approach is the ability to define these methods globally and use them without duplicating code in tests, which also makes it easier to maintain the repository.

Dedicated database for testing

An undoubted advantage is the separation of data from the rest of the environments used in the project, thanks to which the processes do not disturb each other – sudden implementation in a given environment jointly used by the entire team or changes made manually will not affect the performance of tests. To create such a database, you can use the production base, which will give you a very similar test environment to that utilized by end users of the application.
When using a separate database, remember to manage it properly – for example, resetting the data when the tests are completed.

Use of headless mode

Headless mode means that the browser is launched without a graphical interface. Its operation and functions remain unchanged compared to the classic browser, but the resources are not additionally burdened with the rendering of the web application and the launch of the browser interface itself. Running tests using this mode will allow you to reduce the time of tests compared to running tests using a typical browser. This is especially useful when we run multiple tests at the same time – browsers with an active GUI use many more resources than those running in headless mode. An additional advantage is the option of using them to conduct tests on the server or container side, and hence a simple way to include them in the CI/CD process.

Suitable equipment for testing

Performing tests, in particular E2E, in browsers requires quite a lot of computing power. Tests perform many operations (launching a browser instance, rendering an application, executing a test code, etc.). Carrying them out with poorly performing equipment, especially if it is done in parallel, can not only negatively affect the time of their execution, but also lead to instability. The simplest solution is to bring the computer / server on which we perform tests up to date. This will increase the speed of the tests (and the number of parallel tests).

If there are reasons why updating the hardware is impossible, using cloud services might be a good idea.

Summary

The right test automation tool will certainly help streamline the entire process, but it is the automation tester, as the person responsible for the tests, who should follow good practices and promote their use in the project. With the above tips, you can speed up the execution of automatic tests, so that information on the reliability of the tested application will be available faster. This is important from a business perspective – the competition does not rest on their laurels, customers and end users expect the developed application to run smoothly and reliably, and any fixes will be made as soon as possible. Thanks to the swift testing process, we will be able to meet these expectations.

It is not obligatory to use all the listed methods at the same time, but using each of them will reduce the time of testing by several dozen minutes or even hours when we run lots of tests.

It is important to periodically check the set of tests and discuss optimization options in all respects – each member of the team responsible for software development should suggest corrections which will give us diversified feedback (from a technical and business point of view), making it possible to conduct in-depth discussions and draw appropriate conclusions, using them to optimize the tests.

Read also: Functional Testing in Software Development: A Comprehensive Guide to software testing types and Examples

When should we automate software tests and why?

What is Automation?

Before we focus on the topic of the Atata test automation framework, I would like to briefly remind you what software test automation is in general and why it is worth taking an interest in it. There are a number of myths around this topic.

Testing is a very important part of the software development process. One of the approaches to this aspect is automation, i.e., the use of special tools and frameworks that enable the programming of tests and automatic execution without further human intervention.

Benefits of Automation

Automation of these processes can bring countless benefits, including some of the major ones – time and cost savings. In addition, automatic tests are more repeatable and more accurate than manual tests, which translates to reliability and effectiveness.

Elevate Your Application Development

Our tailored Application Development services meet your unique business needs. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

Scenarios and test cases and automation

It is worth considering test automation in your project due to the option of conducting tests with complex scenarios and in many environments at the same time, which could be a challenge when manually implementing a test scenario on a daily basis. It is possible to perform tests on different platforms, in different hardware and network configurations, which would be difficult or even impossible to achieve in the case of manual tests.

Creating test cases in an automated form ensures their repeatability, and thus increases their effectiveness when it comes to detecting defects. This is why automation is particularly useful in testing the critical functionalities of the app. However, it is worth remembering that the test reality is not black and white , and like everything, it has its pros and cons. What are the downsides of automation?

The Dark Side of Automation

Test automation can be a fantastic solution in many projects, but it is not a universal solution. In some cases, manual testing can be a much better, more effective option, or even the only one available. Therefore, before deciding to automate tests, it is worth carefully analyzing the specifics of the project, the needs, and any available tools and frameworks.

You also need to bear in mind the time that must be invested in the creation of automated test cases. This requires many hours of work, configuration, and troubleshooting that you do not have to deal with during manual software testing. This is a process that can develop very slowly at first, and credible results are often visible only after putting a lot of work into the project. There is also a need to hire an automation tester who will be able to design the entire system so that the tests are stable and in line with good programming practices.

Types of test automation frameworks

Automation frameworks are tools that make this task easier by providing solutions to many common problems. Depending on the needs and specifics of the project, we can choose from different types of frameworks. Below is a brief description of some of them.

Library Architecture Framework

The Library Architecture Framework (LAF) is a popular software test automation technique. As part of this methodology, libraries with ready-made test functions that can be called in the test code are used. LAF is often used in cases where tests are complicated and require multiple stages, as well as in situations where tests must work on different operating systems and web browsers.
This type of framework is particularly useful for low-level tests, such as unit tests.

Module Based Testing Framework (MBTF)

This is a type of software test automation framework that is based on a modular approach to test design. In this model, tests are designed in a modular way, and each module covers one functionality or test cases related to one application module. In MBTF, test modules are designed in such a way as to be universal and can be used repeatedly for various applications and systems. Once you’ve created a set of modules, you can use them in different projects without having to rewrite the code.

Linear Automation Framework

Linear Automation Framework is a type of test automation framework that is based on a linear approach. At LAF, tests are designed to reflect business processes or application usage scenarios. This means that they are performed in a sequential manner. In this model, tests are designed as sets of steps, and each step of the test is associated with a user action or result that is expected in the application. The advantage of the Linear Automation Framework is that the tests are closer to the actual use cases, which can help us to find more realistic defects in the application, such as those that the user would come across.

Hybrid Test Automation Framework

This is a kind of framework that combines the features of all the above-mentioned approaches. This enables the design and execution of tests in a flexible way, tailored to the needs of the project. In HTAF, tests are created in a modular way, similar to the Module Based Testing Framework, but with more flexibility and access to a wide range of test tools. This framework allows the integration of various tools and technologies, such as Selenium WebDriver, TestNG, Appium, JMeter, etc., which allows us to automate tests in various environments, such as web, mobile, desktop applications and many others.

Read also: End to end testing with Cypress

Atata automation framework

Atata is a full-featured test automation framework that is based on Selenium WebDriver and designed for web applications. It was written in C# and offers a wide range of functionalities that make it easier to automate web application tests.

Atata – Advantages

• Availability – The Atata framework is fully open source and available on the GitHub platform under the Apache License 2.0.
• Transparency and ease of use – one of the key elements of the Atata framework is the use of the Fluent Page Object Pattern. This pattern allows you to write code that is easy to read and understand. The built-in login system makes it easy to track and troubleshoot problems during testing.
• Triggers – what is more, the framework has a unique trigger functionality that allows you to run custom code when specific events occur.
• Low Entrance Threshold – Atata is designed to be easy to use and maintain. The framework provides a simple and intuitive syntax for defining and using the objects on the page. This requires a minimum amount of code, which positively affects readability and facilitates the introduction of further testers to the project.
• Ready-made components – the framework also provides a set of ready-to-use components to facilitate work. These components include controls, pages, and layouts. Atata provides a wide range of built-in controls that can be used to interact with web elements on the site. Controls include buttons, text boxes, check boxes, radio buttons, and many, many more.
• Data-driven testing – Atata also supports data-driven testing, allowing testers to create tests using the Data-Driven Testing function. With this feature, developers can run the same test scenario with different datasets.
• Parallel testing – the framework also supports parallel testing. With parallel testing, testers can run multiple tests at the same time, which can significantly reduce the time taken for testing. This is especially useful for large web applications that require extensive quality control.

Atata – disadvantages

Despite its many advantages, the Atata framework also has some drawbacks.

• Limited use – one of the main drawbacks is that the framework usage is limited to websites only. This is an important limitation, for example, for companies developing software that also design other types of applications in addition to websites. In this case, you may need to use another test automation tool.
• Available only for C# – this framework is only created for the C# language, which makes it unavailable to many programmers who specialize in other popular programming languages.
• Small community – another disadvantage, compared to other test automation tools, may be the small community surrounding it. This framework is not as well-known as other test tools which have been available on the market for years. We need to take into account the fact that there are fewer online resources and documentation, as well as support materials available to users. Fortunately, the user community is highly engaged and ready to help. It is also worth noting that contact with the creators of the framework is extremely simple, and communication is highly efficient.

However, despite these disadvantages, the Atata framework is still one of the best tools for automating C# web tests. Its numerous advantages make it popular among testers who are looking for a simple and intuitive way to automate.

Atata vs Selenium

Let me start by saying that the question “Atata or Selenium?” is not quite appropriate in the context of the issue we are discussing. In particular, comparing Selenium to an overlay, which in fact Atata is, may seem baseless. After all, Atata is nothing more than a framework covering the database with its functionalities, which provides us with the necessary basic tools.

Thanks to this we get a “nicer” version of Selenium, which is more user-friendly and faster to implement. Nevertheless, I decided to lay out the important differences in the table below. It is worth being aware of the benefits of using such an overlay for the project. After all, when planning work, choosing a framework is not easy; the more options there are, the harder it is to make a decision even within frameworks that rely on other tools which are already available.

Atata

One of the biggest advantages of the Atata framework is its useful login system which allows you to track and diagnose problems during testing. Atata also offers unique features such as a trigger mechanism that allows you to execute custom code in response to specific events during testing.

Atata also offers an intuitive syntax for defining and using page objects, making it easy to program and maintain tests. It supports the use of the Page Object Pattern model, which is a huge advantage for fans of this pattern. This makes the structure clear and easier to understand.

The framework has a built-in set of ready-made components such as controls, pages and layouts, which speeds up the test automation process. Below I have provided an example of what Atata offers – it not only contains basic elements of pages such as text fields, but also differentiates fields for e-mail addresses, passwords, numbers or files. Furthermore, it supports choice lists, clickable items, and many, many more. A full list of specifications and capabilities can always be found in the very clear documentation at https://atata.io/components/.

using _ = TextBoxPage; //alias for better readability
    [Url(TextBoxUrl)]      //Url extension to base url placed in SpecFlowHooks.cs
    [VerifyTitle("ToolsQA")]
    public class TextBoxPage : BasePage<_> 
    {
        private const string TextBoxUrl = "/text-box";


        [FindById("userName")] //Selector to find element
        public TextInput<_> UserNameTextBoxInput { get; set; } //An element with its type and name


        [FindById("userEmail")]
        public EmailInput<_> UserEmailTextBoxInput { get; set; }


        [FindById("currentAddress")]
        public TextArea<_> CurrentAddressTextBoxTextArea { get; set; }


        [FindById("permanentAddress")]
        public TextArea<_> PermanentAddressTextBoxTextArea { get; set; }


        [FindById("submit")]
        public Button<_> SubmitTextBoxButton { get; set; }
       
        [FindById("output")]
        public Text<_> TextBoxOutputBox { get; set; }


        [FindById("name")]
        public Text<_> TextBoxOutputName { get; set; }


        [FindById("email")]
        public Text<_> TextBoxOutputEmail { get; set; }


        [FindById("currentAddress")]
        public Text<_> TextBoxOutputCurrentAddress { get; set; }


        [FindByXPath("output")]
        public Text<_> OutputBox { get; set; }


        [FindById("permanentAddress")]
        public Text<_> TextBoxOutputPermanentAddress { get; set; }
    }

Use of ready-made components

public class ProjectDetailsPage : BasePage<ProjectDetailsPage>
    {
        public int findGarageParkingPlaces()
        {
            var element = AtataContext.Current.Driver.FindElement(By.XPath("//div[contains(text(),'Garage Parking Places')]"));
            return int.Parse(element.FindElement(By.ClassName("value")).Text);
        }
    }

Calling AtataContext.Current.Driver

Despite its enormous capabilities, however, it sometimes turns out that we lack a component or it is impossible to use a ready-made solution offered by Atata. Luckily, the developers haven’t removed the option to access a pure version of Selenium, and we can search for our elements or apply specific actions to them with no issues. Below is an example of a WebDriver call.

Fans of reports and tables will also find a useful function here. Atata offers a built-in tool to take screenshots according to the selected pattern, e.g., after performing each step. If we connect this function with a library for creating reports, such as ExtendedReports, we will get a very pleasant environment for generating reports in the form of a .html file. Once the tests are run, we have an immediate view of the results along with statistics, marked errors, tags and graphs. At first glance, we can say that IDE itself can also offer such data but the form of compiling them in this library is much more readable and easy on the eye. After small changes to adapt to the needs of the project, we get a report that can be presented almost effortlessly during a business meeting.

[Screenshot(AfterAnyAction)]
    public abstract class BasePage<TOwner> : Page<TOwner>
        where TOwner : BasePage<TOwner>
    {
        //Some general helping methods that can be used in all pages
        public int[] GetElementSize(Text<TOwner> element)
        {
            return new int[2] { element.ComponentSize.Width.Value, element.ComponentSize.Height.Value };
        }
        
        public void AcceptAlert()
        {
            AtataContext.Current.Driver.SwitchTo().Alert().Accept();
        }


        public string GetAlertMessage()
        {
            return AtataContext.Current.Driver.SwitchTo().Alert().Text;
        }
    }

An example of adding screenshots to a report (1 line), I’ve thrown in the context of the whole class for clarity but I’ll leave it to you to edit as it suits you.

[Binding]
    public sealed class TextBoxSteps : BaseSteps
    {
        private readonly ScenarioContext _scenarioContext;


        [Obsolete("Visual Studio IntelliSense Work Around", true)]
        public TextBoxSteps(ScenarioContext scenarioContext)
        {
           _scenarioContext = scenarioContext;   //Initializing scenario context, you can save test session data in it
           SetRandomUser(_scenarioContext);      //Creating new fake user data and saving it in context so it can be used in multiple tests without generating new set 
        }


        [When(@"User input all personal data on address form page")] //SpecFlow binding to feature files, methods are named the same as scenario steps 
        public void WhenUserInputAllPersonalDataOnAddressFormPage()
        {
            Go.To<TextBoxPage>()                 //Open page   
                .UserNameTextBoxInput.Set(User.FirstName + " " + User.LastName) //Make actions on web elements from page you just opened eg. set -> input text 
                .UserEmailTextBoxInput.Set(User.Email) //User.Email is a reference to our generated fake data
                .CurrentAddressTextBoxTextArea.Click()
                .CurrentAddressTextBoxTextArea.Set(User.Country)
                .PermanentAddressTextBoxTextArea.Set(User.Country);
        }


        [When(@"confirm the data by clicking button")]
        public void WhenConfirmTheDataByClickingButton()
        {
            On<TextBoxPage>()                     //On already opened page
                .SubmitTextBoxButton.Click();
        }


        [Then(@"user should see provided data below")]
        public void ThenUserShouldSeeProvidedDataBelow()
        {
            On<TextBoxPage>()
                .TextBoxOutputBox.Should.BeVisible()  //Should is assertion verification if something is as you wanted
                .TextBoxOutputName.Content.Should.Contain($"Name:{User.FirstName} {User.LastName}")
                .TextBoxOutputEmail.Content.Should.Contain($"Email:{User.Email}")
                .TextBoxOutputCurrentAddress.IsVisible.Should.BeTrue()
                .TextBoxOutputCurrentAddress.GetContent(ContentSource.Value).Should.Contain(User.Country)
                .TextBoxOutputPermanentAddress.GetContent(ContentSource.Value).Should.Contain(User.Country);
        }
    }

Example of page navigation (Go.To and On), I paste again the full context of the class

The issue of navigating between application pages during one session was also intuitively solved. In the example below, you can see that using the Go function, It (go to) or He (stay on the page) is sufficient. Just upload the appropriate page address according to the method – there is no need to open and close the session properly (as with Selenium) without an overlay. This contributes to significant time savings when configuring your environment.

Unfortunately, this framework was created only for the C# programming language, which is a significant limitation when compared to Selenium, which also works with Java and Python.

Which test automation framework is best for my project?

The choice of test automation framework depends on many factors, such as project type, business requirements, developer preferences, and resources available. Here are some steps that can help you choose the best framework for your project:

1. Define your requirements

Before deciding to use a given framework to automate tests, you need to know what your requirements are. For example, if your project involves the development of a web application, then a framework that supports user interface testing might be the best choice. There is also no point in choosing a technology that supports other types of applications as well at the expense of the one that interests us.

2. Find the right tools

Consult developers and testers who have experience in test automation and ask them for recommendations on tools. Collect information about different frameworks, their functionalities, limitations and social resources. 

3. Test the frameworks

Test selected frameworks in practice. Prepare a set of tests representative of your project and implement them in each of the frameworks. Analyze which frameworks are the most intuitive and easy to use.  

4. Check the integration

If the developed application uses external tools, make sure that the framework you choose integrates with them. For example, if you use a Continuous Integration platform, make sure the framework you choose has a plugin for it.  

5. Pay attention to the availability of documentation and community support

Choose a framework that has good documentation and a wide community of users. This will help to solve the technical problems you may encounter when working with the framework.  

Summary

To sum up, the choice of a test automation framework depends on many factors and there is no clear answer to the question of which one will work best in your project. Currently, there are many types of frameworks available on the market, so you need to explore different options and choose the one that best suits your requirements and project specifics. I hope that with this article I have also managed to detail the capabilities of the not as yet well-known tool Atata.

Although Selenium is one of the more popular and widely appreciated tools, it is always worth knowing about other options. Atata is constantly being developed by the developer community, which means that anyone can contribute to advancements by reporting bugs, making improvements or writing documentation. This way, Atata becomes a better tool.

1. 1. What is test automation? How does it work?
2. 2. Types of automated tests
3. 3. Automation testing and myths
4. 4. Benefits of automation testing
5. 5. Test automation tools
6. 6. Summary

What is test automation? How does it work?

Before I discuss the test automation process further, a quick reminder: automated testing is software testing by executing a set of test cases using dedicated tools. Now, let’s see how such tests are created.

Firstly, we specify areas requiring automated testing, focusing on relevant issues from a business perspective. We set up the goal and terms depending on the tested system, bearing in mind one of the principles of testing: “Testing is context-dependent”.

Once the goal is set up, we proceed to the technical aspects. We choose a testing tool, and create the testing environment and testing scripts. Having run tests, we analyze the results and verify whether we have managed to obtain business value as assumed earlier on.

Deterministic automated tests work in a quite simple way. First of all, we put the application in the initial state to test a desired action, for example how the application acts while being used by, say, 1000 users. The aim of the verification is to compare the assumed results with the actual state.

Read also: Software testing: do we really need more tests?

Types of automated tests

There are several different types of automated tests. To get the most out of it, you need to know the key differences between the two. Tests are divided into functional and non-functional.

Functional testing

Is a quality assurance process and a type of black box testing that bases its test cases on the specifications of the software component under test.

• Unit testing: Unit tests are the first level of testing and form the basis for other more extensive types of testing. These are usually done manually by the developers.
• Integration testing: This type allows for layered testing. This type allows for layered testing. Generally, it’s about integrating 1 component with 2. For example, API with a database. This is often at the code level. 1st class with 2nd class (if developers do it).
• System testing: It’s a type of functional test that only covers the most crucial features of the software to ensure that it could be further tested.
• Regression testing: A type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software.
• User Acceptance Testing: Software test, the purpose of which is not to detect errors, but only to obtain a formal confirmation of the implementation of the software of appropriate quality.

Non-functional testing

Is testing an application or system against non-functional requirements: how the system works, not the specific behaviors of that system.

• Documentation testing: includes the plans, results, and testing of a system or system component. It includes test case specifications, test plans, test procedures, test reports, and test logs.
• Installation testing: Also known as Implementation testing and it is done to check whether the software is properly installed in the new system with its prerequisites and necessary data and is working as expected or not.
• Performance testing: The practice of evaluating how a system performs in terms of responsiveness and stability under a particular workload. Performance tests are typically executed to examine speed, robustness, reliability, and application size.
• Reliability testing: An important software testing technique that is performed by the team to ensure that the software is performing and functioning consistently in each environmental condition as well as in a specified period.
• Security testing: Type of Software Testing that uncovers vulnerabilities, threats, risks in a software application and prevents malicious attacks from intruders.

Automation testing and myths

1. Each test can be automated

According to one of the widespread myths, each test can be automated. Let’s think it over: is it really possible? As per the International Software Testing Qualifications Board – the software testing certification organization – the testing process is infinite, which is why we cannot forecast and test everything.

Test automation is still incomparable to efficiency tests, volume testing, and stress testing. Automated testing has its limitations, though. For example, it cannot be used for non-functional tests. These are related to our personal perception of esthetics, legibility and transparency.

In the abovementioned area we are unable to define the initial requirements and the expected final results. Some of the aspects, for example the appearance and transparency of the application, are perceptible only to the human eye. The same thing applies to accessibility testing, which aims to determine whether users with disabilities can use the system with ease.

All in all, knowing that some of the tests cannot be automated, we understand that it is impossible to entirely replace manual testing with automated testing – as per the second common myth. There will always be areas requiring manual testing.

2. Automated tests can replace manual testing

This misunderstanding brings to mind a common myth from the past, that one day television would entirely replace radio. Nowadays, as we can see, radio, television and the Internet coexist and complement each other. It works similarly when it comes to automated and manual tests.
The first ones are advantageous due to their repeatability. With appropriately implemented tests we can be sure that every test will be run in the same, replicable manner.

 Additionally, contrary to human nature, automatons do not need coffee, the weekend or sleep, so they can work perpetually. On the other hand, humans are better when it comes to intelligence, intuition and imagination, which are essential to running an effective test of the application.

The majority of tests related to functional and non-functional requirements can be performed manually. There will always be a need for manual tests, calling for the engagement and knowledge of a tester – namely usability tests, accessibility tests and security tests.

Test automation supports manual testing, but cannot replace exploratory testing or the technique of error guessing, wherein the tester’s knowledge in the context of the product and tools is essential.

3. Automated tests help to catch a lot of defects

Contrary to popular opinion, automated tests will not help to detect a lot of defects. One can ask: “How can that be? Isn’t that what we run them for?” Of course, it is one of the goals we are trying to attain; however, automated tests are quality-oriented, not quantity-oriented. To give an example, automated tests are useful in catching functionalities’ change-related defects in the event that software code has been amended. These are called regression tests and are used to ensure that changes implemented to either software, configuration or the environment will not impact the developed product itself.

Automated regression tests can successfully replace manual testing carried out by entire development teams, which – depending on the software’s complexity – would normally take hours, days or even weeks. Our trust increases with every single automated test. Even though they do not catch a lot of defects, they allow us to save time and money, enabling us to focus on the quality of the product instead.

4. Each automatable case should be automated

Knowing that not all tests can be automated and that there are areas in which we will still need manual testing – is it worth automating every automatable testing case?

There are situations in which swift software testing is required and engaging additional resources, tools or funds (which is often related to test automation) is not an option. Let’s paraphrase a quote by Roger Needham, Microsoft’s Research Laboratory Director, to clear up doubts: automation is a process of replacing something that works with something that almost works, but is quicker and cheaper.
This reminds us of the two principal ways of defining whether we should automate tests or not:

• automated tests should be faster than manual verification
• the cost of programming tests cannot exceed the income gained thanks to test automation

From a business perspective, the cost of tools and time needed for the implementation of automated tests in comparison with manual testing should be the decisive factor. This should facilitate the decision-making process in terms of automating testing cases.

Read more: The test pyramid – what, when and how to test software to get profits

Benefits of automation testing

Automated tests have many benefits. Application testing is a process. Every time the code is modified, it needs to be changed and retested in different modifications or systems. In such a situation, manual testing would be extremely time-consuming and thereby costly. With test automation, you can run tests multiple times at no extra cost. 

Test automation also allows for continuous testing and a quick check of the app. This allows you to avoid bottlenecks at the end of the software development process. Test also automation helps make your life easier. Reporting in manual tests is uncomfortable and thanks to test automation, reporting is smooth.

Test automation tools

So how to run the tests in the most effective way? Here is a list of a few best practices for test automation, as well as the tools and automation frameworks that will help you achieve success.

To start with, determine which test cases should be automated first. Then select an automation tool for testing. If you have already defined your needs, you can proceed and choose the appropriate test solution, for example:

• Selenium
• Cypress
• Cucumber
• LambdaTest

Summary

Despite many myths related to test automation, automated tests are gaining greater understanding and popularity. Myths related to automated tests keep coming up, which proves that a lot still needs to be done to clarify doubts. I hope that I have managed to clear up some of them and that the article will help to define further discussion on automated tests. This is priceless help for testers in the process of delivering a high-quality product.