Reliability Test Automation using Playwright

 

Reliability Automation Testing With Playwright

What is Reliability Testing?

Reliability testing is a type of software testing process that verifies whether the software functions in an error-free way in the given environment for a particular time.

The objective of Reliability testing is to find errors and failures in the application and fix them before deployment. 

Reliability testing is important to perform by any software testing teams because it helps in finding ongoing repeated failures, exposes the root cause of the failure and reduces the risk of failures.

The purpose of Reliability testing is to determine product reliability and therefore ensuring the customers reliability requirements are met. 

Reliability testing is gaining importance and in an hour of the need in many organisations now especially into banking and insurance sectors.

Types of Reliability Testing 

According to the books there are three reliability testing types mentioned below:

1. Feature Testing: To test new features for every release.

2. Regression Testing: Check the impact on the existing functionality when the new features are added.

3. Load Testing: Evaluate the current system performance under typical and peak load. For more info, Please check my blogpost on Performance testing here : Performance Testing With K6.

Actually, Reliability testing is also called as Portability testing and is a part of Non-Functional testing.

Lets put this article in two steps :

1. Reliability test cases with Playwright

2. Run test cases on Gitlab CI

1. Reliability test cases with Playwright: 

In this article, Let's discuss some Scenarios/Test cases for Reliability testing. I have picked up 4 test cases and out of those two test cases on the UI and two test cases with Playwright Intercepting and Mocking concept.

Objective: The objective is to execute some reliability tests to find and focus on the errors in the application when the service or server is down, network issues, loading spinner or other errors like 400, 401, 404, 500, 502, 503 etc....

Please find the GitHub Repo here: Reliability Test Automation - Playwright

Test Case 1: TC_01_reliability_mock.test.js

Scenario 1 (Intercept/Mock):

Intercept the Request with api endpoint and mock the response

Scenario Description:

Intercept the request and modify/mock the response with a user defined error

Test Steps:

1.Intercept The request

2.Modify the 'text' with user defined error message

3.Navigate to the website and click on Open App tab

4.User is on My Facts page and checks the mocked response on the UI

5.User validates the error message on the UI

Application Under Test

We are using https://cat-fact.herokuapp.com/ as the Application Under Test.

*/

In this test, I'm Intercepting the HTTP request using the api endpoint **/facts?animal_type=cat therefore using the command 'route'. Then I'm using the 'fulfill' command to modify and mock the response like actually the response is coming from the server but actually it is not we are setting it according to our requirement.

I'm sending the error text as 'This is an Internal error and the error has been logged.' and therefore mocking the response which actually I need.

The output will look like below:

Test Case 2: TC_02_reliability_mock.test.js

Scenario 2 (Intercept/Mock):

Intercept the Request with api endpoint and mock the response

Scenario Description:

Intercept the request and modify/mock the response with a user defined error

Test Steps:

1.Intercept The request

2.Modify the Book 'Title', 'Author', 'Price' and 'Preview Rating' with user defined error message

3.Navigate to the website

4.User is on Books - crime category page and checks the mocked response on the UI

5.User validates the modified mocked message on the UI

Application Under Test

We are using https://danube-web.shop/category?string=crime as the Application Under Test.

*/

In this test case I'm doing the same like in Test case 1 as described above. Here, I modified the 'Title', 'Author', 'Price' and 'Rating' and therefore mocking the response.

So, this will be shown as output of my test:

I have picked up status (200 OK) from above two test cases but you can check for errors like 400, 501, 503 etc.. for any latency, network issues and service not available etc... This Intercepting concept is very handy and useful in testing these kind of regularly occurred and repeated reliability related errors!

This concept is same as the cy.intercept() from cypress! For reference on Cypress Intercepting concept click the link: Cypress - Intercept

Test Case 3: TC_03_reliability_ui.test.js

Scenario 3 (UI): 
Login to the app and capture the error message for the Reliability test

Scenario Description: 
Capture Error message for Reliability Test on the UI

Test Steps:
1.User navigates to Home Page
2.User verifies the title and url
3.User clicks on the Let Me Hack Button
4.User fills the form and leaves the Message description bo empty and clicks on Submit Button
5.User Validates the Error message as a part of the Reliability test.
Application Under Test
We are using https://automationintesting.online/ as the Application Under Test. 

In this test case, I have performed Reliability testing on the UI without any mocking as the check is directly validated on the UI itself. We validate the two error messages as shown below on the UI after clicking the submit button without filling the 'Message' box leaving this empty.

Output:

Test Case 4: TC_04_reliability_ui.test.js
Scenario 4 (UI): 
Login to the app and capture the error message for the Reliability test
Scenario Description: 
Capture Error message for Reliability Test on the UI
Test Steps:
1.User is on the Login Page
2.Enter credentials and click login button
3.User is on the Home page
4.User verifies the title and url
5.User clicks the Request Loan option from the side nav bar menu
6.User clicks on the Apply Now Button
7.User Validates the Error message as a part of the Reliability test.
Application Under Test
We are using https://parabank.parasoft.com/parabank/index.htm as the Application Under Test. 
*/

In this test case also like the test case 3 mentioned above we will validate the error messages shown on the UI. This is an Example Bank Website and register yourself to this first and then login with your credentials. After clicking the "Apply Now" button on the ‘Request Loan’ page, now validate the error message.

Output:

2. Run test cases on Gitlab CI:

I have ran the test cases on the CI, It was just quite easy to run these test cases on GitLab CI. You can run on your own choice CI tool like Jenkins or Bitbucket etc.

GitLab Repo: Playwright - Reliability Test Automation

GitLab Pipeline:  Pipeline         Pipeline Run 

Happy Automation Testing Guys :)!