Frontend Test Automation: April 2021

JavaScript & TypeScript Test Automation Tools:

When it comes to frontend automation on JavaScript modern web applications like React.js, Vue.js or Angular 2+ Typescript, We have different tools available out there in the market but we always prefer the best automation tool isn't it?

For JavaScript/Typescript automation we have list of tools like below

Playwright
Puppeteer
WebdriverIO
WebdriverJs
Cypress.io
TestCafe
Robot Framework
Tricentis Tosca Testsuite

So which is the best tool from the above all? Well, there is no such tool which is called "Best". All the mentioned tools above are best with their unique features.

It actually depends on our project and requirements what we want and then we select the right automation tool fitting our project.

Points to ponder for Simple Test Automation:

Selection of Test Automation tool
Unique id's for elements in the DOM
Test Automation Framework with design pattern
Writing Test cases and functions with Async/Await
Test runner like Mocha, Jest, Karma or AVA
Assertions using Assertion libraries like Chai or Jasmine
Retry logic of your tests to tackle flakiness
Cross browser testing
Parallel testing
Test Reporting with Allure, Cucumber-HTML-Report, Mochawesome
Running automation tests in containers with Docker, if needed!
Integrating with your CI with Headless browser

Selection of Test Automation tool: It depends individually to work on the tools as a POC and see your desired results matching your project requirements.

Selenium Tools: People coming from Selenium background prefer WebDriverJs (OR) WebdriverIO. From the last few years, WebdriverIO gained more attention and popularity than WebDriverJs.

Non-Selenium tools: Playwright, Puppeteer, TestCafe, Cypress and Tosca.

Playwright: Playwright is a Node library from Microsoft and an extension to Puppeteer which is gaining more attention and reputation since last year. Same people at google moved on to Microsoft and extended Puppeteer to Playwright. It supports multiple language bindings like typical Selenium Webdriver. Playwright supports JavaScript, TypeScript,.NET, Java, Python. It looks similar to puppeteer which supports page object design pattern and cross browser testing with only Chrome, Firefox and WebKit aka Safari browsers.

Puppeteer: Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol.It Supports only Firefox and Chrome browsers for now.You can use Mocha or Jest test runners with it.Integrating with CI with puppeteer is very good running fast saving test execution time which is rich feature to opt for.It supports page object design pattern.

WebdriverIO: WebdriverIO is a next generation Selenium automation test framework for browser and mobile automation on Node.js. It supports Visual Regression testing, Page Object design pattern, retries, Test reporting with multiple reporters, Mobile automation with appium on iOS and Android.

WebDriverJs: WebdriverJs is Selenium-Webdriver automation tool used exclusively for JavaScript frontend applications.Since we are waiting for Selenium 4.0 to hit the market soon until then it is good to use the stable version of selenium 3.4 or 3.6. It supports page object design pattern, test reporting with Mochawesome on using Mocha test runner.WebdriverIO is well preferred because of it's big community and support than WebDriverJs in lot of companies. WebDriverJs is last used actively until 2019 from my opinion.

Cypress: Cypress is a frontend automation testing tool built for modern web applications.Test and mock API's for API automation testing. Cypress's App Action Pattern meaning, it is an alternative to
Page Object Model(POM) where actions (such as login, navigation, and API interactions) are encapsulated in functions rather than classes, making them more modular and reusable.

Yes, Cypress supports both the Page Object Model(POM) and the App Action Pattern, but it naturally aligns more with the App Action Pattern due to its architectural design.

App action pattern using the test data in the fixtures folder and re-usable code in the support folder in commands.js section of cypress.

Using cypress dashboards make cypress a paid tool but videos, screenshots and retries of tests are well supported by cypress.Coming to cross browser testing cypress supports electron, Chrome, Firefox and Microsoft edge browsers.

TestCafe: Testcafe is a free and open source test framework based on Node.js to automate end-to-end web testing with latest ES6+ Typescript syntax supports headed and headless execution of tests. Testcafe do not control the browser like Webdriver or Puppeteer. TestCafe is a proxy server that runs in the background and converts all HTML and JS code required for test automation. It handles pageloads well for the elements basically no need for explicit waits.

Robot Framework : Robot Framework is a generic open-source automation framework for acceptance testing, acceptance test-driven development (ATDD), and robotic process automation (RPA).

Robot Framework allows testers to easily write automated test scripts using a keyword-driven testing approach. It is a wrapper just like Appium on Selenium.

Robot framework uses many Libraries without writing any code using a plain text file with keywords and called as keyword driven framework.
It uses libraries like Selenium and Browser (Playwright) for UI automation.Similarly, It uses appium library for mobile test automation.

Tricentis Tosca Testsuite: Tosca Testsuite from Tricentis company is a script less model based automation tool meaning we don't need to develop any code for any automation scripts, instead we create modules or use available Tosca modules and re-use them by a simple drag and drop to your test cases. Tosca handles huge amounts of test data and it has its own Test data service called TDS writing the test data into this repository. It is a code less test automation tool until 'Automation Engineer level 1' training course. From 'Automation Engineer Level 2' you need to know basics of C# programming language to deal with custom controls and Special execution task(SET).
TOSCA is a paid tool. Frontend and Backend API Automation, Automation of Legacy mainframe, Automation with Databases and BI is achieved with it.

Unique id's for elements in the DOM:

Unique id's are always important and are a part of best practices for automation. Using code centric frameworks like Vaadin and Liferay for frontends and jumping from Java 8 version to Java 12/13 versions with these frameworks create a hassle for automation. The id's will not be unique and then you may encounter with same classnames used multiple times on the HTML document and also classnames changing very frequently for every release.

We also end up in writing very long xpaths or CSS for these elements making our tests more flaky. To avoid these situations speak to your developer to add unique id's like data-test-id or data-validationid or data-testid for every element we want to automate. Now our automation tests won't be flaky anymore.

Test Automation Framework with design pattern:

For better maintainability and re-usability we use page object design pattern in our test automation framework when there are element changes with selectors or new addition of features we don't need to change the entire code instead just change the specific selectors only in your JavaScript file alone which will reflect in all your entire framework saving time. It is the best practices followed in Test Automation.

Cypress and Tricentis Tosca are out from this list as Cypress supports APP action pattern and Tricentis Tosca supports Reusable test step blocks(RTSB) in a way contributing to the same concept in a different way.Except Cypress and Tosca all the tools listed above support page object design pattern.

Writing Test cases and functions with Async/Await:

Except Cypress and Tosca, Rest all the automation tools listed above support the use of JavaScript flavour of async/await keywords for handling promises.

Test runner like Mocha, Jest, Karma or AVA:

Usually we use Mocha or Jest test runners. TestCafe, Cypress and WebdriverIO comes with inbuilt test runners.
Playwright uses Folio test runner with Jest. Puppeteer, WebDriverJs have options to choose test runners as Mocha or Jest. Tosca is all together a different new setup and we don't use any test runners.

Assertions using Assertion libraries like Chai or Jasmine:

Assertions are most important key point feature in Test Automation. Without Assertions automation have no meaning. Mocha comes with Chai assertion library and Jest comes with Jasmine assertion library. Both Chai and Jasmine do the same thing and it depends which one we choose. Usually the assertion keywords are assert, expect and should to catch bugs on the frontends to identify if something is changed.

Tosca assertion keyword is verify.
Testcafe comes with its own assertion format like await t.expect(marketPage.title).ok()

Retry logic of your tests to tackle flakiness:

We have the test retries for running the tests number of times to tackle flakiness. Cypress and WebdriverIO have inbuilt retries defined cypress in run and open modes in cypress.json file. WebdriverIO with it's specFileRetries in wdio.conf.js configuration file.For Puppeteer, Playwright and WebDriverJs when choosingthe test runner Jest with jest.retryTimes() and with Mocha as this.retries().

In Tosca it is Settings->TBox->Recovery on TestCase, TestStep and TestStepValue retries.

TestCafe in package.json file will be like testcafe firefox my-tests --retry-test-pages

Cross browser testing:

TestCafe is the winner here as it supports all the browsers like Chrome, Firefox, Opera, Edge,Internet Explorer and Safari. Cypress supports only Firefox, Chrome, Electron and Edge.
WebDriverJs and WebdriverIO because they are native selenium supports all browsers except Opera I think. Also Safari browser is a bit tedious with WebdriverIO and WebDriverJs. Puppeteer supports only Chrome and Firefox.

Playwright supports Chrome, Firefox, Edge and Safari browsers.

Tosca supports Internet Explorer, Firefox and Chrome.

Parallel testing:

TestCafe, WebdriverIO and Cypress are very well supported on Parallel testing. Tosca uses Distributed Execution (DEX) for handling parallel test execution. With Playwright Parallel testing is very easy by default.You can just not only run different .js or .ts test files in parallel but also you can run individual tests in the same .js or .ts files in parallel.

Test Reporting with Allure, Cucumber-Report, Mochawesome:

All the tools when used used with Mocha test runner generate Mochawesome report when used with Jest have integration to allure reporting. When Used BDD framework with Cucumber to get Cucumber HTML report. Tosca have its own test reporting.

Running automation tests in containers with Docker,If needed!:

In this pandemic times with remote work or Home office sharing your test code with your colleagues is easy with Docker. First of all it is a very brilliant way of configuring and packaging our automation tests and running them on a container.

You can push all your required images to Docker Hub and share the image with your colleagues in a faster way. All you need to have is simple Docker file and Docker installed on your computer.

Integrating with your CI with Headless browser:

Different companies and projects use different CI tools like Jenkins, Gitlab, Bitbucket, CircleCI, Bitrise etc. Again, it actually depends which one meets your project requirements. All the priority tests are run locally for cross browser testing and we consider chrome headless for test automation execution of tests for CI. Once the automated regression suite is setup and integrated to CI it saves a lot of time checking these tests manually in release times.

For every new features deployed the regression suite automatically triggers checking whether the existing functionality is impacted against with newly deployed features. This way failure on regression suites stop deployments to analyse the actual reason of failures and then taking a Go or a No-Go decision.

For sample Boilerplate code, Please check out my GitHub Repos below:

Playwright Test Runner with JavaScript:

https://github.com/JayKishoreDuvvuri/Playwright-ReactJS.git

https://github.com/JayKishoreDuvvuri/Playwright-JavaScript-SauceDemo

https://bitbucket.org/jaykishore96/test-automation-playwright/src/ui-test/

Playwright Test Runner with TypeScript:

https://github.com/JayKishoreDuvvuri/Playwright-ReactJS-TypeScript

https://github.com/JayKishoreDuvvuri/Playwright-Test-Runner-TypeScript

https://gitlab.com/jaykishore96/playwright-typescript

https://bitbucket.org/jaykishore96/playwright-typescript-angular/src/master/

Playwright with Jest: https://github.com/JayKishoreDuvvuri/Playwright-Jest-Test-Automation-Framework

Puppeteer with Jest: https://github.com/JayKishoreDuvvuri/Puppeteer-Jest-Docker-Automation-Framework