How Taiko compares to other browser automation tools | Gauge Blog

August 23, 2019 | Soumya Swaroop Gupta, Nivedha Senthil

How Taiko compares to other browser automation tools

ThoughtWorks has been writing free and open source automated browser testing tools for over 15 years. Selenium RC was released in 2004, and WebDriver in 2007. Today, there are a lot of free and open source tools to automate browser based tests. Unfortunately, problems like flakiness and the cost of test maintenance have plagued this category of tests for a long time.

Over time, as users, we have opted for workarounds to deal with some of these issues. But sometimes all it needs is a different approach.

This year, the Gauge (by ThoughtWorks) team released Taiko 1.0, a simple NodeJS library to address some of the problems of browser-based automated tests.

In this post, we’ll compare Taiko with five popular open source browser automation tools

Selenium
Webdriver IO
Testcafe
Cypress
Puppeteer

Since we are a part of the team that built Taiko, our intention to compare is to discuss our rationale behind each point of comparison (along with examples) and how we think Taiko’s approach helps in each instance.

While analysing, we considered these parameters for a holistic comparison

Test maintenance
Reliability (reduced flakiness)
Performance
Test frameworks integration
Ease of test failure analysis
Cross Browser support
Language support

1. Test Maintenance

The goal: Tests should change only when there is a change in functionality. A change in the structure of a web page should not impact a test validating functional correctness.

However, most browser automation tools create tests that are hard to maintain. That’s because they use locators like XPath, ID's, CSS selectors to identify and perform actions on web page elements.

“Maintaining locators must be calculated as part of test maintenance cost.” - FireFox Test Engineering blog

These locators depend on the structure of a web page. The underlying structure can (and does) change frequently throughout the development process. Even minor changes in page structure break such tests. So, it increases the test maintenance costs.

Taiko’s approach

Taiko allows you to select elements on the page based on what you see. It's not based on the specific underlying structure that only a developer sees.

This means, to click that "Purchase" button you can use click( "Purchase"). It's based on the text on the screen and not based on the specific style, location or hidden identifier.

Multiple matches can be resolved with proximity selectors or a combination of them. Yet, if this is unsuitable, Taiko has fall back options. You can choose to select an element on the web page via XPath or CSS selector.

Benefits

Taiko’s API treats the browser like a black box. Tests written in Taiko are resilient to page structure changes as they minimise the need of using complicated locators.

Compare with examples

This example compares scripts automating the TODO MVC application. The tests currently pass for the React flavor of the TODO MVC. However, only tests written in Taiko pass when they are run against the AngularJS flavor of TODO MVC. This is because Taiko focuses on testing functionality and not the underlying page or framework.

Key Findings

2. Reliability (reduced flakiness)

The goal : Eliminate the root cause of flakiness in browser based tests.

In modern web applications, elements on the page can dynamically appear and disappear based on user actions. For example, imagine clicking on a user's icon to display their photos.

However, many libraries expect the test code to handle the wait time to perform an action. Other tools provide some granular and low level controls to explicitly wait for elements or time however, it's hard to learn and use these controls.

“1 in 7 of the tests written by our world-class engineers occasionally fail in a way not caused by changes to the code or tests.” - Google Testing blogs

Explicit waits make test code unpredictable. Improper configuration, inappropriate use of APIs along with ineffective handling of waits by the tool makes tests flaky.

Taiko’s approach

As shown below, both Taiko and TestCafe have very good implicit wait mechanism before performing actions. This reduces flakiness in tests. However, unlike TestCafe, Taiko also ensures that the performance of the test run remains good!

In case you want finer control, Taiko also has good fallback options to override the default behavior. You can define explicit wait for conditions for a given action in the test code to suit your needs.

Benefits

Modern test libraries "await" the element to appear instead. This makes your tests run more resilient to things like slow network connections. Implicit waits increase predictability.

Compare with examples

In this example only Taiko and TestCafe don’t need any explicit waits. To observe flakiness, comment out all the explicit wait conditions in tests of other tools and run it a few times. A detailed comparison is available in the readme file inside the example.

Key Findings

3. Performance

The goal: Get fast and reliable feedback on test failures

“Usually the concern is application performance, which can have a direct impact on the bottom line. Testing performance, on the other hand, has a direct impact on developer productivity.” - Why your choice of software testing suites matters

Slow test suites increase feedback time and time to fix, when something breaks. Teams generally tend to ignore slow test suites.

Taiko’s approach

Taiko explicitly chooses reliability over extreme performance. If this tradeoff is unsuitable, you can choose to override it to improve performance. Explicit waits can be defined for a given action.

Benefits

Fast and more importantly, reliable feedback helps improve developer productivity. Flaky tests can be difficult to work with. You not only waste time analysing them but also genuine failures are re-run to check for flakiness. So as important as it is to be fast, it’s important that tests are reliable.
Run tests in parallel for free. Running parallel tests in Cypress is a paid service. Unlike Cypress; Selenium, Puppeteer and Taiko tests can integrate with any test framework like Gauge, Mocha, and Jest. These can then use the framework’s parallel execution feature for free.

Compare with examples

In this example, we ran all the tests sequentially, multiple times, to compare performance across tools. Machine and other details of the run are available in the readme file of this example.

A shorter time and lower CPU indicate better performance. Here are some results of benchmark tests run in cpu with 8 core.

Key Findings

Tools	Total(sec)	Performance
Selenium - 4.0.0-alpha.7 (chromedriver - 83.0.0)	13.240	Average
WebdriverIO - 6.1.17 (chromedriver - 83.0.0)	5.044	Good
Testcafe - 1.8.0	23.977	Basic
Cypress - 4.8.0	14.247	Average
Puppeteer - 4.0.0	2.719	Excellent
Taiko - 1.0.12	4.757	Good

view raw ComparePerformance.md hosted with ❤ by GitHub

Testcafe’s execution time could be 16 seconds if there were no reloads between tests. Reloading added an additional 4 seconds to the test suite.

Some observations

We added waits for a time period in Selenium tests while measuring the performance. We found that it required a lot more effort to write optimized code (wait for required conditions) to reduce flakiness. Further optimisations can reduce the time taken for execution.
Since it is difficult to integrate TestCafe and Cypress with other test frameworks, they have to depend on their inbuilt parallelization support. WebdriverIO supports parallelization.
Running parallel tests in Cypress is a paid service.
Further optimisations can reduce the time taken for execution in Taiko. Users can choose to override implicit waits for a given action and define it explicitly in test code.

4. Test framework integration

The goal: A testing library like Taiko must be able to take advantage of the rich testing oriented feature set provided by modern test frameworks like Gauge, Mocha, and Jest.

Taiko's approach

Taiko is a simple NodeJS library that can easily integrate with any Javascript test frameworks.

Benefits

Avoiding vendor lock-in and increasing the portability of tests.
Available for free. E.x., Gauge’s reporting, parallel execution, IDE support, integration with CD workflows, and more.

Key Findings

5. Ease of test failure analysis

The goal: Tools must make it easy to collect contextual data to analyse test failures.

“The analysis part is normally done manually and if the analysis is not done correctly, there could be genuine failures that are overlooked or masked by other issues.” - 5 reasons why automated tests fail to find regression bugs

Failure analysis feature allows testers and developers to quickly find issues and fix test failures. Contextual data allows easier test failure analysis.

Taiko’s approach

Apart from the default data available on failure, more contextual data can be collected with custom plugins.

Taiko’s plugin architecture allows you to extend it in ways that suit your requirements.

Benefits

Users can use/ build a plugin to suit the requirements to collect relevant data. Example:- screencast plugins capture video of the current run.

Key Findings

Since Selenium, Taiko and Puppeteer can integrate with any test framework they can take advantage of the framework’s debugging, IDE, CI/CD support and more.
The paid services of Cypress have more options for test failure analysis in CI/CD environments.

6. Cross browser support

The goal: Ensure that tests run consistently across modern browsers.

Running tests across browsers adds significant overhead to both resources and test run times. We can avoid this overhead by choosing what to test. Modern Javascript applications, frameworks, shims and transpilers like babel do a good job of ensuring cross browser compatibility.

Browsers are now starting to adopt standards for better web compatibility and less fragmentation of underlying web platforms

Taiko’s approach

Like Puppeteer, Taiko also uses the excellent Chrome DevTools Protocol(CDP) to automate browsers. Both work seamlessly with Chromium based browsers.

Benefits

The next version of Microsoft Edge is adopting the Chromium open source project. The Firefox team is working on adding support for CDP.

Taiko scripts are standards compliant and portable once browsers adopt these standards.

Below are the details of our key findings of the browsers supported with various tools.

Key Findings

7. Number of languages supported to author test code

The goal: The language available for authoring tests should allow users to write tests expressively.

All the tools we’ve chosen for comparison, support either one or more programming languages to write tests. With first class support for commonly used programming languages, these tools can leverage the team’s capabilities as well as IDE support.

Taiko’s approach

Taiko only supports Javascript to author test code, because it’s built on top of CDP (Chrome DevTools protocol) and the best library for working with CDP is in JavaScript.

Benefits

It works well with the Node.js ecosystem and frameworks like React, Angular and Vue.js
The team doesn’t have to learn a new language if they are building web applications
Taiko can automate any part of the browser

Key Findings

Other Observations

Cypress has a paid service that gives you access to recorded tests when running Cypress tests from your CI provider. The dashboard provides insights about test runs.
WebDriverIO has support for integration with cloud platforms to aid cross browser testing under its cloud services.
Puppeteer has the best performance. However, testing is not the focus of Puppeteer. It is widely used for web scraping. When used to write tests, the learning curve is high.
Taiko unlike a UI recorder, Taiko’s REPL takes instructions given in the terminal and performs the action on the browser. The user can continue to give instructions to complete a workflow. Only successful actions are recorded as a script. This generates human readable code and keeps the learning curve low.

Summary

Here is the report of the tools comparison

Check CompareBrowserAutomationTools, a GitHub repository to validate our claims made in this blog.

We hope you find this comparison summary useful! Through various examples listed above we can deduce that Taiko is a reliable, cost effective browser automation tool with a good performance. There will be pros and cons of using any tool. If you’ve come across a tool that isn’t in the comparison but solves a problem better, leave us a comment.

As always, the team welcomes any feedback that helps improve Taiko. You can install Taiko and explore more!

Disqus Comments

We were unable to load Disqus. If you are a moderator please see our troubleshooting guide.

Join the discussion…

Comment

Log in with

or sign up with Disqus or pick a name

Disqus is a discussion network

Don't be a jerk or do anything illegal. Everything is easier that way.

Read full terms and conditions

This comment platform is hosted by Disqus, Inc. I authorize Disqus and its affiliates to:

Use, sell, and share my information to enable me to use its comment services and for marketing purposes, including cross-context behavioral advertising, as described in our Terms of Service and Privacy Policy, including supplementing that information with other data about me, such as my browsing and location data.
Contact me or enable others to contact me by email with offers for goods or services
Process any sensitive personal information that I submit in a comment. See our Privacy Policy for more information

Acknowledge I am 18 or older

4
Discussion Favorited!

Favoriting means this is a discussion worth sharing. It gets shared to your followers' Disqus feeds, and gives the creator kudos!

Find More Discussions

Tweet this discussion
- Share this discussion on Facebook
- Share this discussion via email
- Copy link to discussion
Best
- Newest
- Oldest

Excellent comparations. Thank you for sharing this out.

I'm just concerned about testing in different browsers. You said that Firefox and Microsft Edge are walking to using CDP, but what about others like Opera, Safari?

Thank you again!

Opera is based on Blink and Chromium so technically Taiko should work with Opera. However, there are a few issues while running some tests, will take a look at fixing that soon.

Safari does not have good support for CDP nor are they planning to add it anytime soon.

could you plz suggest how to launch chrome browser with taiko. i have no idea how to other browser than chromium.

Please refer the issue https://github.com/getgauge...

Show more replies

Very interesting. Great comparison chart at the end for quick reference.

Even if I don't plan to use Taiko, the comparison between available testing solutions is very useful to gauge my current choice. I did not see any mocking capability mentioned though, does Taiko address that? For example, TestCafe allows to proxy HTTP requests, which I use to mock environments too complex to deploy for tests.

Yes you can stub and mock requests. Here is a link that you may find useful https://taiko.gauge.org/#re.... Thanks

Do you have any idea of how many and which companies use taiko in their daily lives?

How to make use of lighthouse feature in chrome using Taiko? Any suggestions?

Taiko diagnostics, and taiko accessibility in https://docs.taiko.dev/plug... are examples. Hope you find it useful. Thanks

Interesting that didn't include protractor for the perf test. I guess there are lots of others though as you say. I recommend checking out Courgette https://courgette-testing.com which pairs cucumber and protractor for web / wdio for native mobile testing and contains a ton of time saving things in it

Language support (for writing tests) - Any plan for Typescript support in the future?

Please refer https://github.com/getgauge... for updates. There's a link to bindings.

Hi. would you mind including a comparison of Gwen in here or for future evaluations of web automation tools? It would help getting a non biased review. https://github.com/gwen-int...

Gwen is based on Selenium. So it should be like comparing it to Selenium in the article here.

+1 for Karate:

Twitter Embed

found this article + code comparing browser automation tools: https://gauge.org/2019/08/21/how-taiko-compares-to-other-browser-automation-tools/…

had to make some minor changes to Karate (dev branch) but here is how we compare to Taiko - "friendly locators" and all