Multi-Device End-to-End Testing: Selenium Vs. Testsigma

To talk about application testing today is considering an extensive array of devices currently available in the market. The reason is the high unpredictability of even speculating which device our user would use. Even if we restrict the geographical area, we can only list the devices based on present trends because of the geo-location testing issues. However, if we want our application to absorb any fluctuations in these trends, we should opt for making a device-independent application.

In our endeavors to achieve the same, we focus on two different things – covering multiple devices and performing end-to-end testing. Both processes are essential in protecting details that may go unnoticed considering current complex processes. This post will cover the benefits, the impacts, and what kind of application we deliver to the end-user. Along the sidelines, we compare two popular software on various parameters required in multi-device end-to-end testing and how each helps us in its way.

Why do we need multi-device end-to-end testing?

Multi-device end-to-end testing is a term coined by joining two concepts that are very vast in themselves. In the multi-device approach, we are generally not concerned about the generic bugs i.e. functional or non-functional. We are looking for device-specific bugs, i.e., bugs related to a specific device or a subset of devices but not all.

1. Multi device testing importance and a brief overview

Considering an extensive overview, the reasoning behind the importance of multi-device is straightforward. In the fourth quarter of 2024, mobile phone sales revenue was billed at $450 billion, a 7% increase in sales annually. But that’s not quite the problem; the actual product and its manufacturer are.

Generally, a mobile manufacturer releases multiple devices in the market to cater to the needs of people in various segments. In 2024, 63 mobile phones were launched worldwide, expanding their operating systems from Android to iOS to cyanogen. These devices work divergent when we run an application such as a browser. So it can have a direct impact on the website as well. A similar thing happens with mobile applications.

2. End-to-end testing importance and a brief overview

On the other hand, end-to-end testing does not care about the specific devices but the general functionality of the web application or the mobile application. The term “end-to-end” comes from our goals when we initiate this testing – to test the app from one end to another.

Here, we define one end at the start of the application, i.e., when the URL is hit on the address bar or the mobile app is launched from the icon. Users may leave the app anytime, but they should have a good experience wherever they do. In simpler terms, this is what end-to-end testing is all about.

How to do multi-device end-to-end testing?

While both multi-device and end-to-end testing may take up one good post by themselves, in this post, we will focus on another important concept – how to perform multi-device end-to-end testing?.

A simple, naive and expensive approach is to do everything by yourself. Buy tens of devices and keep installing, re-installing, and testing your app on each of them continuously. As with end-to-end testing, you must go through the manual if you do it independently. That is not the way to go through unless you consider your app to be viewed by selective people on pre-decided platforms and a defined geographical area.

A better way is to take advantage of a robust and efficient framework that works alongside you and concludes your tests much faster. In the same direction, we pick up two popular multi-device end-to-end testing frameworks – Selenium and Testsigma. These frameworks have unique strengths and have been shaped by years of community feedback and global engineering experience. Let’s see how these two frameworks stand to each other on different parameters.

Learning curve

As an engineer, when new technology comes in, one of the first questions is, “How complex is it?”. In other words, people want to know whether they can start their work sooner or will it take time to learn. In software, the learning curve depends on two things – where the features are and how to implement a feature.

The first part is to learn where each of the elements is located. Where is the “refactoring the code” option? Where is the “test coverage” option etc.? Testers often verify whether the opportunities that come into use popularly are directly in front of them or accessible with easy shortcuts (2 key presses). The second part is how much effort the tester takes to achieve a task. If you can do something in a single step, like recording your steps, it’s excellent; two steps are still good, three steps are a bit of work, but four steps to record your tests is too much.

Selenium has an easy learning curve and has been this way since its release. That has been a major factor in gaining a good grip over the community and how it is used today. Another reason is that Selenium has done a lot of “first in testing” things. So testers have not a got a whole cake to them but one piece at a time over the years.

Testsigma has an easier learning curve too. In addition to being a cloud-based platform, Testsigma also incorporates many things that are automated in nature. For example, a tap on a button in the application can generate a “click” test script automatically. This process is convenient and faster when test script generation is concerned.

Bottom line: Testsigma is comparatively easier to learn than Selenium.

Programming language support

Another thing developers and testers look for is programming language support in a tool. Since a long list of programming languages has emerged in the market with their unique features and strengths, Having language options are always welcomed. Some people like Python because it is easier to type, while others like Javascript because of no installation and additional setup. The following image shows a few of the popular programming languages:

As per my experience, I find programming languages an entirely personal choice for the developer as well as the tester.

In multi-device end-to-end testing, programming languages are used to write automated test scripts that can test the functionality of multiple devices. The problem arises because a team has more than 10 people, and it is rare that all members like or have experience in the same programming language. Here, we need to settle and train people who are not well versed in the selected language. A similar treatment is done to the new hires as well. Once hired, the team recruits people specific to the language chosen or trains them at their own expense.

Selenium has a long list of programming languages that it can support, which includes Python, Java, and C#.

It has been a famous USP for Selenium as testers are not required to switch to another language for a project if they are using Selenium. So, as far as Selenium goes, you don’t need to worry about programming language support. There are high chances that it supports your favorite one.

Testsigma focuses on a codeless automation approach and facilitates writing tests in a neutral language, i.e., English. Since English is widely used for communication, testers do not feel the burden of learning a programming language that is harder to write and understand. English-based test cases also make it easier for everyone to understand (especially new hires), which helps contribute to non-technical teams such as business analysts. So, how does Testsigma achieve this?

Testsigma uses natural language processing on cloud servers that analyze your English-based test cases and convert them to machine-understandable language. But the tester need not worry about this transition. All he needs to care about is mapping his ideas into English. For example, if you want to enter a website and open it in a browser, just type Navigate to <URL> or use the NLP to “navigate to”.

Test domain

Since we now live in a world where mobile devices are probably more important than desktops (considering the internet traffic stats), if you have an application, we need to test all of them before releasing the product.

Mobile internet traffic share

Earlier, testers used to go through a very rigorous phase of testing. They would test on Windows, then macOS, Android, iOS, Windows phone, etc. This cross-platform testing consumed a lot of time and cost us even more because one or two engineers were not enough for all the platforms. Then came the cross-platform end-to-end testing frameworks. These frameworks could execute the same tests on multiple operating systems, including mobile ones. The testers often prefer these frameworks as they save time, repetitive work, and costs.

Selenium is not available for mobile device testing currently. It does offer a Selendroid framework (which is not a part of Selenium) for Android-specific testing, but it is not as great as Appium. The limitation of not covering mobile devices impacts the use of Selenium as mobile applications have become a priority in the market. If you are not into mobile applications and wish to test your web application on a mobile device, you can achieve that with Selenium.

Testsigma is a bit different than Selenium in this aspect. While Selenium started as a browser-specific framework, Testsigma has always focused on mobile as well as being a desktop browser-based automation framework. This platform allows the installation of mobile applications and directly interacting with them while generating cases. In addition to it, Testsigma works in various other domains as well. This includes data-driven testing, API testing, and regression testing.

Bottom line: If you aim to just perform browser automation, both the tools can satisfy your requirements. If you need more than that, Testsigma would undoubtedly serve you well.

Recording test steps/cases

Selenium’s most prominent feature has been “record-and-replay,” which allows the tester to record the test and replay it in the future.

This is a simple process; first, the tester himself works on the application manually, like clicking on a button, entering values, etc. The system records these actions through an extension or by you directly working on a platform like Testsigma. This recording process is done automatically. When the tester stops the recording, the test steps are generated and can be easily repeated by changing a few variables such as browser, OS, etc.

Regarding multi-device end-to-end testing, record and replay facilitate the process since we have a long list of devices. Since record and replay ask the tester to interact with the application first on their own, it is an excellent feature to look at when we are concerned with end-to-end testing. Selenium popularised its record and playback feature as a “codeless” tool in the market. Personally, I think what Selenium does is partially codeless and too limited in nature. It was a breakthrough, but talking about it in 2024, better tools like Katalon are doing the same job.

Also, note that this is a Selenium IDE feature and not provided by Selenium Webdriver.

Testsigma provides two types of recorders as part of its record and playback feature – One browser recorder that records steps in the browser and the other a mobile recorder that records and generates steps on mobile apps. The browser recorder also provides intelligent suggestions based on the locators and reusable steps that can be used again in the future. The mobile recorder of Testsigma is an interesting feature to work upon.

With this, the user can interact with the actual application by installing it on Testsigma. While he is interacting, the same actions are generated automatically into test cases in the English language. You can also rerun them by saving them into your account.

Bottom line: Selenium’s record and playback is limited to browsers only, while Testsigma offers good features for browsers and mobile applications. I feel Testsigma’s recorder is faster, robust, and provides many good features for the tester.

Resource usage

Multi-device end-to-end testing demands a lot of resource usage. For instance, if I consider the chrome browser, it takes a little less than 1GB of memory when 10 tabs are running.

Apart from this, the framework’s behavior may impact the resource too. For example, if we talk about Selenium, we must have browser drivers and complete browsers installed on our system. This consumes a lot of memory when we need to run hundreds of tests on a driver. In addition, parallelization can load our systems as well. Since we deal with hundreds of devices, this could become a significant issue and a major breaking point while discussing multi-device end-to-end testing in our projects.

Selenium’s problem while dealing with so many devices is that it needs the driver and browser installed in the system. Moreover, one driver does not work with every version of Google Chrome. So it becomes a very repetitive task of installing and re-installing each browser version along with the driver and running the same tests. The best method to work with Selenium is to choose any online cross-browser testing platform that supports Selenium and preferably Selenium grid. This way, we save on resources and can take advantage of Selenium. But, as far as a standalone framework is concerned, Selenium is not a good choice as of current market trends.

Testsigma, on the other hand, is luckier than Selenium. Since Testsigma is a cloud-based platform, all it requires from the tester is a browser and an internet connection. With this, you just need to login into the platform, select the browser on which you wish to run the end-to-end tests, and that is it. For mobile devices, Testsigma provides real devices already set up on their farm that you can connect with your browser. The resources consumed here are just one browser with probably three or four tabs which comes to less than 500 MB for Google Chrome.

Bottom Line: Regarding resource usage, the absolute winner is Testsigma due to its cloud-based nature.

What are your icebreaking criteria?

Multi-device end-to-end testing is one of the most crucial testing methods in the testing phase. This is because if you are not testing your application on multiple devices, it will likely fail in front of the end users. The high unpredictability of the device in use makes the releases frightening and exciting in nature.

In this post, we took five of the essential parameters a tester would consider before settling on a tool for the project. Since there are a lot of tools available, we decided to pick one from the left category and one from the right:

• Installable tools and cloud-based tools.

• Scripted tools and codeless tools.

Selenium and Testsigma both have their own set of uniqueness and powerful features that helps establish multi-device end-to-end testing. The five parameters we took in this post are based on popularity and are not a standard to look forward to. If you have your own parameter to judge a tool, do let us know in the comment section, and we will include it in the post. Thank you for your time.