Real devices vs simulator vs emulator in mobile testing

Mobile testing with only real devices?

The fact that only testing on real devices assures us that our mobile application really works as we expected is incontestable. According to statistics, users tend to have higher expectations when it comes to a mobile application than desktop one and - at the same time -  mobile apps are more dependent on hardware. One of the examples is when an application takes advantage of various sensors accessible only on a real phone. Many software houses face problems of choosing an adequate testing laboratory. It consumes a lot of time - starting with preparing an analysis of the market to buy phones that are most popular in a particular country or region. 

   The most popular smartphones in 2019 across countries, source: DeviceAtlas.  

Sometimes also the clients ask for specific hardware and operating system combinations which software testers should take into account when proceeding tests. It could be challenging to maintain all in-house devices and stay up to date with market news. And in the end, we must remember that’s all about the money.  

Mobile market fragmentation is still a major challenge for QA specialists, source: Statcounter.com

Another thing is that testing on real devices gives us more information when we’re working with a fully finished or almost finished app, but when we are at the first phase of the development process it could not always be possible to see accurate results on the phone. To address problems mentioned before, software testers and developers often decide to include simulators and emulators in testing and development strategy, starting from the very beginning of the development process.

How do simulators and emulators work and when to use them?

Mobile simulator and mobile emulator are virtual devices that help developers and quality assurance specialists with the development and mobile testing process of an application and let them present their work progress to the client during a review meeting. They are very similar and they can be used interchangeably, but there are still some differences between emulators and simulators:

Mobile simulatorsMobile emulators Written in a high-level language Written in machine-level language Simulators mimic the internal behavior of the device but don’t mimic hardware Emulators mimic device software, hardware, and operating system Simulators are faster than emulators because they don’t mimic hardware Emulators are slower. Emulating hardware usually causes slow-down in the software Simulators are not suitable for debugging purpose Emulators are more suitable for debugging purpose

There are many types of simulators/emulators available dedicated to different testing needs and approaches. If you want just to check the responsiveness of your web app you can easily open Chrome development tools and switch to Device mode to inspect whether new changes still fit all required screen sizes. It’s the most commonly-used tool for Responsive Web Design checking, which enables to select from a couple of most popular mobile viewports and change device orientation from portrait to landscape. Not everybody knows that it has also other features:

• throttling the network,
• throttling the CPU,
• simulating geolocation,
• setting orientation.

It’s worth to keep in mind that using chrome device mode we can’t prove that our product will work on selected devices (previously chosen from the device list), but it helps in simulating “look and feel” of an application.   

Iphone 5s simulator view

Another way is to use more advanced tools that you first have to install on your computer or virtual machine. Xcode Simulator or Android Studio Code Emulator are the most recognized of them. Xcode gives an opportunity to open a simulator in two ways - without running your app or by running it inside the simulator. First, we have to choose the demanded device model and after a while, we will see a simulation of real iPhone, iPad or Apple watch with a well known home screen filled with apps’ icons (you‘ll be able to switch between screens). The set of apps mimics the real iOS environment and brings opportunities to proceed the integration tests on your apps. For example, when you test the app for making appointments with a feature that allows to integrate and save information about planned appointments in the native calendar, tests could be executed in the simulator. You can also use other commonly-used apps like Contacts or Game center. Filling with input data could be done in two ways - using a device keyboard (like on a real device) or switch to a system keyboard to make testing faster. Hardware menu opens the possibility of simulating the user’s action like device rotation - to see the horizontal view, making shake gesture or image zooming using the pinch gesture. 

Remote & cloud-based mobile testing 

We need to remember that simulators use the resources of the system on which they run, therefore they cannot be reliable in performance measurement, which is quite different than on real devices. If - for some reason - it is a problem for us to install simulators or emulators locally on our computer we could consider cloud-based emulators and simulators, that give the ability to select an operating system, browser version, and device type. To open an app we’ll just need a URL. We can even see multiple emulators at the same time on one screen to compare how the app will look like on different devices. Those services offer additional options to make our test more precise like making snapshots, recording video and recording network traffic. 

There are also services built as Device labs providing real phones and tablets which are controlled remotely by user, so functional test could be closest to the “real world” environment. Despite being helpful with the testing process, Emulators and Simulators are not perfect tools. Both of them cannot mimic a mobile device’s camera and battery. They do not reflect realistic memory consumption and they are not able to mimic interruptions like SMS or incoming calls. 

Which approach you should choose? 

To summarize, as a Software Quality Assurance, we should be aware of the pros and cons of emulators and simulators and that these tools are not able to completely replace real devices, but are quite helpful in the initial process of app development, debugging and mobile testing. However, maybe we will not run away from testing mobile applications on real devices, but we can use the potential of these tools and combine them with testing on real devices.