Mobile app testing.jpg

Testing mobile apps presents more and different challenges to testing desktop applications. Mobile applications have to be built to work on devices with limited processing power and memory, with smaller screens, intermittent network coverage and finite battery power. In addition, there are multiple operating systems and versions in common use.

The effort required to verify enterprise level applications across a myriad of devices with varying operating systems and resources is considerable. Finding cost effective and efficient testing methods can be deceptively challenging. In this article, we explore some of the key considerations as well as common pitfalls and ways that you can avoid them.

What challenges do mobile apps face?

The most apparent challenge is the diverse range of devices in use. As well as the three main operating systems (Android, iOS and Windows Mobile), each has numerous versions in common use. With Android, different manufactures also further customise the OS to their handsets, adding further variation. The keypad and input methods vary considerably (for example, the latest Android devices allow for proximity of the finger to the device to be registered as well as an actual ‘finger press’).
The end result of this is that even if an application is tested on a given device or OS, there can be no guarantee that it will work on another device, even if it is from the same product family.

As well as the functional testing of the application, consideration must also be given to the non-functional aspects. These fall into broad areas, including:

  • Battery Life
    An app that makes heavy use of the network connection, screen, GPS or processor can rapidly drain a mobile battery. Customers soon tire of this and dismiss the application.
  • Performance
    Although modern mobile devices have significant power (quad-core are becoming common and octa-core phones have been introduced at this weeks’ CES), apps will almost certainly have to work on older devices as well, which often have very modest performance levels.
  • Security
    Many apps store data on the phone, or transmit the data to a cloud store, using the phone’s WiFi or 3/4G Connection. Although 3/4G connections are inherently secure, there have been many cases of packet interception when open Wi-Fi connections are used. Data on the phone is also vulnerable if the phone is ever lost or stolen.
  • Network
    Mobile users will often switch fluidly between mobile and Wi-Fi networks. Apps making use of the data connection must cope with this switching. In addition, connection speeds vary widely between 2G, 3G and 4G connections, often interspersed by periods with no connection. If a data connection is essential, then considerations must be given to buffering of data or some level of local storage.

Mobile apps testing strategy

With many thousands of device and OS variations in common usage, the first consideration is the range of devices that testing will be performed on. Generally the view is that it is best to concentrate on the most popular combinations. A balance must be struck between cost, the time available for testing and the level of confidence required in the application.

Having established the range of devices used for testing, the next question is how the testing will be delivered. There are three clear choices available to the tester:

  1. Physical Devices
    Using real devices gives the most accurate results. In addition, most non-functional testing can only be tested on physical devices. Understanding how an app uses a battery can only really be tested by a real world device. However, the range of devices that must be maintained and updated can be considerable and the cost restrictive. It’s also very physically demanding for the tester who has to spend their time tapping away at a small screen all day long!
  2. Emulators
    Modern emulators are getting much better at accurately mirroring mobile devices. They can simulate different connection speeds, screen sizes and device performance. Testing is usually carried out within a browser window, with many emulators available for free. Google offer a free emulator built into Chrome which can simulate a range of mobile devices and connection speeds. They are easy to use and can allow much of the testing to be automated and of course you avoid the cost of buying a range of mobile devices. They do not however completely mirror actual physical devices can only be to aid physical device testing, not to replace it.
  3. Cloud Testing Solutions
    Accessed via a web interface in a similar manner to emulators, mobile cloud testing uses real devices accessed on a demand basis. Larger providers such as BrowserStack allow testing across wide range of devices (over 700). They present the best of both worlds; they’re easy to use, allow the application to be tested on many devices without the cost of buying the devices, they accurately reproduce issues found on the devices (because you are actually using a device) and you can use automation. They don’t allow for you to carry out fully real world testing of things like power usage and behaviour in an intermittent coverage area, but you can conduct much of the functional testing via cloud solutions.

Often, testing will use a combination of the three methods outlined above. The use of automation can reduce testing time and allow testing to be delivered on multiple devices far more rapidly. However, you can never completely remove real devices from the mix though. The look and feel of the app when used on a small device held in your hand, often from remote areas with poor or intermittent signal can never be fully replicated via a web browser.

Types of testing

Having established the range of devices that will be used for testing and the methods used to undertake the testing, you must also establish the types of testing that will be undertaken. These include functional testing and the following key areas:

Usability

From visibility of the text, feedback to any interactions and general navigation, usability testing is often the first mobile testing that applications receive.

Compatibility

Testing the application across as wide a range of devices as time and money permit. In addition to variations in screen size, resolution and OS, the interaction of the application with other (often memory resident) applications must also be established.

Performance

This can be considered in three main areas:

  1. Connectivity Testing: checking connection (2/3/4G), switching from mobile to WiFi and working across different mobile provider networks.
  2. Application Response Time:this could be impacted by the network performance, memory use or processor demands.
  3. Battery Consumption: often heavily impacted by network access, screen use or GPS connection. Heavy use of the processor will also cause a significant drain.

Operational

Exploring the issues caused by a phone crash, flat battery or other phone events taking over from the application. These may be notifications, phone calls and reminders.

Security

Encryption used for data transmission and reception, use of VPN and safeguarding of locally stored data if the phone is lost or stolen.

Conclusion

The testing of mobile applications introduces considerable challenges that can severely impact on the time and cost required to produce mobile apps. An effective mobile testing strategy taking into account how the application will be used will mitigate the challenges and reduce the time required for testing. If you need help testing your mobile apps, please contact us. At Acutest, we will work with you to both plan and deliver your mobile testing to make best use of the available tools to deliver effective testing of your mobile application.

Contact acutest