iOS Archives - Stephen Janaway

Although there can be no doubt that testing mobile applications and websites is a major growth area in the industry at the moment, there are a number of challenges that mobile testing introduces that are unique to the discipline. In particular, there is the need to ensure application functionality and compatibility across a wide range of devices and in a wide range of different situations. In order to do this efficiently and effectively then it is essential that companies maintain a suitable mobile testing lab.

What Is A Mobile Test Lab?

A mobile test lab is a broad term for a collection of the materials and articles that a tester or test group will require in order to effectively test software that is intended to be accessed via a mobile device. This may include a number of different items required to support the testing itself, as well as items that are required in order to replicate certain specific test conditions. Many test conditions, especially those for situations like low signal strength and low battery levels are unique to mobile devices and are unlikely to have been considered if a testers experience is in another testing domain.

What Should You Include In One?

The contents of a mobile test lab will of course vary depending on the applications and devices under test. A test lab should contain a suitable number of desktop machines with internet access via Wi-Fi and should be situated in an area where there is access to suitable cellular signals (2G/3G/4G). Situating a mobile lab close to the test team is also a sensible choice.

A wide selection of mobile devices

Probably the most difficult part of a mobile testing strategy is ensuring that there is sufficient device coverage. This is a particular problem when testing the Android ecosystem where there are a large number of different OS versions and a number of different vendors who produce a large number of different sorts of devices. However it is also an important consideration when testing on other OS’s as well. The primary focus should be on the devices that the application is targeted towards, as well as the version of the OS with the largest deployment in the field. For example, although Jelly Bean is the most recent version of Android, 64% of devices are still running the older Gingerbread version. Ignoring the devices in the field would be a critical mistake.

It is important to ensure that the different screen sizes and mechanical styles are covered, in particular differences between portrait and landscape screens, and hardware QWERTY keypads and touch screens. Also important is to ensure that devices from a variety of different carriers are included. Mobile device manufacturers typically adapt their software to the needs of different carriers, particularly in the US, and therefore they may work differently, even if the overall OS version is the same. A suitable suit of mobile devices can represent a significant outlay to a company or individual and another solution is to use a cloud based service although these are not as easy or quick to use as having the device in the hand. A low cost option is to make a device library by using the devices owned by the testers in the team. Obviously this is not ideal since some test cases may require user data to be removed or device software to be changed, for some applications then it can be a low cost solution. Whatever solution is chosen, a suitable indexing and booking system should be used in order to ensure that the usage and whereabouts of each device is tracked.

A suitable selection of SIM cards

Not all SIM cards are created equally and some have more functionality than others. SIM cards which support cellular data and particular network supported functionality are likely to be required. It is also important to ensure that the test lab has SIM cards that are not only active and usable in the country in which the lab is situated, but also that are relevant to the particular region in which the application will launch.

SIM cards vary greatly in their access speeds and in particular, when testing contacts and address book applications, which may access the SIM card, then it is important to ensure that the test lab contains known slower cards which can uncover timing related bugs. If the test strategy also intends to make use of cloud based services then it is possible to also get access to a number of SIM cards in the cloud based devices. However, be aware that it is difficult to generate timing related issues on cloud based services, primarily due to the slower speed of device access.

Memory cards

It is important to ensure that the mobile test lab contains a number of memory cards which can be used to generate test conditions such as full cards, card removed during read or write operation or card inserted during a specific operation.

Apparatus to simulate low signal strength areas of both the cellular and Wi-Fi signals

One commonly overlooked area of mobile testing is that of low signal strength. Applications are typically tested in the perfect environment of the office, with good cellular and Wi-Fi signals, and bugs are only uncovered after launch when the applications are used in real-world environments. Applications often fail when the signal strength is low, or in areas where there are frequent handovers from the 2G to 3G cellular signals or Wi-Fi. Testing for low signal strength can be carried out in a variety of ways. It may well be that ‘dead spots’, areas with no suitable Wi-Fi or 2G/3G cellular signal exist in the office, or these can be simulated by using shielded boxes or even old microwave ovens. Simulating handover from 2G to 3G cellular signals is more difficult and may require that the tester leave the test lab or office in order to find a suitable signal area outside. Mobile operators often have specific low signal and handover areas and routes are driven whilst testing, specifically for this purpose. It is also possible to purchase mobile network simulators from companies such as Anite and Anritsu which are able to simulate these conditions but they represent a significant financial outlay.

Apparatus to simulate low battery levels

Testing for battery life considerations is another area of testing which is often overlooked. Poorly written applications can have a significant effect on the battery life of devices, and applications themselves can often exhibit unwanted behaviour when battery levels on devices are low. A mobile test lab should contain both the ability to simulate low battery levels, and the apparatus to monitor the effect that an application can have on battery life. Simulating low battery levels can be achieved either via dummy batteries which are connected to power supplies on which the voltage and current can be varied, or a suitable supply of batteries with varying amounts of charge. Monitoring the effect an application has on battery life can be achieved by using the on device diagnostic tools or applications that are available for most popular mobile devices OS’s.

A suitable selection of peripherals

This should include a selection of headsets, both wired and Bluetooth, plus a selection of USB and other connectivity cables which can be used to connect devices to desktop PCs to test upgrade/ downgrade, back-up/ restore and to add test data such as address books, photos, videos and music files. There should also be a selection of chargers and enough power sockets to ensure that phones can remain on charge. This is especially important if the test automation solutions also reside within the test lab and mobile devices are required to remain on charge whilst testing is on-going. If appropriate, a mobile test lab could also contain suitable car kits or other peripherals such as Bluetooth speakers and other audio devices which can connect to the devices under test.

Access to a server or other device to send SMS/ MMS messages

Where it is required to send SMS/ MMS messages to the devices under test then it is important to ensure that the test lab supports this. Internet based servers can be used, or for smaller numbers of messages then sending can be carried out via other devices situated within the test lab.

Screen capture or screen recording facilities, provided by a video camera if the device OS does not allow screen capture

Capturing screen contents, particularly for inclusion in bug reports, is more difficult when testing mobile devices and applications. Some OS’s support screen capture and some do not, and it can be particularly difficult to capture screen grabs when reproducing complicated, timing dependant bugs since it is not then possible to stop the test in order to record the screen contents. For the reason it is advisable to equip the test lab with a small video camera which, together with a suitable white table top, can be used to video the screen and steps required in order to reproduce the bugs.

Keeping it up-to-date

Once a mobile test lab is setup and being used then it is important to ensure that its contents are regularly reviewed and updated. The pace of change in mobile is very fast, with new devices and OS versions being brought out frequently. Ensuring that the mobile test lab has the latest devices is important and regular reviewing of the contents is essential to ensuring continued use.

Putting It All Together

A mobile test lab can represent a significant financial outlay for a company and it is important that the money is spent wisely. Whilst it is possible to build an entire mobile test strategy around cloud based services, these are often slow and are currently not suitable to completely replace devices ‘in the hand’. Ensuring that there is a sufficient selection of devices is critical to ensuring that a mobile test lab adds value to an organisation, as is a suitable selection of different SIM cards and peripherals. Being able to replicate external situations such as low signal strength, handover between different cellular and Wi-Fi signals, and low battery life is also necessary in order to ensure that a mobile test strategy is comprehensive enough to cover these requirements, which are not typically tested for in the desktop world. Ensuring that a mobile test lab is suitably sized to support the testing team and applications under test is important. For simple applications then it may be enough to simply use personal devices, together with some facility to allow charging and screen capture, together with the ability to test low signal strengths. For more complicated applications, or where the application will be available on a wide number of OS versions or devices, then a more detailed approach will be required. A good mobile test lab adds value to the test organisation and the launch of a good quality mobile application is far less likely to succeed without one.

Gartner have recently published their Q4 2011 smartphone sales report. You can view it here: What does it tell us about the current state of the industry, and affect can this have on our work as testers in the mobile space?

Operating System	4Q11Units	4Q11 Market Share (%)	4Q10Units	4Q10 Market Share (%)
Android	75,906.1	50.9	30,801.2	30.5
iOS	35,456.0	23.8	16,011.1	15.8
Symbian	17,458.4	11.7	32,642.1	32.3
Research In Motion	13,184.5	8.8	14,762.0	14.6
Bada	3,111.3	2.1	2,026.8	2.0
Microsoft	2,759.0	1.9	3,419.3	3.4
Others	1,166.5	0.8	1,487.9	1.5
Total	149,041.8	100.0	101,150.3	100.0

Android Is Still King and iOS Is Increasing

Looking at the volume data, i.e. the number of devices running each platform, we can see a number of key points. Android increased market share from the same time last year, up to 50% of the worldwide market from only 30% a year ago. This represents a lot of devices. Android covers a wide price bracket, and devices based upon Android are available in a number of different form factors, display sizes and hardware configurations. The platform is clearly fragmenting even more significantly in order to covers these different needs.

Apple continued to increase market share, up to 24% of the market in Q4. The launch of the iPhone 4S has had an affect, as well as cost cutting of the earlier devices, a standard Apple launch policy.

As for the rest – Symbian decreased from 32% to 12%, primarily as a result of Nokia’s decision to announce the death of the platform and the knock on effect to consumer demand. RIM’s Blackberry OS lost ground, down to 9% from 14%. Microsoft’s WP platform didn’t fare well, only 1.9% of devices sold ran this OS.

What Does This Mean for Testers?

What’s very clear from the smartphone sales in Q4 is that there are still a large number of manufacturers in the market, and they are producing a lot of products. Growth was 47% year-on-year. This has some impacts for testing:

Mobile applications will become impossible to ignore for a lot of companies in 2012. That means mobile applications testing will become more and more important.
There are a lot of different device manufacturers and OS’s.
Testers will need to cover a lot of devices and test strategies will need to reflect this.
Anecdotal evidence is that the testing and QA efforts are not increasing to meet the current demand. Many applications are launched with serious bugs still present, indicating testing was not sufficient.

Probably the most difficult decision when designing a mobile test strategy is to decide the coverage of the OS and of the devices themselves. A typical mobile application launch strategy these days will focus on iOS and Android as the primary launch platforms for good reason – these are where the market share is, and therefore where the money is. Typically a third launch platform would then focus on Symbian, Blackberry or WP. For testers this means that there is a need to learn the skills and gain the experience with these platforms, and a clear focus on iOS and Android will initially be a good strategy.

For anyone who focuses on Android then there are additional challenges. The sheer variety of Android devices available now is staggering, from cheap, often un-licensed local brands right up to the flagship devices running the latest version OS version, Ice Cream Sandwich. This poses many issues for anyone developing mobile applications and those testing them – being able to cover all the different configurations is difficult and care is needed to ensure sufficient coverage. It can help to know the target market for the particular application, the sort of devices available to the customers and the expectations towards key criteria such as performance and usability, as well as just ensuring that the functional aspects of the application are OK.

Mobile Website Testing Needs Additional Focus

For those testing mobile websites then the problem becomes even larger. A mobile test strategy for a particular application can be designed around the particular OS that application is written for, and relatively easily adapted for other OS’s at a later date. The same cannot be said of a mobile website strategy. There’s a need to provide as much coverage as possible across a significantly wider selection of OS’s and devices, some outside of smartphone scope as well, in order to ensure that the website is functioning correctly and adequately displayed. Tools such as the validator from W3 http://validator.w3.org/mobile/ can help to some extent but they are not a substitute for real testing on a real device.

Covering Most Devices

Covering the largest selection of devices is very important. It is not easy. Companies such as SOASTA, PerfectoMobile and DeviceAnywhere are worth checking out, since they take away the burden of device ownership from the company or tester. Crowd sourced testing will become important as well, and there are a number of companies working in this space. But there is significant scope for mobile testers here, and a significant need for more testing than currently.

Overall we can see that the smartphone sector is still growing and the market for applications is increasing. Anecdotal evidence is that the testing and QA efforts are not increasing to meet this demand, and that’s where testers can play their part.

Stephen Janaway

Tag Archives: iOS

Fundamentals of a Mobile Testing Lab