Category Archives: Experiences

Q4 Smartphone Sales – What Does It Mean for Testing?

Gartner have recently published their Q4 2011 smartphone sales report. You can view it here: What does it tell us about the current state of the industry, and affect can this have on our work as testers in the mobile space?

Operating System 4Q11Units 4Q11 Market Share (%) 4Q10Units 4Q10 Market Share (%)
Android 75,906.1 50.9 30,801.2 30.5
iOS 35,456.0 23.8 16,011.1 15.8
Symbian 17,458.4 11.7 32,642.1 32.3
Research In Motion 13,184.5 8.8 14,762.0 14.6
Bada 3,111.3 2.1 2,026.8 2.0
Microsoft 2,759.0 1.9 3,419.3 3.4
Others 1,166.5 0.8 1,487.9 1.5
Total 149,041.8 100.0 101,150.3 100.0

Android Is Still King and iOS Is Increasing

Looking at the volume data, i.e. the number of devices running each platform, we can see a number of key points. Android increased market share from the same time last year, up to 50% of the worldwide market from only 30% a year ago. This represents a lot of devices. Android covers a wide price bracket, and devices based upon Android are available in a number of different form factors, display sizes and hardware configurations. The platform is clearly fragmenting even more significantly in order to covers these different needs.

Apple continued to increase market share, up to 24% of the market in Q4. The launch of the iPhone 4S has had an affect, as well as cost cutting of the earlier devices, a standard Apple launch policy.

As for the rest – Symbian decreased from 32% to 12%, primarily as a result of Nokia’s decision to announce the death of the platform and the knock on effect to consumer demand. RIM’s Blackberry OS lost ground, down to 9% from 14%. Microsoft’s WP platform didn’t fare well, only 1.9% of devices sold ran this OS.

 

What Does This Mean for Testers?

What’s very clear from the smartphone sales in Q4 is that there are still a large number of manufacturers in the market, and they are producing a lot of products. Growth was 47% year-on-year. This has some impacts for testing:

 

  1. Mobile applications will become impossible to ignore for a lot of companies in 2012. That means mobile applications testing will become more and more important.
  2. There are a lot of different device manufacturers and OS’s.
  3. Testers will need to cover a lot of devices and test strategies will need to reflect this.
  4. Anecdotal evidence is that the testing and QA efforts are not increasing to meet the current demand. Many applications are launched with serious bugs still present, indicating testing was not sufficient.

 

Probably the most difficult decision when designing a mobile test strategy is to decide the coverage of the OS and of the devices themselves. A typical mobile application launch strategy these days will focus on iOS and Android as the primary launch platforms for good reason – these are where the market share is, and therefore where the money is. Typically a third launch platform would then focus on Symbian, Blackberry or WP. For testers this means that there is a need to learn the skills and gain the experience with these platforms, and a clear focus on iOS and Android will initially be a good strategy.

For anyone who focuses on Android then there are additional challenges. The sheer variety of Android devices available now is staggering, from cheap, often un-licensed local brands right up to the flagship devices running the latest version OS version,  Ice Cream Sandwich. This poses many issues for anyone developing mobile applications and those testing them – being able to cover all the different configurations is difficult and care is needed to ensure sufficient coverage. It can help to know the target market for the particular application, the sort of devices available to the customers and the expectations towards key criteria such as performance and usability, as well as just ensuring that the functional aspects of the application are OK.

 

Mobile Website Testing Needs Additional Focus

For those testing mobile websites then the problem becomes even larger. A mobile test strategy for a particular application can be designed around the particular OS that application is written for, and relatively easily adapted for other OS’s at a later date. The same cannot be said of a mobile website strategy. There’s a need to provide as much coverage as possible across a significantly wider selection of OS’s and devices, some outside of smartphone scope as well, in order to ensure that the website is functioning correctly and adequately displayed. Tools such as the validator from W3 http://validator.w3.org/mobile/ can help to some extent but they are not a substitute for real testing on a real device.

 

Covering Most Devices

Covering the largest selection of devices is very important. It is not easy. Companies such as SOASTA, PerfectoMobile and DeviceAnywhere are worth checking out, since they take away the burden of device ownership from the company or tester. Crowd sourced testing will become important as well, and there are a number of companies working in this space. But there is significant scope for mobile testers here, and a significant need for more testing than currently.

Overall we can see that the smartphone sector is still growing and the market for applications is increasing. Anecdotal evidence is that the testing and QA efforts are not increasing to meet this demand, and that’s where testers can play their part.

My First STC Meetup

Meeting New People

Last night I went to my first Software Testing Club Meetup in Guildford, Surrey.

What’s A Software Testing Club Meetup?

A Software Testing Club Meetup is simply a meeting of like-minded people. Mostly testers 🙂 It’s an opportunity to talk testing and normally have a beer or two. And in this case it was pie night in The Keystone so there was some great food as well.

So What Happened?

It was a very enjoyable evening. There were a great bunch of people there. ‘Hello’ everyone at IDBS, and all the other people that I met, talked testing and ate pies with. This was the first meetup that’s been done in this area, and also the first time I’ve gone on my own to anything like this, but it was really easy to settle in and get chatting. We had some good discussions about Selenium, software testing certification, and I got some great tips on freelancing from Alan at StoryIQ.

If you’ve never been to an event like this then you really should go. People are friendly, and it’s a great place outside of work to talk, and to find out experiences from other parts of the industry and from testers working in other companies.

Rather stupidly I forgot my camera, but fortunately other’s didn’t so there’s some pictures here.

Thanks to Lynda for organising, and I hope there’s another one soon.

!(Certification) = !(New Job)

There’s a lot said in the testing press and blogs about certification. There’s some well known haters of ISTQB and a few, albeit quieter, exponents. There is of course the training providers shouting loudly about their guarenteed pass rates, how their courses are faster than all the others, and how you won’t survive in testing without the qualifications that you can get from them. Is certification as important as they say? I’m beginning to think that maybe it is, but not for the reasons their sales people present.

Firstly some background. I’m ISEB Foundation and Practitioner certified. I enjoyed the courses which I did with the excellent Grove Consultants a few years ago. OK, the exams were not fun but the courses were. I felt like I learned something and I went along because I wanted to learn. The qualification was good, but secondary. I felt it wasn’t essential. I still feel this way, I’m not an out-and-out ISTQB basher but I feel things are beginning to go too far.

Once I became a team leader, and then a test manager I continued to send people on the courses. Some didn’t want to go, but I felt it was important for them to learn something new, and more importantly to learn the same way, and using the same information, that the rest of the team had already learnt. It gave some consistency. That was useful.

Fast forward a few years. I now have a team of testers and delivery ops people. Times have been hard and training has been hard to come by, by the time these people joined the team there was no training available that would lead to the ISTQB/ ISEB certifications. Has the quality of what we do decreased? Well, no. If anything, we’ve gone out and trained ourselves, trained ourselves, and updated our ways-of-working in even better ways. We are still consistent in our approach, and as a bonus, some people can now train others. Also a good skill. Not getting the ISTQB training has annoyed some, whilst others weren’t bothered at all.

Now my team and I find ourselves in a new situation. Soon we will all lose our jobs as R&D is moved overseas. Suddenly the issue of certification slams itself forward again. Most of the job ads scream ISTQB certified, for recruiters it’s almost the first question asked “Are you ISTQB certified?”. How have we come to this?

I think a lot of the testing community is stuck in a vicious circle. If we get lazy with our recruitment then we quickly fall into a trap of just putting “ISTQB certified” in the “Essential Requirements” section of our job ads. We are the ones who caused the recruiters to ask “Are you ISTQB certified?” Certification within the industry becomes self fulfilling. And those of us recruiting testers don’t necessarily get better testers.

So what’s the solution? More certification? I think all those of us who recruit for software testers need to re-visit what we look for in a tester, to adjust our outlook and our requirements so that we are trying to find those who are good at what they do, not what they have studied. A few years ago I used to run a written interview test for candidates which was based on the ISTQB syllabus. Many of those with the qualification failed.

And to my team, without certification and needing to find new jobs? I’ve sent them on ISTQB courses. It’s only fair, they need the best start they can get in their job searches. But if I find myself in this situation again then I hope that it’s not this way….

* For those of you without any programming knowldge – ! in the title means “Not” 🙂

Image: jscreationzs / FreeDigitalPhotos.net

Some Brief Words On ET for Mobiles

No not the short, funny coloured alien guy….

Yesterday I attended a exploratory testing session for the latest change that my team are testing, on new feature phones. So we’re talking mass market devices, ones where a bug let out into the field can wipe out your profit and destroy your reputation. So we like to get our QA right, you could say.

Overall the session went well. We found some bugs even though the feature under test is pretty close to release maturity now. There was a great mix of developers, testers and some of the release ops team (aka the CI guys). We like to use these sessions to bring the team together. Using Agile sdlc’s helps here anyway, but having everyone testing together and helping each other is great for the team spirit and the togetherness. It was good fun.

So here’s what went well and what to try differently next time:

* Make sure you do not limit your sessions to just the testers in your company. Get everyone together, it’s fun.
* The leader of the session is important to keep things flowing. Our test lead brought ready flashed phones. And food. That helped.
* Make sure you have a structure to the session. We used charters.

And some areas to learn from and change:

* Getting results and statistics from the session wasn’t easy. Next time we will try session based (as I’ve used in other teams).
* Get some commitment from participants and make sure they come on time. People who drift in and out break the focus of those testing.
* Make sure there are chargers and SIMs with relevant features available.

Overall it was most enjoyable. I can’t wait for the next one.

*and in the true spirit of mobile, I’ve written this whole post on a Nokia E7 using cutepress. Yep, I’m still a Nokia user, and still mobile obsessed 🙂