Tag Archives: testing

blog, mobile, Testing

Smartphone Sales Pass Feature Phone Sales in Japan for the First Time

Leave a comment

Something interesting has just happened in the mobile market. You may have missed it or it may surprise you if you live in the US or Western Europe and are a middle class iPhone or Android owner.

What’s happened is that smartphone sales have passed feature phone sales for the first time. That may surprise you, you probably thought this happened years ago. After all, we’re all smartphone users now, right? Nope. And it didn’t: worldwide 70% of device sold are still feature phones (meaning cheaper devices running OS’s like Nokia’s S40 or other proprietary OS’s, typically with smaller screens and a lack of multi-tasking). Although the amount of money that manufacturers make on these devices is much smaller than smartphones, they ship in their millions and billions. Nokia recently shipped it’s 1.5 billionth S40 device for example.

You can read more in the MobiLens report which surveyed over 400 Japanese customers, and was compiled by market-watcher comScore  for the three months to February 2012. A quote from the report:

“Smartphones surpassed feature phones as the most acquired device type in February 2012, signalling an important shift in Japan’s mobile market,” said Daizo Nishitani, vice president of comScore Japan KK. “The rise in smartphone adoption opens the door to tremendous opportunity for publishers and advertisers to expand their reach and increase engagement with key consumer segments through this channel. Japanese mobile phone users were already highly engaged with their devices, but with the added functionality and higher levels of mobile media consumption we should expect to see significant changes in behaviour among the Japanese mobile population in 2012.”

Why Is This a Big Deal?

Japan typically leads the smartphone market and this is therefore a good indicator that slowly but surely the tide is beginning to turn towards smartphones in mature markets.

For testers this means more opportunity – smartphones typically mean a more open OS and therefore a significantly greater number of complicated applications that require testing. Feature phones are typically tested primarily by the manufacturers themselves; the only 3rd party runtime available is normally the Java ME platform and whilst there are a lot of applications launched written in Java (check out GetJar if you want some proof), there’s no evidence to suggest a large testing population at work ensuring that they work. Feature phones are also more likely to be lower powered, with smaller screens, ITU-T keyboards and generally lower spec without hardware like GPS.

However, with this move away from feature phones also comes further testing challenges; as the market switches to smartphones then it is inevitable that this will mean greater fragmentation of the OS’s themselves as manufacturers attempt to cover more and more price segments with different products. This will mean more display sizes, more hardware configurations and more differentiation in mechanics. For testers this will mean increased complication and mobile device testing strategies will need to evolve further than previously to cover a wider range of devices under test.

Also, as the market evolves then so does the installed base on devices. This presents additional challenges and further fragmentation issues. Ignore the devices in the field at your peril.

management, mobile, Testing

I’m Speaking At the Next Generation Testing Conference on 23rd May

Leave a comment

I’m excited to be able to announce that I’ll be speaking, and sitting on a panel, at the Next Generation Testing Conference which is on Wednesday 23rd May in London.

The conference is in its seventh year and is trying out a new format this year, with more emphasis on panels and discussions, together with case studies. This should hopefully mean that there’s some great audience participation and discussion on software testing.

I’ll be on Panel 1: Testing Today – What are the main challenges? It’ll be moderated by Dr Richard Sykes and they’ll be a group of us representing various areas of software testing, including Tony Bruce of London Tester Gathering fame. We’ll be debating various topics including:

  • Testing in the cloud
  • Testing mobile applications
  • Testing Big Data migrations
  • Games Testing
  • Non-software systems testing

I’m then be presenting a case study: Mobile Testing – That’s Just a Smaller Screen, Right? This will go into the background behind mobile device testing, how it differs from the desktop world, and giving some pointers towards areas to consider when formulating a mobile device strategy.

Hope to see you there. You can find out more about the event at http://next-generation-testing.com/

 

"rapid software testing", blog, management

Test Leadership vs. Test Management – Is the Balance Right?

3 Comments

As I’ve blogged about recently, I’ve been recently studying for the Level 5 Certificate in Management and Leadership from the Chartered Management Institute. I’m now three sessions into the four session course, and it’s getting really interesting to see and think about how one might apply general management principles more specifically to the software testing area.

The most recent session was all about Management and Leadership. As part of this we did an exercise where we had to sort out a number of different phrases into groups that either apply to ‘Management’ or ‘Leadership’. No sitting on the fence, no spending hours deciding which to put where, but a quick exercise to make you think. There  were right and wrong answers (although primarily to seed further discussion).

The list, as I saw it, is below. What do you think? Do you agree?

Leadership Management
Enables others Implements and maintains
Inspires vision Focus on systems and structures
Encourages head and heart Adopt short term view
Acts authentically Completes transactions
Asks what and why Asks how and when
Has long range perspective Brings order and coordination
Focuses on doing the right thing Focuses on doing things right
Inspires trust Accomplishes tasks through others
Acts as an innovator Focuses on performance
Challenges Provides stability
Transforms Controls
Committed to the cause Imitates
Gives purpose and meaning Complies
Focus on people

After the exercise I got thinking about how I might apply this more generally to software testing. Specifically to Test Managements vs. Test Leadership, i.e. what separates those who run testing projects, probably as part of the design and delivery of a specific product, vs. those who lead groups and provide inspiration both inside those groups, and to the wider testing community. Are the skills needed in those situations different?

We clearly need within our community those who can challenge and innovate. Right now this is coming mostly from the context-driven community as I see it, where also the vision and long range perspective is clearly evident. These people are driving things along nicely in my opinion. As someone who has recently attending Rapid Software Testing with James Bach then I can say first hand that he’s certainly encouraging head and heart. I don’t need to mention challenging, right?

The question then comes, who is taking things further? Leaders need managers to take their vision and turn it into practice. They need someone who can focus on the short term view, give the control and ensure completion. They need people who implement. As a community do we currently have enough of the managers listening and implementing the vision that our leaders are sharing?

blog, management, Testing

Reversal?

2 Comments

I’ve been studying recently for a Chartered Management Institute Level 5 qualification. As my time in my current role comes to an end then it’s important to make sure that the management activities and experience that I have built up in my 9 years at Nokia is accurately reflected on my C.V. in a way that future recruiters will recognise. But the good thing is that through the course I have also been introduced to some techniques which are new to me, are useful for the future and can be applied to the software testing field as much as anywhere else.

Recently I learnt about a problem solving technique called reversal. It’s nothing new I know, but it appealed to my testers mind. It appealed to the part of me who wants to see how things could be made worse, how faults could be found, and how increased risk could be identified. The sort of mind-set that can equally be applied to software testing situations.

The reversals technique is a basic information based decision making technique. It breaks problem solving down into the following activities:

Identify the situation under study.

  • Consider how the situation could be made worse.
  • Create as many options as possible which will make the situation worse.
  • Reverse the options to identify ways of improving the situation.

As an example, consider a situation where a lack of road capacity has been identified. Using the technique, one could come up with options such as:

  • Close all the motorways.
  • Discourage homeworking.
  • Close alternative transport mechanisms.

Then by reversing these, one could come up with solutions which could improve the situation:

  • Build more motorways.
  • Encourage homeworking.
  • Build more alternative transport mechanisms.

A simple example I know, but you get the picture.

I think we can apply this to software testing as well. I recently read the blog post by Ilari Henrik Aegerter about What you Can Do if your Brain just Refuses to Understand and one addition I’d make to his list would be to consider some problem solving techniques. I find they help me unlock the locked parts of my brain. Reversal is one such technique.

Consider the situation where you know a bug exists but you just cannot reach the root cause. If you’ve run out of options when it comes to reproducing a bug, consider reversal. Think about what really will not cause the bug to happen, what combination of key presses or code paths that won’t help. Then reverse them. Consider a situation where you need to load test a system. What could you do that would actually make it faster? Then reverse them.

Hopefully it gives you a new perspective and another tool in the belt.

 

 

"ministry of testing", "rapid software testing", blog, Experiences

Learning Time

1 Comment

Tomorrow I’m leaving the safety of the South to journey far North* for something that I’ve been looking forward to for a long time. I’ve been fortunate enough to be able to sign-up for James Bach’s Rapid Software Testing course which has been arranged by The Ministry of Testing in Cambridge.

To say I’ve been looking forward to this for a while would be an understatement. Unfortunately due to budget constraints then it’s not been possible for me to get on courses like this in the past few years but ‘fortunately’ now that things are closing, then there’s a bit more money available for training and this will be money very well spent.

First stop is the Software Testing Club meetup tomorrow in Cambridge then on Wednesday the course starts. I don’t know what to expect but if it’s three days of hard but rewarding learning then I will be very happy. Having already taken a look at the course outline and slides then I’m sure it will be.

I’ll try and blog daily about my experiences, assuming I have the time and brain power left to do so 🙂

 

* (non-UK readers – we have a big North-South joke thing going on in the UK. If a place is north of Watford, which is a bit north of London, then us native southerners joke we’re out of the safety of the south and that it’s grim up north 🙂 Even though it isn’t and Cambridge is not even in really in the north anyway).

blog, Testing

Rumours of the Death of the Test Case

Leave a comment

The picture above shows Samuel Langhorne Clemens, better known by his pen name of Mark Twain. You may remember him as the author of Huckleberry Finn, and a number of other books. Why is he here in this post about testing on this blog about testing? Well Mark Twain was also the author of this famous quote, which came about when his obituary was published in an American newspaper while he was still very much alive:

“The rumours of my death have been greatly exaggerated”. 

 Recently there has been a lot in the testing press, online and on social networks regarding the death of test cases. “Test cases are so 1980’s” was one tweet I saw. I looked at this and thought “Why?” Why are test cases suddenly out of date or dead, why is scripted testing suddenly seem as a bad thing? Are people just trying to be cool, thinking only exploratory testing or session based testing is needed and test cases are suddenly obsolete?

I think there is a place for both disciplines and it seems others agree with me. Sure, test cases that add no value, and test cases which are merely endlessly repeated by testers who are thinking more about what to have for dinner tonight than what they are testing, are of course worse than useless. They give you a false vision of quality, a sense that everything is OK when clearly it is not. Where testing should add value and provoke debate and opinion they merely suppress the discipline to ‘button pushing’. But the act of designing a good test case, of spending the time and brain power to ensure that the system under test is exercised in a clear, efficient and repeatable way can only be a good thing. It should not be reduced to something that is perceived to be out of date, stuck in the 80’s or worthless. It’s still a very valuable tool in the testers arsenal, allowing them to get an upfront view of the changing requirements of the system they are testing and to document in a clear and consistent way how they intend to test.

Scripted testing and exploratory techniques should sit hand-in-hand. Both add value. In my experience you need both. An illustration of context driven testing is “The essential value of any test case lies in its ability to provide information (i.e. to reduce uncertainty).” A good test case does exactly that. It provides information. It can help show a system will be testable at a point where that system may not even be is testable. It enables others to learn the system and it provides a way of sharing information between those working on the system and outside of it. No two testers have the same level of experience. Test cases can enable consistency in the areas of a test strategy where that is needed. In the case of regulatory approvals and safety critical systems then they are mandatory anyway.

 

So let’s not let the test case become an old, out-of-date idea. Let’s make sure that it’s one technique we use as part of a well-rounded test strategy whose aim is to add value to whatever system we find ourselves testing.

blog, Experiences, mobile, Testing

Q4 Smartphone Sales – What Does It Mean for Testing?

Leave a comment

Gartner have recently published their Q4 2011 smartphone sales report. You can view it here: What does it tell us about the current state of the industry, and affect can this have on our work as testers in the mobile space?

Operating System 4Q11Units 4Q11 Market Share (%) 4Q10Units 4Q10 Market Share (%)
Android 75,906.1 50.9 30,801.2 30.5
iOS 35,456.0 23.8 16,011.1 15.8
Symbian 17,458.4 11.7 32,642.1 32.3
Research In Motion 13,184.5 8.8 14,762.0 14.6
Bada 3,111.3 2.1 2,026.8 2.0
Microsoft 2,759.0 1.9 3,419.3 3.4
Others 1,166.5 0.8 1,487.9 1.5
Total 149,041.8 100.0 101,150.3 100.0

Android Is Still King and iOS Is Increasing

Looking at the volume data, i.e. the number of devices running each platform, we can see a number of key points. Android increased market share from the same time last year, up to 50% of the worldwide market from only 30% a year ago. This represents a lot of devices. Android covers a wide price bracket, and devices based upon Android are available in a number of different form factors, display sizes and hardware configurations. The platform is clearly fragmenting even more significantly in order to covers these different needs.

Apple continued to increase market share, up to 24% of the market in Q4. The launch of the iPhone 4S has had an affect, as well as cost cutting of the earlier devices, a standard Apple launch policy.

As for the rest – Symbian decreased from 32% to 12%, primarily as a result of Nokia’s decision to announce the death of the platform and the knock on effect to consumer demand. RIM’s Blackberry OS lost ground, down to 9% from 14%. Microsoft’s WP platform didn’t fare well, only 1.9% of devices sold ran this OS.

 

What Does This Mean for Testers?

What’s very clear from the smartphone sales in Q4 is that there are still a large number of manufacturers in the market, and they are producing a lot of products. Growth was 47% year-on-year. This has some impacts for testing:

 

  1. Mobile applications will become impossible to ignore for a lot of companies in 2012. That means mobile applications testing will become more and more important.
  2. There are a lot of different device manufacturers and OS’s.
  3. Testers will need to cover a lot of devices and test strategies will need to reflect this.
  4. Anecdotal evidence is that the testing and QA efforts are not increasing to meet the current demand. Many applications are launched with serious bugs still present, indicating testing was not sufficient.

 

Probably the most difficult decision when designing a mobile test strategy is to decide the coverage of the OS and of the devices themselves. A typical mobile application launch strategy these days will focus on iOS and Android as the primary launch platforms for good reason – these are where the market share is, and therefore where the money is. Typically a third launch platform would then focus on Symbian, Blackberry or WP. For testers this means that there is a need to learn the skills and gain the experience with these platforms, and a clear focus on iOS and Android will initially be a good strategy.

For anyone who focuses on Android then there are additional challenges. The sheer variety of Android devices available now is staggering, from cheap, often un-licensed local brands right up to the flagship devices running the latest version OS version,  Ice Cream Sandwich. This poses many issues for anyone developing mobile applications and those testing them – being able to cover all the different configurations is difficult and care is needed to ensure sufficient coverage. It can help to know the target market for the particular application, the sort of devices available to the customers and the expectations towards key criteria such as performance and usability, as well as just ensuring that the functional aspects of the application are OK.

 

Mobile Website Testing Needs Additional Focus

For those testing mobile websites then the problem becomes even larger. A mobile test strategy for a particular application can be designed around the particular OS that application is written for, and relatively easily adapted for other OS’s at a later date. The same cannot be said of a mobile website strategy. There’s a need to provide as much coverage as possible across a significantly wider selection of OS’s and devices, some outside of smartphone scope as well, in order to ensure that the website is functioning correctly and adequately displayed. Tools such as the validator from W3 http://validator.w3.org/mobile/ can help to some extent but they are not a substitute for real testing on a real device.

 

Covering Most Devices

Covering the largest selection of devices is very important. It is not easy. Companies such as SOASTA, PerfectoMobile and DeviceAnywhere are worth checking out, since they take away the burden of device ownership from the company or tester. Crowd sourced testing will become important as well, and there are a number of companies working in this space. But there is significant scope for mobile testers here, and a significant need for more testing than currently.

Overall we can see that the smartphone sector is still growing and the market for applications is increasing. Anecdotal evidence is that the testing and QA efforts are not increasing to meet this demand, and that’s where testers can play their part.

Experiences, Testing

My First STC Meetup

Leave a comment

Meeting New People

Last night I went to my first Software Testing Club Meetup in Guildford, Surrey.

What’s A Software Testing Club Meetup?

A Software Testing Club Meetup is simply a meeting of like-minded people. Mostly testers 🙂 It’s an opportunity to talk testing and normally have a beer or two. And in this case it was pie night in The Keystone so there was some great food as well.

So What Happened?

It was a very enjoyable evening. There were a great bunch of people there. ‘Hello’ everyone at IDBS, and all the other people that I met, talked testing and ate pies with. This was the first meetup that’s been done in this area, and also the first time I’ve gone on my own to anything like this, but it was really easy to settle in and get chatting. We had some good discussions about Selenium, software testing certification, and I got some great tips on freelancing from Alan at StoryIQ.

If you’ve never been to an event like this then you really should go. People are friendly, and it’s a great place outside of work to talk, and to find out experiences from other parts of the industry and from testers working in other companies.

Rather stupidly I forgot my camera, but fortunately other’s didn’t so there’s some pictures here.

Thanks to Lynda for organising, and I hope there’s another one soon.

blog, Experiences, management, practices, team, Testing

!(Certification) = !(New Job)

16 Comments

There’s a lot said in the testing press and blogs about certification. There’s some well known haters of ISTQB and a few, albeit quieter, exponents. There is of course the training providers shouting loudly about their guarenteed pass rates, how their courses are faster than all the others, and how you won’t survive in testing without the qualifications that you can get from them. Is certification as important as they say? I’m beginning to think that maybe it is, but not for the reasons their sales people present.

Firstly some background. I’m ISEB Foundation and Practitioner certified. I enjoyed the courses which I did with the excellent Grove Consultants a few years ago. OK, the exams were not fun but the courses were. I felt like I learned something and I went along because I wanted to learn. The qualification was good, but secondary. I felt it wasn’t essential. I still feel this way, I’m not an out-and-out ISTQB basher but I feel things are beginning to go too far.

Once I became a team leader, and then a test manager I continued to send people on the courses. Some didn’t want to go, but I felt it was important for them to learn something new, and more importantly to learn the same way, and using the same information, that the rest of the team had already learnt. It gave some consistency. That was useful.

Fast forward a few years. I now have a team of testers and delivery ops people. Times have been hard and training has been hard to come by, by the time these people joined the team there was no training available that would lead to the ISTQB/ ISEB certifications. Has the quality of what we do decreased? Well, no. If anything, we’ve gone out and trained ourselves, trained ourselves, and updated our ways-of-working in even better ways. We are still consistent in our approach, and as a bonus, some people can now train others. Also a good skill. Not getting the ISTQB training has annoyed some, whilst others weren’t bothered at all.

Now my team and I find ourselves in a new situation. Soon we will all lose our jobs as R&D is moved overseas. Suddenly the issue of certification slams itself forward again. Most of the job ads scream ISTQB certified, for recruiters it’s almost the first question asked “Are you ISTQB certified?”. How have we come to this?

I think a lot of the testing community is stuck in a vicious circle. If we get lazy with our recruitment then we quickly fall into a trap of just putting “ISTQB certified” in the “Essential Requirements” section of our job ads. We are the ones who caused the recruiters to ask “Are you ISTQB certified?” Certification within the industry becomes self fulfilling. And those of us recruiting testers don’t necessarily get better testers.

So what’s the solution? More certification? I think all those of us who recruit for software testers need to re-visit what we look for in a tester, to adjust our outlook and our requirements so that we are trying to find those who are good at what they do, not what they have studied. A few years ago I used to run a written interview test for candidates which was based on the ISTQB syllabus. Many of those with the qualification failed.

And to my team, without certification and needing to find new jobs? I’ve sent them on ISTQB courses. It’s only fair, they need the best start they can get in their job searches. But if I find myself in this situation again then I hope that it’s not this way….

* For those of you without any programming knowldge – ! in the title means “Not” 🙂

Image: jscreationzs / FreeDigitalPhotos.net

blog, Experiences, Exploratory Testing, Testing

Some Brief Words On ET for Mobiles

Leave a comment

No not the short, funny coloured alien guy….

Yesterday I attended a exploratory testing session for the latest change that my team are testing, on new feature phones. So we’re talking mass market devices, ones where a bug let out into the field can wipe out your profit and destroy your reputation. So we like to get our QA right, you could say.

Overall the session went well. We found some bugs even though the feature under test is pretty close to release maturity now. There was a great mix of developers, testers and some of the release ops team (aka the CI guys). We like to use these sessions to bring the team together. Using Agile sdlc’s helps here anyway, but having everyone testing together and helping each other is great for the team spirit and the togetherness. It was good fun.

So here’s what went well and what to try differently next time:

* Make sure you do not limit your sessions to just the testers in your company. Get everyone together, it’s fun.
* The leader of the session is important to keep things flowing. Our test lead brought ready flashed phones. And food. That helped.
* Make sure you have a structure to the session. We used charters.

And some areas to learn from and change:

* Getting results and statistics from the session wasn’t easy. Next time we will try session based (as I’ve used in other teams).
* Get some commitment from participants and make sure they come on time. People who drift in and out break the focus of those testing.
* Make sure there are chargers and SIMs with relevant features available.

Overall it was most enjoyable. I can’t wait for the next one.

*and in the true spirit of mobile, I’ve written this whole post on a Nokia E7 using cutepress. Yep, I’m still a Nokia user, and still mobile obsessed 🙂