Category Archives: Testing

Seeing the Wood For the Trees

I’ve recently been reading through my ISEB Practitioner notes, which I got when attending a course organised by Grove Consultants a few years back.

Don’t switch off yet. I know I mentioned ISEB. So before I go further, it’s worth stating my thoughts on the whole ISEB/ISTQB debate. I summed up my frustrations in a previous blog post; the fact that in the UK having ISTQB certification is practically the only way to get past the recruiters gate, but I also feel that there is some merit to the courses if taught properly, and followed up with a context driven approach such as Rapid Software Testing.

Reading back through my notes from Grove I can now see that this is what they were trying to work towards. At the time I had no idea how important the context driven school of testing was, nor the work of James Bach, Michael Bolton, Cem Kaner and others. But looking in the notes, the names are there. The techniques, albeit in nowhere near the detail that James or Michael teach in class of course, were hinted at and some approaches, particularly Exploratory Testing, are mentioned in some detail. Some slides are directly referenced from James and Michael’s work.

Unfortunately, due to the need to pass the exam, and with these areas being marked as ‘not exam’ then I didn’t pay them the attention that they warranted, and so it took a few more years to discover how and why the context driven approach can be so powerful. Which maybe shows the true problem with ISEB/ ISTQB certification after all.

More Experiences of Rapid Test Management

If you remember from the last post, I recently attended Rapid Test Management, taught by Michael Bolton in London. For a general overview and what happened in the first day then take a look at the previous post. As before, the following caveat applies:

Caveat – this is not a list of exactly what happens in a Rapid Test Management class, nor is it a list of exercises and course material. You won’t become good at Rapid Test Management by reading this post. Sometimes I forget things. If you really want to know about Rapid Test Management then you need to sign up for the class. It will be money well spent and this post will tell you a little about what might happen if you do take the class.

We started day two talking about test strategy and how one might incorporate Rapid Software Testing into a test strategy. The Heuristic Test Strategy model was introduced and we studied this in some detail, and followed it up with an exercise to define a strategy for a smartphone. I was happy with the choice of product under test, given my background then it made it easier to take into account my domain knowledge, and so our group could come up with a large number of options to consider. Equally interesting was what other groups had come up with and it was very interesting to see how each group had approached the problem differently and come up with different solutions. This goes to show that diversity in teams and experience is very important in testing.

Back on day 1 we had produced a list of areas that we, as a group, wanted the class to be focused upon and the major areas that so far had not been covered were risk and test coverage. As usual, Michael had some great experiences to share and course material to cover the risk areas and the details of risk based testing. In fact, the amount of class material was another one of the great things about this class – you get a lot, far more than you can look at or can be taught in the class itself. Together with some useful testing tools and also some more exercises and demo’s, this gives a great set of material to use afterwards. Continual learning is important and so to gain access to all the notes, examples and slides is just the start.

Day 2 was concluded with a look at test coverage and ways to visualise test coverage. Rapid Software Testing provides you with some great ways to do this, from the simple to the more complex. It got me thinking about how I might incorporate this into the Kanban project management processes that my own team use to manage our work.

So, should you sign-up for the class? You bet’cha. Rapid Test Management is a class that you should attend. If you want it to, and you take the time to study all the information and material that you receive, then it will make you a better tester and it will make you a better test manager. I’m on the way; I’m just beginning to take on-board all I’ve learnt and to read through the material we didn’t cover in class, but with everything I read I’m confident that it’s improving my skills.

Experiences of Rapid Test Management

Caveat – this is not a list of exactly what happens in a Rapid Test Management class, nor is it a list of exercises and course material. You won’t become good at Rapid Test Management by reading this post. Sometimes I forget things. If you really want to know about Rapid Test Management then you need to sign up for the class. It will be money well spent and this post will tell you a little about what might happen if you do take the class.

Last week I had the opportunity to attend one of the first Rapid Test Management classes, and certainly the first one to have been run in the UK. It was a great experience and the beginning of a lot more learning. There were attendees from a number of different areas of software testing and together I think we formed a very effective group, learning from each other and sharing experiences.

Rapid Test Management follows on from the Rapid Software Testing class that James Bach and Michael Bolton teach. I’d attended James’ class in Cambridge at the beginning of March and so, with the class material and my ideas rolling around in my head, I pitched up at the Hilton hotel in leafy Kensington (well almost Shepherds Bush actually) for two days of intensive learning from Michael.

Rapid Software Testing is a context driven technique, intended to enable testers to excel at our chosen craft. Since it focuses on testing as a craft itself, instead of merely on the production of test cases and then their use for testing (or it could be argued merely for checking) then this does put some questions in mind when considering how best to manage it. By way of some preparation, I noted down some of the questions that I had:

  • How can I best manage each testers time when adopting a Rapid Testing based strategy.
  • How does estimation fit in with the Rapid Software Testing ethos?
  • How can I sell the idea of Rapid Software Testing not only to those within my team who may be sceptical, but more importantly to those who manage me, who are most likely not only sceptical but also less knowledgeable about software testing in general?

Day 1

The class was split into two days, the first morning focused mostly on refreshing understanding of Rapid Software Testing and ensuring that, as a group, we had effectively collected our own desired outcomes from the class. It was good to spend some time going back over the methodology itself even though I had learnt it very recently. It was great that Michael seemed happy to tailor the class to the audience in any way that was necessary, taking a lot of time to ensure that everyone’s opinion could be heard.

What I found great about this class, and the RST course, are the little nuggets of information, stories and quotes that James and Michael pass on. Looking back through my notes from this class I find that’s what I mostly write down, quotes like:

 “Quality is value to someone who matters”

“Testers tell stories about products”

We also spoke in depth about the testers Elevator Pitch, that two minutes where you have to explain what testing is all about. Being able to explain and justify the role of testers and test teams is a critical skill and Rapid Software Testing gives you that skill.

When implementing any changes, such as introducing Rapid Test Management for example, then it’s critical to take into account the existing knowledge and processes that are already in use. Michael stressed the importance of ensuring that it’s not only the documented processes that are considered; in fact it’s the informal processes that are far more important to understand. He spoke about tacit vs explicit knowledge and how it is important for managers to be aware of the tacit knowledge in a test team and not merely the explicit knowledge.

We then took a look at how to put together a good team and how to train a new tester who was coming onto the team, through guidance and suitable heuristics. It was great to discuss with the others in the class about the issues they had faced when coming into a new team or bringing new team members in and also what had gone well. Ensuring that people are trained in a fault tolerant environment and gradually introduced to new tasks may seem fairly obvious stuff but it’s  often over-looked, as is the need to ensure that feedback is both given and asked for from new team members.

Day 1 concluded with a look at metrics and how important they really are, v.s. how important most organisations seem to think they are. Michael pointed us towards Cem Kaner and Walter Bond’s paper Software Engineering Metrics: What Do They Measure and How Do We Know which I would definitely recommend reading.

 

Find out what happened on day 2 soon…

Smartphone Sales Pass Feature Phone Sales in Japan for the First Time

Something interesting has just happened in the mobile market. You may have missed it or it may surprise you if you live in the US or Western Europe and are a middle class iPhone or Android owner.

What’s happened is that smartphone sales have passed feature phone sales for the first time. That may surprise you, you probably thought this happened years ago. After all, we’re all smartphone users now, right? Nope. And it didn’t: worldwide 70% of device sold are still feature phones (meaning cheaper devices running OS’s like Nokia’s S40 or other proprietary OS’s, typically with smaller screens and a lack of multi-tasking). Although the amount of money that manufacturers make on these devices is much smaller than smartphones, they ship in their millions and billions. Nokia recently shipped it’s 1.5 billionth S40 device for example.

You can read more in the MobiLens report which surveyed over 400 Japanese customers, and was compiled by market-watcher comScore  for the three months to February 2012. A quote from the report:

“Smartphones surpassed feature phones as the most acquired device type in February 2012, signalling an important shift in Japan’s mobile market,” said Daizo Nishitani, vice president of comScore Japan KK. “The rise in smartphone adoption opens the door to tremendous opportunity for publishers and advertisers to expand their reach and increase engagement with key consumer segments through this channel. Japanese mobile phone users were already highly engaged with their devices, but with the added functionality and higher levels of mobile media consumption we should expect to see significant changes in behaviour among the Japanese mobile population in 2012.”

Why Is This a Big Deal?

Japan typically leads the smartphone market and this is therefore a good indicator that slowly but surely the tide is beginning to turn towards smartphones in mature markets.

For testers this means more opportunity – smartphones typically mean a more open OS and therefore a significantly greater number of complicated applications that require testing. Feature phones are typically tested primarily by the manufacturers themselves; the only 3rd party runtime available is normally the Java ME platform and whilst there are a lot of applications launched written in Java (check out GetJar if you want some proof), there’s no evidence to suggest a large testing population at work ensuring that they work. Feature phones are also more likely to be lower powered, with smaller screens, ITU-T keyboards and generally lower spec without hardware like GPS.

However, with this move away from feature phones also comes further testing challenges; as the market switches to smartphones then it is inevitable that this will mean greater fragmentation of the OS’s themselves as manufacturers attempt to cover more and more price segments with different products. This will mean more display sizes, more hardware configurations and more differentiation in mechanics. For testers this will mean increased complication and mobile device testing strategies will need to evolve further than previously to cover a wider range of devices under test.

Also, as the market evolves then so does the installed base on devices. This presents additional challenges and further fragmentation issues. Ignore the devices in the field at your peril.

I’m Speaking At the Next Generation Testing Conference on 23rd May

I’m excited to be able to announce that I’ll be speaking, and sitting on a panel, at the Next Generation Testing Conference which is on Wednesday 23rd May in London.

The conference is in its seventh year and is trying out a new format this year, with more emphasis on panels and discussions, together with case studies. This should hopefully mean that there’s some great audience participation and discussion on software testing.

I’ll be on Panel 1: Testing Today – What are the main challenges? It’ll be moderated by Dr Richard Sykes and they’ll be a group of us representing various areas of software testing, including Tony Bruce of London Tester Gathering fame. We’ll be debating various topics including:

  • Testing in the cloud
  • Testing mobile applications
  • Testing Big Data migrations
  • Games Testing
  • Non-software systems testing

I’m then be presenting a case study: Mobile Testing – That’s Just a Smaller Screen, Right? This will go into the background behind mobile device testing, how it differs from the desktop world, and giving some pointers towards areas to consider when formulating a mobile device strategy.

Hope to see you there. You can find out more about the event at http://next-generation-testing.com/

 

Reversal?

I’ve been studying recently for a Chartered Management Institute Level 5 qualification. As my time in my current role comes to an end then it’s important to make sure that the management activities and experience that I have built up in my 9 years at Nokia is accurately reflected on my C.V. in a way that future recruiters will recognise. But the good thing is that through the course I have also been introduced to some techniques which are new to me, are useful for the future and can be applied to the software testing field as much as anywhere else.

Recently I learnt about a problem solving technique called reversal. It’s nothing new I know, but it appealed to my testers mind. It appealed to the part of me who wants to see how things could be made worse, how faults could be found, and how increased risk could be identified. The sort of mind-set that can equally be applied to software testing situations.

The reversals technique is a basic information based decision making technique. It breaks problem solving down into the following activities:

Identify the situation under study.

  • Consider how the situation could be made worse.
  • Create as many options as possible which will make the situation worse.
  • Reverse the options to identify ways of improving the situation.

As an example, consider a situation where a lack of road capacity has been identified. Using the technique, one could come up with options such as:

  • Close all the motorways.
  • Discourage homeworking.
  • Close alternative transport mechanisms.

Then by reversing these, one could come up with solutions which could improve the situation:

  • Build more motorways.
  • Encourage homeworking.
  • Build more alternative transport mechanisms.

A simple example I know, but you get the picture.

I think we can apply this to software testing as well. I recently read the blog post by Ilari Henrik Aegerter about What you Can Do if your Brain just Refuses to Understand and one addition I’d make to his list would be to consider some problem solving techniques. I find they help me unlock the locked parts of my brain. Reversal is one such technique.

Consider the situation where you know a bug exists but you just cannot reach the root cause. If you’ve run out of options when it comes to reproducing a bug, consider reversal. Think about what really will not cause the bug to happen, what combination of key presses or code paths that won’t help. Then reverse them. Consider a situation where you need to load test a system. What could you do that would actually make it faster? Then reverse them.

Hopefully it gives you a new perspective and another tool in the belt.

 

 

Experiences of Rapid Software Testing

Last week I had the pleasure of attending Rapid Software Testing training, organised by The Ministry of Testing. Rapid Software Testing is a technique popularised by James Bach and Michael Bolton and hopefully is not something new to you, but in case it is then I’d recommend looking at James Bach’s website. He explains it much better than me 🙂

The course was in Cambridge in the UK, and after a quick and easy train ride up then it was easy to find the hotel, dump the bags, and then go out to the Software Testing Club Meetup. This was a good start to the course, an initial networking event which James also attended, as well as a lot of local testers who were not attending the course. We had some great discussions and I met a lot of new people; you can see some photos from the event which have been posted up by Rosie, the organiser.

Day 1

Then to the first day of the course itself. From the moment the course started it was apparent that this was not your typical technical course. James’ style is well—known, just search YouTube if you want to see him in action, and he carried this into the course itself. He certainly knows his stuff, and presents in typical provocative style and is capable of causing many eureka moments. It’s very enjoyable, but initially tough, stuff.

We focused mostly in the first day on what Rapid Software Testing is, and the overall philosophy of the techniques. Rapid Software Testing is most useful to encourage testers to defend and think for themselves from a position of technical authority, especially useful in periods of uncertainty. Plenty of examples were given and James was able to draw upon his many years of experience in testing, both hardware and software. The time went by quickly and there was plenty of audience participation. James’ style is very much to put the audience members on the spot and ask some very difficult and blunt questions in order to replicate the pressures that testers can feel as part of project teams. To a few this comes naturally, but to most of the audience, this was a long way out of the comfort zone. We tried to help out whoever had been picked for a particular challenge, in order that the class as a whole could benefit.

The day concluded with an exercise on testing some everyday objects. Sounds simple, right? Well no. In case you will go on the course yourself then I will not give too much away, but suffice to say that there is much more to testing something which appears simple, than one at first thinks. It’s these sorts of exercises that open the mind and help learning.

Day 2

After a good dinner with some new software testing friends, and a decent night’s sleep, it was time for day 2. Here we went into more details of Rapid Software Testing and the relevant testing models. Again the examples given were general, intended to make you think like a tester irrespective of your background, and plenty of pressure was applied to those who James selected from the audience. We looked at the differences between scripted and un-scripted testing and exploded some myths about both areas. We also talked a lot about oracles and why they are essential in testing. As an example, I was surprised to find that a person can be considered as oracle.

We also discussed heuristics a lot. Rapid Software Testing has many heuristics, the fact that James can remember and explain them all straight from his head is somewhat impressive. As with a lot of the techniques and information, a fair amount of common sense thinking was clearly applied when inventing the heuristics, but it was good to get names put to techniques that I was using already, for consistency if nothing else. There is a danger of quoting too many heuristics of course, especially when dealing with other’s within project teams and management. James’ view seems to be that by bombarding those outside of testing with information and explanations, using the relevant heuristics, that testers gain legitimacy. I do not agree with his approach to the length that he presents it – clearly testers need to be able to explain themselves – but there is a danger of losing credibility if too many heuristics are invented and then explained, which merely represent ‘day-to-day’ work. Take a look at James’ slides and see if you agree.

By the end of the day we were questioning practically everything about testing and about the way we were working. There is a danger from this course that one starts to question too much but one needs to start small and work up I think. That’s certainly what I intend to do.

Day 3

The final day of the course started bright and early with more of the same. We focused on exploratory testing again, with more details, and talked a lot about documentation, metrics and information. The idea of focusing on a particular testing task, using some heuristics, but knowing when it is not working and de-focusing at this point, was a great learning for me. We also went into more detail on exploratory and session based techniques, something which I wish we had spent a bit more time on in previous days.

The main exercise for the day was based around finding a pattern for a system based upon dice. I won’t go into too many details on this (it’s explained pretty well at Better Testing) and also I do not want to give away a potential solution to anyone. But suffice to say it was a great opportunity to put into practice some of the techniques that we had learnt. Our group were not the quickest but neither were we the slowest, and it was certainly a good challenge.

The day then concluded with a wrap-up and overview of what we had learnt. Then some brief goodbyes and swapping of LinkedIn invites, and home to try and make sense of a busy three days and how what I had learnt could be applied to myself and the team members in my teams.

Overall

If you get the chance to go on Rapid Software Testing then go. Don’t think too hard about it, the course if very worthwhile and you will get a lot out of it. It is not easy, you will most likely feel uncomfortable at times with the training approach and some of the content may well seem obvious on first pass. But once you think more, and you start to question your own approach, with the techniques, tools, and even just the words, to back-up what you already know, then this course should make you a better tester. It would have been good to have seen a little more on session based techniques in detail and more about the tools that can be used, but I understand James does a separate course on this.

Thanks also go to Rosie Sherry, the course organiser. This was the first course that The Ministry of Testing have organised and if this first one is anything to go by then Ministry of Testing has a bright future. The venue and organisation was great, there was a really friendly, small company feel about things, and it was very easy to meet new people and learn together. Definitely three days well spent.

Rumours of the Death of the Test Case

The picture above shows Samuel Langhorne Clemens, better known by his pen name of Mark Twain. You may remember him as the author of Huckleberry Finn, and a number of other books. Why is he here in this post about testing on this blog about testing? Well Mark Twain was also the author of this famous quote, which came about when his obituary was published in an American newspaper while he was still very much alive:

The rumours of my death have been greatly exaggerated”. 

 Recently there has been a lot in the testing press, online and on social networks regarding the death of test cases. “Test cases are so 1980’s” was one tweet I saw. I looked at this and thought “Why?” Why are test cases suddenly out of date or dead, why is scripted testing suddenly seem as a bad thing? Are people just trying to be cool, thinking only exploratory testing or session based testing is needed and test cases are suddenly obsolete?

I think there is a place for both disciplines and it seems others agree with me. Sure, test cases that add no value, and test cases which are merely endlessly repeated by testers who are thinking more about what to have for dinner tonight than what they are testing, are of course worse than useless. They give you a false vision of quality, a sense that everything is OK when clearly it is not. Where testing should add value and provoke debate and opinion they merely suppress the discipline to ‘button pushing’. But the act of designing a good test case, of spending the time and brain power to ensure that the system under test is exercised in a clear, efficient and repeatable way can only be a good thing. It should not be reduced to something that is perceived to be out of date, stuck in the 80’s or worthless. It’s still a very valuable tool in the testers arsenal, allowing them to get an upfront view of the changing requirements of the system they are testing and to document in a clear and consistent way how they intend to test.

Scripted testing and exploratory techniques should sit hand-in-hand. Both add value. In my experience you need both. An illustration of context driven testing is The essential value of any test case lies in its ability to provide information (i.e. to reduce uncertainty).” A good test case does exactly that. It provides information. It can help show a system will be testable at a point where that system may not even be is testable. It enables others to learn the system and it provides a way of sharing information between those working on the system and outside of it. No two testers have the same level of experience. Test cases can enable consistency in the areas of a test strategy where that is needed. In the case of regulatory approvals and safety critical systems then they are mandatory anyway.

 

So let’s not let the test case become an old, out-of-date idea. Let’s make sure that it’s one technique we use as part of a well-rounded test strategy whose aim is to add value to whatever system we find ourselves testing.

Q4 Smartphone Sales – What Does It Mean for Testing?

Gartner have recently published their Q4 2011 smartphone sales report. You can view it here: What does it tell us about the current state of the industry, and affect can this have on our work as testers in the mobile space?

Operating System 4Q11Units 4Q11 Market Share (%) 4Q10Units 4Q10 Market Share (%)
Android 75,906.1 50.9 30,801.2 30.5
iOS 35,456.0 23.8 16,011.1 15.8
Symbian 17,458.4 11.7 32,642.1 32.3
Research In Motion 13,184.5 8.8 14,762.0 14.6
Bada 3,111.3 2.1 2,026.8 2.0
Microsoft 2,759.0 1.9 3,419.3 3.4
Others 1,166.5 0.8 1,487.9 1.5
Total 149,041.8 100.0 101,150.3 100.0

Android Is Still King and iOS Is Increasing

Looking at the volume data, i.e. the number of devices running each platform, we can see a number of key points. Android increased market share from the same time last year, up to 50% of the worldwide market from only 30% a year ago. This represents a lot of devices. Android covers a wide price bracket, and devices based upon Android are available in a number of different form factors, display sizes and hardware configurations. The platform is clearly fragmenting even more significantly in order to covers these different needs.

Apple continued to increase market share, up to 24% of the market in Q4. The launch of the iPhone 4S has had an affect, as well as cost cutting of the earlier devices, a standard Apple launch policy.

As for the rest – Symbian decreased from 32% to 12%, primarily as a result of Nokia’s decision to announce the death of the platform and the knock on effect to consumer demand. RIM’s Blackberry OS lost ground, down to 9% from 14%. Microsoft’s WP platform didn’t fare well, only 1.9% of devices sold ran this OS.

 

What Does This Mean for Testers?

What’s very clear from the smartphone sales in Q4 is that there are still a large number of manufacturers in the market, and they are producing a lot of products. Growth was 47% year-on-year. This has some impacts for testing:

 

  1. Mobile applications will become impossible to ignore for a lot of companies in 2012. That means mobile applications testing will become more and more important.
  2. There are a lot of different device manufacturers and OS’s.
  3. Testers will need to cover a lot of devices and test strategies will need to reflect this.
  4. Anecdotal evidence is that the testing and QA efforts are not increasing to meet the current demand. Many applications are launched with serious bugs still present, indicating testing was not sufficient.

 

Probably the most difficult decision when designing a mobile test strategy is to decide the coverage of the OS and of the devices themselves. A typical mobile application launch strategy these days will focus on iOS and Android as the primary launch platforms for good reason – these are where the market share is, and therefore where the money is. Typically a third launch platform would then focus on Symbian, Blackberry or WP. For testers this means that there is a need to learn the skills and gain the experience with these platforms, and a clear focus on iOS and Android will initially be a good strategy.

For anyone who focuses on Android then there are additional challenges. The sheer variety of Android devices available now is staggering, from cheap, often un-licensed local brands right up to the flagship devices running the latest version OS version,  Ice Cream Sandwich. This poses many issues for anyone developing mobile applications and those testing them – being able to cover all the different configurations is difficult and care is needed to ensure sufficient coverage. It can help to know the target market for the particular application, the sort of devices available to the customers and the expectations towards key criteria such as performance and usability, as well as just ensuring that the functional aspects of the application are OK.

 

Mobile Website Testing Needs Additional Focus

For those testing mobile websites then the problem becomes even larger. A mobile test strategy for a particular application can be designed around the particular OS that application is written for, and relatively easily adapted for other OS’s at a later date. The same cannot be said of a mobile website strategy. There’s a need to provide as much coverage as possible across a significantly wider selection of OS’s and devices, some outside of smartphone scope as well, in order to ensure that the website is functioning correctly and adequately displayed. Tools such as the validator from W3 http://validator.w3.org/mobile/ can help to some extent but they are not a substitute for real testing on a real device.

 

Covering Most Devices

Covering the largest selection of devices is very important. It is not easy. Companies such as SOASTA, PerfectoMobile and DeviceAnywhere are worth checking out, since they take away the burden of device ownership from the company or tester. Crowd sourced testing will become important as well, and there are a number of companies working in this space. But there is significant scope for mobile testers here, and a significant need for more testing than currently.

Overall we can see that the smartphone sector is still growing and the market for applications is increasing. Anecdotal evidence is that the testing and QA efforts are not increasing to meet this demand, and that’s where testers can play their part.

My First STC Meetup

Meeting New People

Last night I went to my first Software Testing Club Meetup in Guildford, Surrey.

What’s A Software Testing Club Meetup?

A Software Testing Club Meetup is simply a meeting of like-minded people. Mostly testers 🙂 It’s an opportunity to talk testing and normally have a beer or two. And in this case it was pie night in The Keystone so there was some great food as well.

So What Happened?

It was a very enjoyable evening. There were a great bunch of people there. ‘Hello’ everyone at IDBS, and all the other people that I met, talked testing and ate pies with. This was the first meetup that’s been done in this area, and also the first time I’ve gone on my own to anything like this, but it was really easy to settle in and get chatting. We had some good discussions about Selenium, software testing certification, and I got some great tips on freelancing from Alan at StoryIQ.

If you’ve never been to an event like this then you really should go. People are friendly, and it’s a great place outside of work to talk, and to find out experiences from other parts of the industry and from testers working in other companies.

Rather stupidly I forgot my camera, but fortunately other’s didn’t so there’s some pictures here.

Thanks to Lynda for organising, and I hope there’s another one soon.