I’ve been fortunate enough to have had an article published in the latest edition of Testing Experience magazine. It’s called “Beyond the Smartphone – Testing for the Developing World” and focuses on techniques and tools to use when testing feature phones.
You can download the whole magazine for free by visiting their website and subscribing to the online edition.
Hope you enjoy it.
Although there can be no doubt that testing mobile applications and websites is a major growth area in the industry at the moment, there are a number of challenges that mobile testing introduces that are unique to the discipline. In particular, there is the need to ensure application functionality and compatibility across a wide range of devices and in a wide range of different situations. In order to do this efficiently and effectively then it is essential that companies maintain a suitable mobile testing lab.
A mobile test lab is a broad term for a collection of the materials and articles that a tester or test group will require in order to effectively test software that is intended to be accessed via a mobile device. This may include a number of different items required to support the testing itself, as well as items that are required in order to replicate certain specific test conditions. Many test conditions, especially those for situations like low signal strength and low battery levels are unique to mobile devices and are unlikely to have been considered if a testers experience is in another testing domain.
The contents of a mobile test lab will of course vary depending on the applications and devices under test. A test lab should contain a suitable number of desktop machines with internet access via Wi-Fi and should be situated in an area where there is access to suitable cellular signals (2G/3G/4G). Situating a mobile lab close to the test team is also a sensible choice.
Probably the most difficult part of a mobile testing strategy is ensuring that there is sufficient device coverage. This is a particular problem when testing the Android ecosystem where there are a large number of different OS versions and a number of different vendors who produce a large number of different sorts of devices. However it is also an important consideration when testing on other OS’s as well. The primary focus should be on the devices that the application is targeted towards, as well as the version of the OS with the largest deployment in the field. For example, although Jelly Bean is the most recent version of Android, 64% of devices are still running the older Gingerbread version. Ignoring the devices in the field would be a critical mistake.
It is important to ensure that the different screen sizes and mechanical styles are covered, in particular differences between portrait and landscape screens, and hardware QWERTY keypads and touch screens. Also important is to ensure that devices from a variety of different carriers are included. Mobile device manufacturers typically adapt their software to the needs of different carriers, particularly in the US, and therefore they may work differently, even if the overall OS version is the same. A suitable suit of mobile devices can represent a significant outlay to a company or individual and another solution is to use a cloud based service although these are not as easy or quick to use as having the device in the hand. A low cost option is to make a device library by using the devices owned by the testers in the team. Obviously this is not ideal since some test cases may require user data to be removed or device software to be changed, for some applications then it can be a low cost solution. Whatever solution is chosen, a suitable indexing and booking system should be used in order to ensure that the usage and whereabouts of each device is tracked.
Not all SIM cards are created equally and some have more functionality than others. SIM cards which support cellular data and particular network supported functionality are likely to be required. It is also important to ensure that the test lab has SIM cards that are not only active and usable in the country in which the lab is situated, but also that are relevant to the particular region in which the application will launch.
SIM cards vary greatly in their access speeds and in particular, when testing contacts and address book applications, which may access the SIM card, then it is important to ensure that the test lab contains known slower cards which can uncover timing related bugs. If the test strategy also intends to make use of cloud based services then it is possible to also get access to a number of SIM cards in the cloud based devices. However, be aware that it is difficult to generate timing related issues on cloud based services, primarily due to the slower speed of device access.
It is important to ensure that the mobile test lab contains a number of memory cards which can be used to generate test conditions such as full cards, card removed during read or write operation or card inserted during a specific operation.
One commonly overlooked area of mobile testing is that of low signal strength. Applications are typically tested in the perfect environment of the office, with good cellular and Wi-Fi signals, and bugs are only uncovered after launch when the applications are used in real-world environments. Applications often fail when the signal strength is low, or in areas where there are frequent handovers from the 2G to 3G cellular signals or Wi-Fi. Testing for low signal strength can be carried out in a variety of ways. It may well be that ‘dead spots’, areas with no suitable Wi-Fi or 2G/3G cellular signal exist in the office, or these can be simulated by using shielded boxes or even old microwave ovens. Simulating handover from 2G to 3G cellular signals is more difficult and may require that the tester leave the test lab or office in order to find a suitable signal area outside. Mobile operators often have specific low signal and handover areas and routes are driven whilst testing, specifically for this purpose. It is also possible to purchase mobile network simulators from companies such as Anite and Anritsu which are able to simulate these conditions but they represent a significant financial outlay.
Testing for battery life considerations is another area of testing which is often overlooked. Poorly written applications can have a significant effect on the battery life of devices, and applications themselves can often exhibit unwanted behaviour when battery levels on devices are low. A mobile test lab should contain both the ability to simulate low battery levels, and the apparatus to monitor the effect that an application can have on battery life. Simulating low battery levels can be achieved either via dummy batteries which are connected to power supplies on which the voltage and current can be varied, or a suitable supply of batteries with varying amounts of charge. Monitoring the effect an application has on battery life can be achieved by using the on device diagnostic tools or applications that are available for most popular mobile devices OS’s.
This should include a selection of headsets, both wired and Bluetooth, plus a selection of USB and other connectivity cables which can be used to connect devices to desktop PCs to test upgrade/ downgrade, back-up/ restore and to add test data such as address books, photos, videos and music files. There should also be a selection of chargers and enough power sockets to ensure that phones can remain on charge. This is especially important if the test automation solutions also reside within the test lab and mobile devices are required to remain on charge whilst testing is on-going. If appropriate, a mobile test lab could also contain suitable car kits or other peripherals such as Bluetooth speakers and other audio devices which can connect to the devices under test.
Where it is required to send SMS/ MMS messages to the devices under test then it is important to ensure that the test lab supports this. Internet based servers can be used, or for smaller numbers of messages then sending can be carried out via other devices situated within the test lab.
Capturing screen contents, particularly for inclusion in bug reports, is more difficult when testing mobile devices and applications. Some OS’s support screen capture and some do not, and it can be particularly difficult to capture screen grabs when reproducing complicated, timing dependant bugs since it is not then possible to stop the test in order to record the screen contents. For the reason it is advisable to equip the test lab with a small video camera which, together with a suitable white table top, can be used to video the screen and steps required in order to reproduce the bugs.
Once a mobile test lab is setup and being used then it is important to ensure that its contents are regularly reviewed and updated. The pace of change in mobile is very fast, with new devices and OS versions being brought out frequently. Ensuring that the mobile test lab has the latest devices is important and regular reviewing of the contents is essential to ensuring continued use.
A mobile test lab can represent a significant financial outlay for a company and it is important that the money is spent wisely. Whilst it is possible to build an entire mobile test strategy around cloud based services, these are often slow and are currently not suitable to completely replace devices ‘in the hand’. Ensuring that there is a sufficient selection of devices is critical to ensuring that a mobile test lab adds value to an organisation, as is a suitable selection of different SIM cards and peripherals. Being able to replicate external situations such as low signal strength, handover between different cellular and Wi-Fi signals, and low battery life is also necessary in order to ensure that a mobile test strategy is comprehensive enough to cover these requirements, which are not typically tested for in the desktop world. Ensuring that a mobile test lab is suitably sized to support the testing team and applications under test is important. For simple applications then it may be enough to simply use personal devices, together with some facility to allow charging and screen capture, together with the ability to test low signal strengths. For more complicated applications, or where the application will be available on a wide number of OS versions or devices, then a more detailed approach will be required. A good mobile test lab adds value to the test organisation and the launch of a good quality mobile application is far less likely to succeed without one.
As part of my role I manage testers and also those involved in delivery operations. This means thinking not only about testing and test techniques, but also about person management, and the tools and techniques that a good manager uses in order to have a happy and productive team. These two areas should not, of course, be treated in isolation since, in order to have a successful test team, it’s the managers job to ensure that the two areas fit well together. If this can be done in as seamless as possible a way, maybe even without the team members even being aware of it, then my experience tells me that you can hit that sweet spot where the team is technically excellent as well as being the sort of team that testers want to work in, and are proud to work in.
I’ve studied a lot of theories of management in the past but the one that I keep coming back to again and again is Maslow’s Hierarchy of Needs. At it’s simplest it seeks to explain the needs of human’s in their most basic form, expressed hierarchically in order of those needs.
So, for example, the theory puts forward the hypothesis that it is most important (and therefore at the bottom of the pyramid) that human’s experience a need for food, water, etc. Safety needs come next, followed by a need for belonging, and so on.
As a manager, keeping this simple theory in mind can really help. I’ve found numerous occasions in the past where it has helped me understand team members actions, and to help ensure the team is working effectively together.
But it has also got my thinking about how one might go about applying Maslow’s Theory of Needs to Testing. More specifically, how a tester in a team or project might experience and visualise those needs, as reflected by their status within the team. This lead me to produce this:
Test Mastery is a place I see the respected, happy tester sitting. Probably people like you, who interact and are involved with the software testing community, who self-learn and who are able to articulate the need for testing effectively.
This is just the start of the theory. I’m sure, as a community we can add more to this. What would you add to the different levels and why? Leave a comment below.
I’ve recently been reading through my ISEB Practitioner notes, which I got when attending a course organised by Grove Consultants a few years back, as I mentioned in my previous post. It’s got me thinking about test cases, and in particular the four criteria of a good test case. Having attended both Rapid Software Testing and Rapid Test Management recently, and having rolled out an Exploratory and Session Based Test Strategy in my teams then it’s caused me to question again the validity of test cases and the need for them.
So, to a good test case. Reading my notes, a good test case is, in no particular order, apparently:
1. Exemplary.
2. Evolvable.
3. Economic.
4. Effective.
Straight from the ISTQB/ ISEB of course. But not without merit. Are these still relevant?
Good test cases can test more than one condition at the same time. This is one good reason for taking the time to design test cases in the first place. Just writing test cases because “it’s the done thing” or “because the policy says so” is time wasting, but designing them so that when the testing is carried out it is done so in the most efficient way requires exemplary test cases and can add value. It is also the case that test cases are shared and it’s difficult, especially in large organisations, to ensure that all testers have the same basic level of ability. Having test cases can help.
From a contextual point of view perhaps a good test case is not written down at all but merely in the testers head, and driving the testing into particular areas that the tester feels are worthy of time and effort. Cases where the software is the specification are becoming increasingly common; in-sufficient or non-existent requirements documentation which requires the tester to apply their previous knowledge of the system under test in order to effectively test it. Clearly in this case, if documented test cases are required then they will be need to evolve. As Rapid Software Testing mentions “How do you invent the right tests at the right time – evolve them with an exploratory strategy”.
Time is money and often in testing we have little time and sometimes little money. So using that time and that money in the most efficient way means we minimise the economic impact. Of course, sometimes the best way to get maximum value is to have a purely exploratory approach and spend more of the time and money with the software in hand. A choice which is key to a good test strategy and highly dependant on the industry area one is testing within.
Clearly a good test cases should be effective. We are not in the business of wasting time, particularly when time is precious as it often is in testing projects.
I’d argue that the definitions given in ISEB/ ISTQB are still relevant and can be a good guide as to what is required of a test case. In a lot of industries test cases are still very much required and, particularly where there is strict regulation such as in areas of financial software and even in mobile software. The ability to write a good test case is a skill which should not be forgotten.
I’ve recently been reading through my ISEB Practitioner notes, which I got when attending a course organised by Grove Consultants a few years back.
Don’t switch off yet. I know I mentioned ISEB. So before I go further, it’s worth stating my thoughts on the whole ISEB/ISTQB debate. I summed up my frustrations in a previous blog post; the fact that in the UK having ISTQB certification is practically the only way to get past the recruiters gate, but I also feel that there is some merit to the courses if taught properly, and followed up with a context driven approach such as Rapid Software Testing.
Reading back through my notes from Grove I can now see that this is what they were trying to work towards. At the time I had no idea how important the context driven school of testing was, nor the work of James Bach, Michael Bolton, Cem Kaner and others. But looking in the notes, the names are there. The techniques, albeit in nowhere near the detail that James or Michael teach in class of course, were hinted at and some approaches, particularly Exploratory Testing, are mentioned in some detail. Some slides are directly referenced from James and Michael’s work.
Unfortunately, due to the need to pass the exam, and with these areas being marked as ‘not exam’ then I didn’t pay them the attention that they warranted, and so it took a few more years to discover how and why the context driven approach can be so powerful. Which maybe shows the true problem with ISEB/ ISTQB certification after all.
If you remember from the last post, I recently attended Rapid Test Management, taught by Michael Bolton in London. For a general overview and what happened in the first day then take a look at the previous post. As before, the following caveat applies:
Caveat – this is not a list of exactly what happens in a Rapid Test Management class, nor is it a list of exercises and course material. You won’t become good at Rapid Test Management by reading this post. Sometimes I forget things. If you really want to know about Rapid Test Management then you need to sign up for the class. It will be money well spent and this post will tell you a little about what might happen if you do take the class.
We started day two talking about test strategy and how one might incorporate Rapid Software Testing into a test strategy. The Heuristic Test Strategy model was introduced and we studied this in some detail, and followed it up with an exercise to define a strategy for a smartphone. I was happy with the choice of product under test, given my background then it made it easier to take into account my domain knowledge, and so our group could come up with a large number of options to consider. Equally interesting was what other groups had come up with and it was very interesting to see how each group had approached the problem differently and come up with different solutions. This goes to show that diversity in teams and experience is very important in testing.
Back on day 1 we had produced a list of areas that we, as a group, wanted the class to be focused upon and the major areas that so far had not been covered were risk and test coverage. As usual, Michael had some great experiences to share and course material to cover the risk areas and the details of risk based testing. In fact, the amount of class material was another one of the great things about this class – you get a lot, far more than you can look at or can be taught in the class itself. Together with some useful testing tools and also some more exercises and demo’s, this gives a great set of material to use afterwards. Continual learning is important and so to gain access to all the notes, examples and slides is just the start.
Day 2 was concluded with a look at test coverage and ways to visualise test coverage. Rapid Software Testing provides you with some great ways to do this, from the simple to the more complex. It got me thinking about how I might incorporate this into the Kanban project management processes that my own team use to manage our work.
So, should you sign-up for the class? You bet’cha. Rapid Test Management is a class that you should attend. If you want it to, and you take the time to study all the information and material that you receive, then it will make you a better tester and it will make you a better test manager. I’m on the way; I’m just beginning to take on-board all I’ve learnt and to read through the material we didn’t cover in class, but with everything I read I’m confident that it’s improving my skills.
Caveat – this is not a list of exactly what happens in a Rapid Test Management class, nor is it a list of exercises and course material. You won’t become good at Rapid Test Management by reading this post. Sometimes I forget things. If you really want to know about Rapid Test Management then you need to sign up for the class. It will be money well spent and this post will tell you a little about what might happen if you do take the class.
Last week I had the opportunity to attend one of the first Rapid Test Management classes, and certainly the first one to have been run in the UK. It was a great experience and the beginning of a lot more learning. There were attendees from a number of different areas of software testing and together I think we formed a very effective group, learning from each other and sharing experiences.
Rapid Test Management follows on from the Rapid Software Testing class that James Bach and Michael Bolton teach. I’d attended James’ class in Cambridge at the beginning of March and so, with the class material and my ideas rolling around in my head, I pitched up at the Hilton hotel in leafy Kensington (well almost Shepherds Bush actually) for two days of intensive learning from Michael.
Rapid Software Testing is a context driven technique, intended to enable testers to excel at our chosen craft. Since it focuses on testing as a craft itself, instead of merely on the production of test cases and then their use for testing (or it could be argued merely for checking) then this does put some questions in mind when considering how best to manage it. By way of some preparation, I noted down some of the questions that I had:
The class was split into two days, the first morning focused mostly on refreshing understanding of Rapid Software Testing and ensuring that, as a group, we had effectively collected our own desired outcomes from the class. It was good to spend some time going back over the methodology itself even though I had learnt it very recently. It was great that Michael seemed happy to tailor the class to the audience in any way that was necessary, taking a lot of time to ensure that everyone’s opinion could be heard.
What I found great about this class, and the RST course, are the little nuggets of information, stories and quotes that James and Michael pass on. Looking back through my notes from this class I find that’s what I mostly write down, quotes like:
“Quality is value to someone who matters”
“Testers tell stories about products”
We also spoke in depth about the testers Elevator Pitch, that two minutes where you have to explain what testing is all about. Being able to explain and justify the role of testers and test teams is a critical skill and Rapid Software Testing gives you that skill.
When implementing any changes, such as introducing Rapid Test Management for example, then it’s critical to take into account the existing knowledge and processes that are already in use. Michael stressed the importance of ensuring that it’s not only the documented processes that are considered; in fact it’s the informal processes that are far more important to understand. He spoke about tacit vs explicit knowledge and how it is important for managers to be aware of the tacit knowledge in a test team and not merely the explicit knowledge.
We then took a look at how to put together a good team and how to train a new tester who was coming onto the team, through guidance and suitable heuristics. It was great to discuss with the others in the class about the issues they had faced when coming into a new team or bringing new team members in and also what had gone well. Ensuring that people are trained in a fault tolerant environment and gradually introduced to new tasks may seem fairly obvious stuff but it’s often over-looked, as is the need to ensure that feedback is both given and asked for from new team members.
Day 1 concluded with a look at metrics and how important they really are, v.s. how important most organisations seem to think they are. Michael pointed us towards Cem Kaner and Walter Bond’s paper Software Engineering Metrics: What Do They Measure and How Do We Know which I would definitely recommend reading.
Find out what happened on day 2 soon…
I’m excited to be able to announce that I’ll be speaking, and sitting on a panel, at the Next Generation Testing Conference which is on Wednesday 23rd May in London.
The conference is in its seventh year and is trying out a new format this year, with more emphasis on panels and discussions, together with case studies. This should hopefully mean that there’s some great audience participation and discussion on software testing.
I’ll be on Panel 1: Testing Today – What are the main challenges? It’ll be moderated by Dr Richard Sykes and they’ll be a group of us representing various areas of software testing, including Tony Bruce of London Tester Gathering fame. We’ll be debating various topics including:
I’m then be presenting a case study: Mobile Testing – That’s Just a Smaller Screen, Right? This will go into the background behind mobile device testing, how it differs from the desktop world, and giving some pointers towards areas to consider when formulating a mobile device strategy.
Hope to see you there. You can find out more about the event at http://next-generation-testing.com/
Last week I had the pleasure of attending Rapid Software Testing training, organised by The Ministry of Testing. Rapid Software Testing is a technique popularised by James Bach and Michael Bolton and hopefully is not something new to you, but in case it is then I’d recommend looking at James Bach’s website. He explains it much better than me 🙂
The course was in Cambridge in the UK, and after a quick and easy train ride up then it was easy to find the hotel, dump the bags, and then go out to the Software Testing Club Meetup. This was a good start to the course, an initial networking event which James also attended, as well as a lot of local testers who were not attending the course. We had some great discussions and I met a lot of new people; you can see some photos from the event which have been posted up by Rosie, the organiser.
Then to the first day of the course itself. From the moment the course started it was apparent that this was not your typical technical course. James’ style is well—known, just search YouTube if you want to see him in action, and he carried this into the course itself. He certainly knows his stuff, and presents in typical provocative style and is capable of causing many eureka moments. It’s very enjoyable, but initially tough, stuff.
We focused mostly in the first day on what Rapid Software Testing is, and the overall philosophy of the techniques. Rapid Software Testing is most useful to encourage testers to defend and think for themselves from a position of technical authority, especially useful in periods of uncertainty. Plenty of examples were given and James was able to draw upon his many years of experience in testing, both hardware and software. The time went by quickly and there was plenty of audience participation. James’ style is very much to put the audience members on the spot and ask some very difficult and blunt questions in order to replicate the pressures that testers can feel as part of project teams. To a few this comes naturally, but to most of the audience, this was a long way out of the comfort zone. We tried to help out whoever had been picked for a particular challenge, in order that the class as a whole could benefit.
The day concluded with an exercise on testing some everyday objects. Sounds simple, right? Well no. In case you will go on the course yourself then I will not give too much away, but suffice to say that there is much more to testing something which appears simple, than one at first thinks. It’s these sorts of exercises that open the mind and help learning.
After a good dinner with some new software testing friends, and a decent night’s sleep, it was time for day 2. Here we went into more details of Rapid Software Testing and the relevant testing models. Again the examples given were general, intended to make you think like a tester irrespective of your background, and plenty of pressure was applied to those who James selected from the audience. We looked at the differences between scripted and un-scripted testing and exploded some myths about both areas. We also talked a lot about oracles and why they are essential in testing. As an example, I was surprised to find that a person can be considered as oracle.
We also discussed heuristics a lot. Rapid Software Testing has many heuristics, the fact that James can remember and explain them all straight from his head is somewhat impressive. As with a lot of the techniques and information, a fair amount of common sense thinking was clearly applied when inventing the heuristics, but it was good to get names put to techniques that I was using already, for consistency if nothing else. There is a danger of quoting too many heuristics of course, especially when dealing with other’s within project teams and management. James’ view seems to be that by bombarding those outside of testing with information and explanations, using the relevant heuristics, that testers gain legitimacy. I do not agree with his approach to the length that he presents it – clearly testers need to be able to explain themselves – but there is a danger of losing credibility if too many heuristics are invented and then explained, which merely represent ‘day-to-day’ work. Take a look at James’ slides and see if you agree.
By the end of the day we were questioning practically everything about testing and about the way we were working. There is a danger from this course that one starts to question too much but one needs to start small and work up I think. That’s certainly what I intend to do.
The final day of the course started bright and early with more of the same. We focused on exploratory testing again, with more details, and talked a lot about documentation, metrics and information. The idea of focusing on a particular testing task, using some heuristics, but knowing when it is not working and de-focusing at this point, was a great learning for me. We also went into more detail on exploratory and session based techniques, something which I wish we had spent a bit more time on in previous days.
The main exercise for the day was based around finding a pattern for a system based upon dice. I won’t go into too many details on this (it’s explained pretty well at Better Testing) and also I do not want to give away a potential solution to anyone. But suffice to say it was a great opportunity to put into practice some of the techniques that we had learnt. Our group were not the quickest but neither were we the slowest, and it was certainly a good challenge.
The day then concluded with a wrap-up and overview of what we had learnt. Then some brief goodbyes and swapping of LinkedIn invites, and home to try and make sense of a busy three days and how what I had learnt could be applied to myself and the team members in my teams.
If you get the chance to go on Rapid Software Testing then go. Don’t think too hard about it, the course if very worthwhile and you will get a lot out of it. It is not easy, you will most likely feel uncomfortable at times with the training approach and some of the content may well seem obvious on first pass. But once you think more, and you start to question your own approach, with the techniques, tools, and even just the words, to back-up what you already know, then this course should make you a better tester. It would have been good to have seen a little more on session based techniques in detail and more about the tools that can be used, but I understand James does a separate course on this.
Thanks also go to Rosie Sherry, the course organiser. This was the first course that The Ministry of Testing have organised and if this first one is anything to go by then Ministry of Testing has a bright future. The venue and organisation was great, there was a really friendly, small company feel about things, and it was very easy to meet new people and learn together. Definitely three days well spent.
Tomorrow I’m leaving the safety of the South to journey far North* for something that I’ve been looking forward to for a long time. I’ve been fortunate enough to be able to sign-up for James Bach’s Rapid Software Testing course which has been arranged by The Ministry of Testing in Cambridge.
To say I’ve been looking forward to this for a while would be an understatement. Unfortunately due to budget constraints then it’s not been possible for me to get on courses like this in the past few years but ‘fortunately’ now that things are closing, then there’s a bit more money available for training and this will be money very well spent.
First stop is the Software Testing Club meetup tomorrow in Cambridge then on Wednesday the course starts. I don’t know what to expect but if it’s three days of hard but rewarding learning then I will be very happy. Having already taken a look at the course outline and slides then I’m sure it will be.
I’ll try and blog daily about my experiences, assuming I have the time and brain power left to do so 🙂
* (non-UK readers – we have a big North-South joke thing going on in the UK. If a place is north of Watford, which is a bit north of London, then us native southerners joke we’re out of the safety of the south and that it’s grim up north 🙂 Even though it isn’t and Cambridge is not even in really in the north anyway).