Tag Archives: strategy

blog, Experiences, mobile, Testing

Facebook Overwriting Email Addresses – Breaking Rule #1

2 Comments

As you are probably aware, Facebook has let what looks to be a pretty serious bug out into the wild. First announced over the weekend, and confirmed yesterday, was the news that users who use Facebook as their primary storage for contact information such as email addresses, had found that these addresses on their mobile devices has been overwritten with @facebook.com addresses. Without them knowing or accepting any such change.

 

What Happened?

The official line goes something like this:

Contact synchronization on devices is performed through an API. For most devices, we’ve verified that the API is working correctly and pulling the primary email address associated with the users’ Facebook account.

However, for people on certain devices, a bug meant that the device was pulling the last email address added to the account rather than the primary email address, resulting in @facebook.com addresses being pulled.

We are in the process of fixing this issue and it will be resolved soon. After that, those specific devices should pull the correct addresses.

Now let’s be clear. This is very serious. I’ve worked in the mobile industry for the last 12 years and if there is one thing that is sacrosanct it is user data. You do not change, delete, update or generally mess around with anything that the user has stored on their device without giving them the opportunity to tell you to stop it first. Users have a much greater emotional attachment to their devices than they do to their desktops. They rely on them. Suddenly finding that you cannot contact someone is extremely annoying. Finding that is because another company has changed data that is yours is far more serious.

 

So Why Did This Happen?

I’ve been musing on why this bug may have got released. After all, this seems a pretty obvious use case. Of course Facebook is famous for a) not having testers as such and b) famous for adopting a test in production methodology. Could either of these be to blame?

Would these problems have been found by using manual test techniques? Facebook is a primary user of test automation and I can’t help but feel that a bug this obvious would be found by a few skilled testers adopting an exploratory test strategy.

I also wonder how much of a place the test in production has in the mobile world. In the desktop world, where users are always connected and backups are plentiful then rolling out updates and observing what happens is OK. Just roll back. It’s not so easy on mobile. Being on a mobile device means that you probably don’t have access to backups right away. You may be in a poor signal area or away from WiFi. And so you are stuck. Stuck out of the office or away from your friends and unable to contact people. And if you are stuck then you will become far more frustrated.

It seems to me that there is a typical gap in test strategy at work here. A bug that only manifested itself on mobile, uncovered as a result of changes made at server/ desktop level. When companies start to move onto mobile then this is pretty common. Failing to adopt a combined test strategy, treating mobile and desktop equally, or pushing more testing towards mobile can leave dangerous gaps.

It’ll be interesting to see how Facebook responds to this situation. I don’t know Facebook’s test strategies in detail but it seems to me that something needs changing to adapt to the world of mobile.

 

 

"rapid software testing", blog, Experiences, management, Testing

More Experiences of Rapid Test Management

Leave a comment

If you remember from the last post, I recently attended Rapid Test Management, taught by Michael Bolton in London. For a general overview and what happened in the first day then take a look at the previous post. As before, the following caveat applies:

Caveat – this is not a list of exactly what happens in a Rapid Test Management class, nor is it a list of exercises and course material. You won’t become good at Rapid Test Management by reading this post. Sometimes I forget things. If you really want to know about Rapid Test Management then you need to sign up for the class. It will be money well spent and this post will tell you a little about what might happen if you do take the class.

We started day two talking about test strategy and how one might incorporate Rapid Software Testing into a test strategy. The Heuristic Test Strategy model was introduced and we studied this in some detail, and followed it up with an exercise to define a strategy for a smartphone. I was happy with the choice of product under test, given my background then it made it easier to take into account my domain knowledge, and so our group could come up with a large number of options to consider. Equally interesting was what other groups had come up with and it was very interesting to see how each group had approached the problem differently and come up with different solutions. This goes to show that diversity in teams and experience is very important in testing.

Back on day 1 we had produced a list of areas that we, as a group, wanted the class to be focused upon and the major areas that so far had not been covered were risk and test coverage. As usual, Michael had some great experiences to share and course material to cover the risk areas and the details of risk based testing. In fact, the amount of class material was another one of the great things about this class – you get a lot, far more than you can look at or can be taught in the class itself. Together with some useful testing tools and also some more exercises and demo’s, this gives a great set of material to use afterwards. Continual learning is important and so to gain access to all the notes, examples and slides is just the start.

Day 2 was concluded with a look at test coverage and ways to visualise test coverage. Rapid Software Testing provides you with some great ways to do this, from the simple to the more complex. It got me thinking about how I might incorporate this into the Kanban project management processes that my own team use to manage our work.

So, should you sign-up for the class? You bet’cha. Rapid Test Management is a class that you should attend. If you want it to, and you take the time to study all the information and material that you receive, then it will make you a better tester and it will make you a better test manager. I’m on the way; I’m just beginning to take on-board all I’ve learnt and to read through the material we didn’t cover in class, but with everything I read I’m confident that it’s improving my skills.

"rapid software testing", Experiences, Exploratory Testing, Testing

Experiences of Rapid Test Management

3 Comments

Caveat – this is not a list of exactly what happens in a Rapid Test Management class, nor is it a list of exercises and course material. You won’t become good at Rapid Test Management by reading this post. Sometimes I forget things. If you really want to know about Rapid Test Management then you need to sign up for the class. It will be money well spent and this post will tell you a little about what might happen if you do take the class.

Last week I had the opportunity to attend one of the first Rapid Test Management classes, and certainly the first one to have been run in the UK. It was a great experience and the beginning of a lot more learning. There were attendees from a number of different areas of software testing and together I think we formed a very effective group, learning from each other and sharing experiences.

Rapid Test Management follows on from the Rapid Software Testing class that James Bach and Michael Bolton teach. I’d attended James’ class in Cambridge at the beginning of March and so, with the class material and my ideas rolling around in my head, I pitched up at the Hilton hotel in leafy Kensington (well almost Shepherds Bush actually) for two days of intensive learning from Michael.

Rapid Software Testing is a context driven technique, intended to enable testers to excel at our chosen craft. Since it focuses on testing as a craft itself, instead of merely on the production of test cases and then their use for testing (or it could be argued merely for checking) then this does put some questions in mind when considering how best to manage it. By way of some preparation, I noted down some of the questions that I had:

  • How can I best manage each testers time when adopting a Rapid Testing based strategy.
  • How does estimation fit in with the Rapid Software Testing ethos?
  • How can I sell the idea of Rapid Software Testing not only to those within my team who may be sceptical, but more importantly to those who manage me, who are most likely not only sceptical but also less knowledgeable about software testing in general?

Day 1

The class was split into two days, the first morning focused mostly on refreshing understanding of Rapid Software Testing and ensuring that, as a group, we had effectively collected our own desired outcomes from the class. It was good to spend some time going back over the methodology itself even though I had learnt it very recently. It was great that Michael seemed happy to tailor the class to the audience in any way that was necessary, taking a lot of time to ensure that everyone’s opinion could be heard.

What I found great about this class, and the RST course, are the little nuggets of information, stories and quotes that James and Michael pass on. Looking back through my notes from this class I find that’s what I mostly write down, quotes like:

 “Quality is value to someone who matters”

“Testers tell stories about products”

We also spoke in depth about the testers Elevator Pitch, that two minutes where you have to explain what testing is all about. Being able to explain and justify the role of testers and test teams is a critical skill and Rapid Software Testing gives you that skill.

When implementing any changes, such as introducing Rapid Test Management for example, then it’s critical to take into account the existing knowledge and processes that are already in use. Michael stressed the importance of ensuring that it’s not only the documented processes that are considered; in fact it’s the informal processes that are far more important to understand. He spoke about tacit vs explicit knowledge and how it is important for managers to be aware of the tacit knowledge in a test team and not merely the explicit knowledge.

We then took a look at how to put together a good team and how to train a new tester who was coming onto the team, through guidance and suitable heuristics. It was great to discuss with the others in the class about the issues they had faced when coming into a new team or bringing new team members in and also what had gone well. Ensuring that people are trained in a fault tolerant environment and gradually introduced to new tasks may seem fairly obvious stuff but it’s  often over-looked, as is the need to ensure that feedback is both given and asked for from new team members.

Day 1 concluded with a look at metrics and how important they really are, v.s. how important most organisations seem to think they are. Michael pointed us towards Cem Kaner and Walter Bond’s paper Software Engineering Metrics: What Do They Measure and How Do We Know which I would definitely recommend reading.

 

Find out what happened on day 2 soon…

blog, Experiences, mobile, Testing

Q4 Smartphone Sales – What Does It Mean for Testing?

Leave a comment

Gartner have recently published their Q4 2011 smartphone sales report. You can view it here: What does it tell us about the current state of the industry, and affect can this have on our work as testers in the mobile space?

Operating System 4Q11Units 4Q11 Market Share (%) 4Q10Units 4Q10 Market Share (%)
Android 75,906.1 50.9 30,801.2 30.5
iOS 35,456.0 23.8 16,011.1 15.8
Symbian 17,458.4 11.7 32,642.1 32.3
Research In Motion 13,184.5 8.8 14,762.0 14.6
Bada 3,111.3 2.1 2,026.8 2.0
Microsoft 2,759.0 1.9 3,419.3 3.4
Others 1,166.5 0.8 1,487.9 1.5
Total 149,041.8 100.0 101,150.3 100.0

Android Is Still King and iOS Is Increasing

Looking at the volume data, i.e. the number of devices running each platform, we can see a number of key points. Android increased market share from the same time last year, up to 50% of the worldwide market from only 30% a year ago. This represents a lot of devices. Android covers a wide price bracket, and devices based upon Android are available in a number of different form factors, display sizes and hardware configurations. The platform is clearly fragmenting even more significantly in order to covers these different needs.

Apple continued to increase market share, up to 24% of the market in Q4. The launch of the iPhone 4S has had an affect, as well as cost cutting of the earlier devices, a standard Apple launch policy.

As for the rest – Symbian decreased from 32% to 12%, primarily as a result of Nokia’s decision to announce the death of the platform and the knock on effect to consumer demand. RIM’s Blackberry OS lost ground, down to 9% from 14%. Microsoft’s WP platform didn’t fare well, only 1.9% of devices sold ran this OS.

 

What Does This Mean for Testers?

What’s very clear from the smartphone sales in Q4 is that there are still a large number of manufacturers in the market, and they are producing a lot of products. Growth was 47% year-on-year. This has some impacts for testing:

 

  1. Mobile applications will become impossible to ignore for a lot of companies in 2012. That means mobile applications testing will become more and more important.
  2. There are a lot of different device manufacturers and OS’s.
  3. Testers will need to cover a lot of devices and test strategies will need to reflect this.
  4. Anecdotal evidence is that the testing and QA efforts are not increasing to meet the current demand. Many applications are launched with serious bugs still present, indicating testing was not sufficient.

 

Probably the most difficult decision when designing a mobile test strategy is to decide the coverage of the OS and of the devices themselves. A typical mobile application launch strategy these days will focus on iOS and Android as the primary launch platforms for good reason – these are where the market share is, and therefore where the money is. Typically a third launch platform would then focus on Symbian, Blackberry or WP. For testers this means that there is a need to learn the skills and gain the experience with these platforms, and a clear focus on iOS and Android will initially be a good strategy.

For anyone who focuses on Android then there are additional challenges. The sheer variety of Android devices available now is staggering, from cheap, often un-licensed local brands right up to the flagship devices running the latest version OS version,  Ice Cream Sandwich. This poses many issues for anyone developing mobile applications and those testing them – being able to cover all the different configurations is difficult and care is needed to ensure sufficient coverage. It can help to know the target market for the particular application, the sort of devices available to the customers and the expectations towards key criteria such as performance and usability, as well as just ensuring that the functional aspects of the application are OK.

 

Mobile Website Testing Needs Additional Focus

For those testing mobile websites then the problem becomes even larger. A mobile test strategy for a particular application can be designed around the particular OS that application is written for, and relatively easily adapted for other OS’s at a later date. The same cannot be said of a mobile website strategy. There’s a need to provide as much coverage as possible across a significantly wider selection of OS’s and devices, some outside of smartphone scope as well, in order to ensure that the website is functioning correctly and adequately displayed. Tools such as the validator from W3 http://validator.w3.org/mobile/ can help to some extent but they are not a substitute for real testing on a real device.

 

Covering Most Devices

Covering the largest selection of devices is very important. It is not easy. Companies such as SOASTA, PerfectoMobile and DeviceAnywhere are worth checking out, since they take away the burden of device ownership from the company or tester. Crowd sourced testing will become important as well, and there are a number of companies working in this space. But there is significant scope for mobile testers here, and a significant need for more testing than currently.

Overall we can see that the smartphone sector is still growing and the market for applications is increasing. Anecdotal evidence is that the testing and QA efforts are not increasing to meet this demand, and that’s where testers can play their part.