Category Archives: Testing

"ministry of testing", Experiences, management, recruitment, Testing

Persona Based Interviewing

5 Comments

Why you may need to do some actual testing in the interview for your next testing role

Interview

 

When I was starting to think about writing some posts to help out those who are either recruiting testers (see here for my first post), or for testers looking for new roles, the obvious first post was going to be around interviews. Interviews are scary, interviews need preparation, etc.

But to be honest, there is already so much available on the web that’ll teach you the basics, that I’d be re-inventing the wheel.

This is the second post in a series on finding a job within testing – the first one is Starting a New Job In Testing which you can also read on this site.

Basic interviewing techniques

If you need some help on basic interviewing techniques then I’d recommend spending some time on Ministry of Testing, looking at the recruitment resource section. What I want to discuss in this post is an interviewing technique we are using where I work, in order to help us recruit testers. It’s not new but it’s also not typical, and so hopefully it helps you.

 

A typical interview

A typical interview has a set of questions, and sometimes a script to follow. While the questions may not be written down, an experienced interviewer typically has a number of questions in their head that they can tailor to the interview situation, and use to steer the conversation in a particular direction. There may be a formal test, or the interview may just be conducted verbally.

Simply asking someone questions about testing, and gauging their responses, is one way of understanding in more detail what they know. You can also find out more about who they are, and what they can bring to the company. While we are following this approach, we’re also doing things a little differently.

 

Persona based interviewing

In a persona based interview, each of the interviewers play out part of a task based story, which the candidate is also a part of. One of more scenario’s are played out, normally based around a task that the candidate would typically face if they were successful in joining the company. The interviewers take different persona’s, each playing the part of a role or person that the candidate would need to interact with, in order to successfully pass the task that they are set. In this way the interviewers can understand how the candidate approaches particular tasks, how they solve problems, and how they interact with others.

For our interviewers for mobile test positions we typically play out some scenario’s based around testing our applications on real hardware. I won’t go into too much detail, for obvious reasons, but the candidate receives a certain testing task, and then is expected to start testing and exploring the application in order to successfully find bugs within it.

We play the parts of other people in the story. These could be the Product Owner deciding on re-prioritisation, other testers being able to offer advice, a developer being difficult or helpful, or even a senior manager wanting to know the progress of a testing task. As we go through we throw in these sorts of requests, and other changes to the scenario, in order to see how the tester works when requirements change and the pressure mounts.

Attempting to complete the tasks without interacting with the other roles within the scenario is very difficult and we are looking as much towards how and when the candidate asks for help, as we are to the testing skills that they show.

After the scenario’s are complete then we discuss with the candidate how they think the scenario’s went, what they thought went well and what they would do differently if faced with them again.

 

Is persona based interviewing useful for test roles?

We think it is. Mainly because:

 

It gives us a better idea of how the candidate thinks, and how they approach a testing problem

I’m a firm believer in context-driven testing and Rapid Software Testing in particular, as is the company I work for. To me, being able to observe how someone approaches a testing task, who they communicate with, and what questions they ask is very important. Being able to get an understanding of how they change their approach based upon the context is also much easier within the scenarios. Using persona based interviewing I get to discover more about ‘how they tick’ rather than what testing terms they can remember, or how much of their career story they can tell.

 

We get to see a candidates real, practical skills

By giving the candidates tasks based around testing our live applications, usually on an iPhone or iPad, we give the candidates an opportunity to demonstrate the real, practical skills that they have. We encourage the candidates to talk us through what they are doing and give them the opportunity to demonstrate what they know. Being able to demonstrate ability also puts people more at ease and we get to see what they can really do.

 

It focuses on critical assessment and improvement

By having a de-brief or lessons learnt session after the scenario’s have played out, we also get to see how a candidate critically assesses themselves, and discover what they would do differently given another chance. Making mistakes is human but learning from them is the important thing, and I don’t expect testers in my teams to get everything right first time. However I do expect them to be able to recognise ways in which they can improve.

 

It’s more fun than just answering questions

It’s certainly more fun for the interviewers, and hopefully it’s also a bit more fun for the candidate (as much as interviews can be anyway). Asking questions isn’t much fun, and only being able to show your skills in simple answers, is not the best way to spend your time. Demonstration of skills tests how a candidate works not what they can remember.

So, why not try persona based interviewing next time you are recruiting for testers. It can be a great way of finding really good people who you can have confidence will do a good job.

 

The next post in the series will be about C.V.’s, and what you need to highlight in order to get noticed.

Image courtesy of Ambro / FreeDigitalPhotos.net

 

Experiences, goals, management, plans, Testing

Starting a New Job in Testing

4 Comments
Image

Thing have been a bit quiet around here. Maybe you’ve noticed it. Maybe you’ve only came to this blog for the first time, from a link from Google or Bing, in which case welcome, and do note the gap in posts. I’ve been busy.

Primarily this has been because I’ve been searching for, and then settling into, a new job in a new company. Starting a new job isn’t easy, be it in testing or any other field, and it’s what you do in the first few months in a new company that can really affect how the rest of your time there goes. Those relationships you make early on, and how you are seen by others matter. It matters a lot.

So, while things are fresh in my mind, here’s some handy hints on what you should consider when starting somewhere new. I’ll be sharing further articles about recruitment, job searching and C.V.’s over the forthcoming months.

All presented with a software testing view.

Talk to people

Probably the most important thing you can do when you start, especially in a testing role. Being a good communicator, with the ability to get on with as many people as possible is very important as a tester. Spend your time searching out the important people, introduce yourself, and find out more about them. Ask them about the company, ask them about how you can work together, and find out what they think about how the company does testing. Those relationships you make early on affect how people see you for a long time. Sure, you won’t be able to remember all their names, but make sure they remember yours. And yes, this does mean talking to more than just testers 🙂

 

Ask Questions. A lot of questions

Testing is all about questioning a product in order to evaluate it, right? Take the same approach when you start a new job. Ask everything you can think of, no question is too stupid to ask if you don’t know the answer. Often, so much information in a company, is un-documented, and you want to make sure that you can understand the bigger picture, as well as just knowing enough to do your role. As testers we need as broad a view as possible in order to work effectively so make sure that you learn by questioning, and continue to do so. It takes time to settle into a new job but the more questions you ask then the quicker you should be able to feel like you understand company and your role within it. Don’t sit there, feeling blocked because you don’t feel like you can ask something.

 

Be nice

First impressions count. When you first start you meet a lot of people. Every time you meet someone new then that’s a new first impression. Always remember to be nice. Even if you are really an evil tester 🙂

 

Consider what you are bring to the company

They decided to employ you, there must be a reason, right? 🙂 Remember that you have good, and relevant, experience that you can bring. It’s easy when you first start, to just spend your time learning the company and your new role. But do not forget that your experience from outside the company could be really useful. Maybe your old company did it’s test automation a particular way or maybe you can see the new company works in a way you helped improve in an old role. It’s often new starters who are in the best position to see improvement areas. Don’t just accept that ‘this is the way its done round here’, and don’t feel afraid to speak up.

 

Ask for feedback

It can be really easy when you start somewhere new to just go into things head down and try and pick everything up at once. Remember that you need to pause and ask for feedback. Make sure that you question your own understanding of things, as you learn them, and regularly ask for feedback from those you work with.

 

Don’t forget to continue focusing outwards as well

When you start at a new company it can be really easy to spend all your time learning your new role. After all, there is a lot to pick up and a lot to learn. It is tiring, but don’t forget what you did before. Since you are reading this then I assume you read software testing blogs at least. Keep active in the testing community and keep learning. That way you can bring not only new ideas from your previous job, but also new ideas from the whole testing community.

 

Enjoy it

It takes a lot of effort when you start a new job. There is a lot to learn and a lot of people to meet. Focus on what attracted you to the new job and don’t forget to enjoy it 🙂

 

management, mobile, practices, Testing

I’ll Be a Panelist at the Next Generation Testing Conference on 6th December

1 Comment

I’m excited to be able to announce that I’ll be a panelist at the Next Generation Testing Conference which is on Thursday 6th December in London.

The conference is in its seventh year and is continuing with the  format that was trialed at the last conference in May, with more emphasis on panels and discussions, together with case studies. This should hopefully mean that there’s some great audience participation and discussion on software testing.

I’ll be on two different panels during the day – “Retrospective on 2012” and “Looking Forward to 2013” We’ll be debating various topics including:

  • What have we learned in 2012?
  • TestDevOps – what does it mean to you?
  • Continuous Integration and Continuous Delivery – lessons learned
  • Mind Map techniques that work with Agile
  • Advances in risk-based testing
  • Changes in approach for testing cloud based solutions and mobile applications
  • The rise of TestOps
  • Innovation/renovation
  • What is the next big thing?
  • “Prespective” of 2013

Others presenting or as panelists at the event include Paul Gerrard, Julian Harty, Steve Tulk and Niels Malotaux.

It’s sure to be a great event, as the last one was. Hope to see you there. You can find out more about the event at http://next-generation-testing.com/

"ministry of testing", Experiences, Testing

I’ll be speaking at TestBash 2.0

Leave a comment

 

Some exciting news – I’ll be speaking about “A Tester’s Hierarchy of Needs” at TestBash 2.0 which is being held in Brighton, in the UK, on Friday 22nd March 2013. This will be the first time I’ve spoken at the event, and looking at the others speakers, then I’ll be certainly in very good company. James Bach will be giving the keynote.

You can find out a lot more about TestBash 2.0 at the event website, including how to get Super Early Bird tickets.

Hope to see you there.

Testing

Professional Tester: Move Closer, Transition to DevOps

Leave a comment

I’ve written the lead article in this month’s Professional Tester magazine, about my department’s transition to DevOps and the impact that had on testing.

If you fancy reading it then head over to their website and subscribe, or you can download issue 17 which contains my article, and a number of other articles about TestDevOps, directly from this link.

I hope you enjoy reading about TestDevOps 🙂

Experiences, mobile, Testing

Testing Experience: Beyond the Smartphone – Testing for the Developing World

Leave a comment

I’ve been fortunate enough to have had an article published in the latest edition of Testing Experience magazine. It’s called “Beyond the Smartphone – Testing for the Developing World” and focuses on techniques and tools to use when testing feature phones.

You can download the whole magazine for free by visiting their website and subscribing to the online edition.

Hope you enjoy it.

blog, Experiences, mobile, Testing

Fundamentals of a Mobile Testing Lab

2 Comments

Although there can be no doubt that testing mobile applications and websites is a major growth area in the industry at the moment, there are a number of challenges that mobile testing introduces that are unique to the discipline. In particular, there is the need to ensure application functionality and compatibility across a wide range of devices and in a wide range of different situations. In order to do this efficiently and effectively then it is essential that companies maintain a suitable mobile testing lab.

What Is A Mobile Test Lab?

A mobile test lab is a broad term for a collection of the materials and articles that a tester or test group will require in order to effectively test software that is intended to be accessed via a mobile device. This may include a number of different items required to support the testing itself, as well as items that are required in order to replicate certain specific test conditions. Many test conditions, especially those for situations like low signal strength and low battery levels are unique to mobile devices and are unlikely to have been considered if a testers experience is in another testing domain.

What Should You Include In One?

The contents of a mobile test lab will of course vary depending on the applications and devices under test. A test lab should contain a suitable number of desktop machines with internet access via Wi-Fi and should be situated in an area where there is access to suitable cellular signals (2G/3G/4G). Situating a mobile lab close to the test team is also a sensible choice.

A wide selection of mobile devices

Probably the most difficult part of a mobile testing strategy is ensuring that there is sufficient device coverage. This is a particular problem when testing the Android ecosystem where there are a large number of different OS versions and a number of different vendors who produce a large number of different sorts of devices. However it is also an important consideration when testing on other OS’s as well. The primary focus should be on the devices that the application is targeted towards, as well as the version of the OS with the largest deployment in the field. For example, although Jelly Bean is the most recent version of Android, 64% of devices are still running the older Gingerbread version. Ignoring the devices in the field would be a critical mistake.

It is important to ensure that the different screen sizes and mechanical styles are covered, in particular differences between portrait and landscape screens, and hardware QWERTY keypads and touch screens. Also important is to ensure that devices from a variety of different carriers are included. Mobile device manufacturers typically adapt their software to the needs of different carriers, particularly in the US, and therefore they may work differently, even if the overall OS version is the same. A suitable suit of mobile devices can represent a significant outlay to a company or individual and another solution is to use a cloud based service although these are not as easy or quick to use as having the device in the hand. A low cost option is to make a device library by using the devices owned by the testers in the team. Obviously this is not ideal since some test cases may require user data to be removed or device software to be changed, for some applications then it can be a low cost solution. Whatever solution is chosen, a suitable indexing and booking system should be used in order to ensure that the usage and whereabouts of each device is tracked.

A suitable selection of SIM cards

Not all SIM cards are created equally and some have more functionality than others. SIM cards which support cellular data and particular network supported functionality are likely to be required.   It is also important to ensure that the test lab has SIM cards that are not only active and usable in the country in which the lab is situated, but also that are relevant to the particular region in which the application will launch.

SIM cards vary greatly in their access speeds and in particular, when testing contacts and address book applications, which may access the SIM card, then it is important to ensure that the test lab contains known slower cards which can uncover timing related bugs. If the test strategy also intends to make use of cloud based services then it is possible to also get access to a number of SIM cards in the cloud based devices. However, be aware that it is difficult to generate timing related issues on cloud based services, primarily due to the slower speed of device access.

Memory cards

It is important to ensure that the mobile test lab contains a number of memory cards which can be used to generate test conditions such as full cards, card removed during read or write operation or card inserted during a specific operation.

Apparatus to simulate low signal strength areas of both the cellular and Wi-Fi signals

One commonly overlooked area of mobile testing is that of low signal strength. Applications are typically tested in the perfect environment of the office, with good cellular and Wi-Fi signals, and bugs are only uncovered after launch when the applications are used in real-world environments. Applications often fail when the signal strength is low, or in areas where there are frequent handovers from the 2G to 3G cellular signals or Wi-Fi. Testing for low signal strength can be carried out in a variety of ways. It may well be that ‘dead spots’, areas with no suitable Wi-Fi or 2G/3G cellular signal exist in the office, or these can be simulated by using shielded boxes or even old microwave ovens. Simulating handover from 2G to 3G cellular signals is more difficult and may require that the tester leave the test lab or office in order to find a suitable signal area outside. Mobile operators often have specific low signal and handover areas and routes are driven whilst testing, specifically for this purpose. It is also possible to purchase mobile network simulators from companies such as Anite and Anritsu which are able to simulate these conditions but they represent a significant financial outlay.

Apparatus to simulate low battery levels

Testing for battery life considerations is another area of testing which is often overlooked. Poorly written applications can have a significant effect on the battery life of devices, and applications themselves can often exhibit unwanted behaviour when battery levels on devices are low. A mobile test lab should contain both the ability to simulate low battery levels, and the apparatus to monitor the effect that an application can have on battery life. Simulating low battery levels can be achieved either via dummy batteries which are connected to power supplies on which the voltage and current can be varied, or a suitable supply of batteries with varying amounts of charge. Monitoring the effect an application has on battery life can be achieved by using the on device diagnostic tools or applications that are available for most popular mobile devices OS’s.

A suitable selection of peripherals

This should include a selection of headsets, both wired and Bluetooth, plus a selection of USB and other connectivity cables which can be used to connect devices to desktop PCs to test upgrade/ downgrade, back-up/ restore and to add test data such as address books, photos, videos and music files. There should also be a selection of chargers and enough power sockets to ensure that phones can remain on charge. This is especially important if the test automation solutions also reside within the test lab and mobile devices are required to remain on charge whilst testing is on-going. If appropriate, a mobile test lab could also contain suitable car kits or other peripherals such as Bluetooth speakers and other audio devices which can connect to the devices under test.

Access to a server or other device to send SMS/ MMS messages

Where it is required to send SMS/ MMS messages to the devices under test then it is important to ensure that the test lab supports this. Internet based servers can be used, or for smaller numbers of messages then sending can be carried out via other devices situated within the test lab.

Screen capture or screen recording facilities, provided by a video camera if the device OS does not allow screen capture

Capturing screen contents, particularly for inclusion in bug reports, is more difficult when testing mobile devices and applications. Some OS’s support screen capture and some do not, and it can be particularly difficult to capture screen grabs when reproducing complicated, timing dependant bugs since it is not then possible to stop the test in order to record the screen contents. For the reason it is advisable to equip the test lab with a small video camera which, together with a suitable white table top, can be used to video the screen and steps required in order to reproduce the bugs.

Keeping it up-to-date

Once a mobile test lab is setup and being used then it is important to ensure that its contents are regularly reviewed and updated. The pace of change in mobile is very fast, with new devices and OS versions being brought out frequently. Ensuring that the mobile test lab has the latest devices is important and regular reviewing of the contents is essential to ensuring continued use.

Putting It All Together

A mobile test lab can represent a significant financial outlay for a company and it is important that the money is spent wisely. Whilst it is possible to build an entire mobile test strategy around cloud based services, these are often slow and are currently not suitable to completely replace devices ‘in the hand’. Ensuring that there is a sufficient selection of devices is critical to ensuring that a mobile test lab adds value to an organisation, as is a suitable selection of different SIM cards and peripherals. Being able to replicate external situations such as low signal strength, handover between different cellular and Wi-Fi signals, and low battery life is also necessary in order to ensure that a mobile test strategy is comprehensive enough to cover these requirements, which are not typically tested for in the desktop world. Ensuring that a mobile test lab is suitably sized to support the testing team and applications under test is important. For simple applications then it may be enough to simply use personal devices, together with some facility to allow charging and screen capture, together with the ability to test low signal strengths. For more complicated applications, or where the application will be available on a wide number of OS versions or devices, then a more detailed approach will be required. A good mobile test lab adds value to the test organisation and the launch of a good quality mobile application is far less likely to succeed without one.

blog, Experiences, mobile, Testing

Facebook Overwriting Email Addresses – Breaking Rule #1

2 Comments

As you are probably aware, Facebook has let what looks to be a pretty serious bug out into the wild. First announced over the weekend, and confirmed yesterday, was the news that users who use Facebook as their primary storage for contact information such as email addresses, had found that these addresses on their mobile devices has been overwritten with @facebook.com addresses. Without them knowing or accepting any such change.

 

What Happened?

The official line goes something like this:

Contact synchronization on devices is performed through an API. For most devices, we’ve verified that the API is working correctly and pulling the primary email address associated with the users’ Facebook account.

However, for people on certain devices, a bug meant that the device was pulling the last email address added to the account rather than the primary email address, resulting in @facebook.com addresses being pulled.

We are in the process of fixing this issue and it will be resolved soon. After that, those specific devices should pull the correct addresses.

Now let’s be clear. This is very serious. I’ve worked in the mobile industry for the last 12 years and if there is one thing that is sacrosanct it is user data. You do not change, delete, update or generally mess around with anything that the user has stored on their device without giving them the opportunity to tell you to stop it first. Users have a much greater emotional attachment to their devices than they do to their desktops. They rely on them. Suddenly finding that you cannot contact someone is extremely annoying. Finding that is because another company has changed data that is yours is far more serious.

 

So Why Did This Happen?

I’ve been musing on why this bug may have got released. After all, this seems a pretty obvious use case. Of course Facebook is famous for a) not having testers as such and b) famous for adopting a test in production methodology. Could either of these be to blame?

Would these problems have been found by using manual test techniques? Facebook is a primary user of test automation and I can’t help but feel that a bug this obvious would be found by a few skilled testers adopting an exploratory test strategy.

I also wonder how much of a place the test in production has in the mobile world. In the desktop world, where users are always connected and backups are plentiful then rolling out updates and observing what happens is OK. Just roll back. It’s not so easy on mobile. Being on a mobile device means that you probably don’t have access to backups right away. You may be in a poor signal area or away from WiFi. And so you are stuck. Stuck out of the office or away from your friends and unable to contact people. And if you are stuck then you will become far more frustrated.

It seems to me that there is a typical gap in test strategy at work here. A bug that only manifested itself on mobile, uncovered as a result of changes made at server/ desktop level. When companies start to move onto mobile then this is pretty common. Failing to adopt a combined test strategy, treating mobile and desktop equally, or pushing more testing towards mobile can leave dangerous gaps.

It’ll be interesting to see how Facebook responds to this situation. I don’t know Facebook’s test strategies in detail but it seems to me that something needs changing to adapt to the world of mobile.

 

 

blog, Experiences, management, practices, Testing

A Testers Hierarchy of Needs

16 Comments

As part of my role I manage testers and also those involved in delivery operations. This means thinking not only about testing and test techniques, but also about person management, and the tools and techniques that a good manager uses in order to have a happy and productive team. These two areas should not, of course, be treated in isolation since, in order to have a successful test team, it’s the managers job to ensure that the two areas fit well together. If this can be done in as seamless as possible a way, maybe even without the team members even being aware of it, then my experience tells me that you can hit that sweet spot where the team is technically excellent as well as being the sort of team that testers want to work in, and are proud to work in.

I’ve studied a lot of theories of management in the past but the one that I keep coming back to again and again is Maslow’s Hierarchy of Needs. At it’s simplest it seeks to explain the needs of human’s in their most basic form, expressed hierarchically in order of those needs.

So, for example, the theory puts forward the hypothesis that it is most important (and therefore at the bottom of the pyramid) that human’s experience a need for food, water, etc. Safety needs come next, followed by a need for belonging, and so on.

As a manager, keeping this simple theory in mind can really help. I’ve found numerous occasions in the past where it has helped me understand team members actions, and to help ensure the team is working effectively together.

But it has also got my thinking about how one might go about applying Maslow’s Theory of Needs to Testing. More specifically, how a tester in a team or project might experience and visualise those needs, as reflected by their status within the team. This lead me to produce this:

Testers Hierarchy of Needs

Test Mastery is a place I see the respected, happy tester sitting. Probably people like you, who interact and are involved with the software testing community, who self-learn and who are able to articulate the need for testing effectively.

This is just the start of the theory. I’m sure, as a community we can add more to this. What would you add to the different levels and why? Leave a comment below.

 

 

blog, Experiences, Testing

A Good Test Case Is?

1 Comment

I’ve recently been reading through my ISEB Practitioner notes, which I got when attending a course organised by Grove Consultants a few years back, as I mentioned in my previous post. It’s got me thinking about test cases, and in particular the four criteria of a good test case. Having attended both Rapid Software Testing and Rapid Test Management recently, and having rolled out an Exploratory and Session Based Test Strategy in my teams then it’s caused me to question again the validity of test cases and the need for them.

So, to a good test case. Reading my notes, a good test case is, in no particular order, apparently:

1. Exemplary.
2. Evolvable.
3. Economic.
4. Effective.

Straight from the ISTQB/ ISEB of course. But not without merit. Are these still relevant?

Exemplary

Good test cases can test more than one condition at the same time. This is one good reason for taking the time to design test cases in the first place. Just writing test cases because “it’s the done thing” or “because the policy says so” is time wasting, but designing them so that when the testing is carried out it is done so in the most efficient way requires exemplary test cases and can add value. It is also the case that test cases are shared and it’s difficult, especially in large organisations, to ensure that all testers have the same basic level of ability. Having test cases can help.

Evolvable

From a contextual point of view perhaps a good test case is not written down at all but merely in the testers head, and driving the testing into particular areas that the tester feels are worthy of time and effort. Cases where the software is the specification are becoming increasingly common; in-sufficient or non-existent requirements documentation which requires the tester to apply their previous knowledge of the system under test in order to effectively test it. Clearly in this case, if documented test cases are required then they will be need to evolve. As Rapid Software Testing mentions “How do you invent the right tests at the right time – evolve them with an exploratory strategy”.

Economic

Time is money and often in testing we have little time and sometimes little money. So using that time and that money in the most efficient way means we minimise the economic impact. Of course, sometimes the best way to get maximum value is to have a purely exploratory approach and spend more of the time and money with the software in hand. A choice which is key to a good test strategy and highly dependant on the industry area one is testing within.

Effective

Clearly a good test cases should be effective. We are not in the business of wasting time, particularly when time is precious as it often is in testing projects.

 

I’d argue that the definitions given in ISEB/ ISTQB are still relevant and can be a good guide as to what is required of a test case. In a lot of industries test cases are still very much required and, particularly where there is strict regulation such as in areas of financial software and even in mobile software. The ability to write a good test case is a skill which should not be forgotten.