Category Archives: practices

State of Testing Survey 2017

The guys over at QA Intelligence and Tea Time with Testers are running The State of Testing Survey 2017 again this year.

A survey which allows us to get a wider view of our profession and our community can only be a good thing in my book. It can help us understand our joint challenges far better, and to set the future direction. I’ll be a part of the survey and I hope you will be too.

The survey is open. You can find it at the QA Intelligence blog.

Note: I’m not affiliated with the survey or those running it, I think the survey is a good idea so I’m supporting it.

Agile In The City 2016

I’ve just got back from Agile In The City, which is a relatively new agile conference held in London. It’s in its second year and this was the first time it had been extended to two days. I had a great time; there was a good mix of talks, tutorials and workshops, a decent venue and even some good food as well.

IMG_20160616_095642140

As usual I did some mind maps of the sessions I attended. My favourite session was Managing For Happiness from Jurgen Appelo, a really inspiring keynote about how to manage better, with some great tips.

My mind maps from all the sessions I attended are below.

Keynotes

Managing For Happiness by Jurgen Appelo
Managing For Happiness by Jurgen Appelo
How To Derail Agile Rollouts - Katherine Kirk
How To Derail Agile Rollouts – Katherine Kirk

Track Sessions

IMG_20160618_140329446
Scaling agile development at the Government Digital Service by Adam Maddison

 

IMG_20160618_140338606
Punishment driven development by Louise Elliott

 

Develop The Product Not The Software - David Leach

Develop The Product Not The Software – David Leach (with free water pistol 🙂

 

Value by Andrea Provaglio
Value by Andrea Provaglio

 

From Oil Tankers To Speedboats - Jonathan Smart
From Oil Tankers To Speedboats – Jonathan Smart

 

#NoProjects - beyond projects - why projects are wrong and what to do instead - Allan Kelly
#NoProjects – beyond projects – why projects are wrong and what to do instead – Allan Kelly

Talking About Testing

Have you ever said “I’ll just have a play with the software….”?

As testers we are generally really bad at explaining what we do and the value it brings. In fact we are usually pretty rubbish.

This makes stakeholders take us less seriously, and can affect career prospects, position within the team, or even a job itself. Outsourcing what is perceived to be low skilled work is tempting, especially when times get tough.

We then complain that we are not being taken seriously, and we feel ignored, undervalued and sad.

And so we retreat into our bubbles and the whole thing repeats itself.

So How Can We Change This?

We can get better at explaining what we do and the value we bring as testers.

Keith Klain sums it up nicely here

What is Testing?

“Testing is the infinite process of comparing the invisible to the ambiguous so as to avoid the unthinkable happening to the anonymous”

James Bach

A great definition but not one that lands well with non-testers in my experience. It’s like when myself and Dan Ashby tried to get a company to stop saying ‘manual testing’ and instead say ‘sapient testing’. It made perfect sense but it just didn’t land. People didn’t see it as important. They didn’t see a need to change because we were trying to take a leap that was too large, from what they thought was correct language to what we thought was correct language.

When I’m explaining testing I prefer to start by asking “Do you care about quality?” The usual answer is “of course”, in which case I can then use the Weinberg/ Bach/ Bolton definition:

“Quality is value, to some person who matters”

“So what’s testing then?” they will ask.

Well.  “Testing helps us uncover risks to product quality. It’s about investigating software, in order to discover those risks, enabling others to make decisions about whether it’s suitable for release”.

We Need to Talk More Technically

But that is hard. Just see how complicated it is to explain the Heuristic Test Strategy Model in two minutes for example.

(OK – I know the whole point with this video is that it is impossible, which proves a point I think :))

What Value Do We Bring?

When explaining value, it’s all about the words we use, and the angle we take. It’s about the audience – don’t explain testing to a developer in the same way as you would to the CTO.

Again, Keith Klain nails it with this talk.

So, think very carefully about how you explain your testing. Perhaps, just perhaps, ‘playing with the software’ isn’t what you mean.

So – how do you talk about testing?

A Christmas Retro

Ho, ho. ho, Merry Christmas!

At this time of year it’s a great opportunity to sit down with a team and reflect on the year that has been, and to look forward to the year to come. And being Christmas then it’s also a great opportunity to look a bit silly, dress up in Christmas hats and jumpers and have a  bit of fun.

With that in mind, I decided to organise a Christmas retro for the team I’m working in. We are responsible for a brand new product  which launched this year, so we’ve had a very busy, very exciting and very experimental year. Getting the team together to reflect on that year was very important.

The Idea For a Christmas Retro

My first step was to look round for inspiration for Christmas themed retros. Had anything like this ever been attempted before? Fortunately I was in luck, both Em Campbell-Pretty and David Manske have both written about a great way to run a retro at Christmas, based upon the Dicken’s book ‘A Christmas Carol’.

Em’s blog article, where she applied the technique to a large group was my first inspiration, and then I stumbled upon David’s article as well, which helped to add some more details, and was helpful for the smaller (15 people) group that I work with.

Basically speaking, the idea is that you base the retro around the ideas of Christmas Past, Present and Yet To Come. In A Christmas Carol, the main character, Scrooge, is visited by ghosts on Christmas Eve who show him how Christmas was, how it is presently for others who rely upon him. and how Christmas will be, were he to continue on his current path. It’s a great book and I’d definitely recommend reading it. Or, watch A Muppet’s Christmas Carol. It’s great 🙂

present

Adapting It For a Retro

Since the Muppets theme seemed to be a fun way of introducing the idea of the Christmas Carol retro format then I went with that. The team love Muppets.

Adapting the story to use in the retro was pretty straight-forward and heavily based on Em and David’s ideas. In order to get the team thinking about everything that had happened throughout the year then myself and our Product Owner prepared a set of posters, one per month of the year and then pinned them up on the wall of the room where the retro was to be held. It was pretty amazing to look back at everything that the team had achieved throughout the year. We focused on delivered work; features, numbers of tickets, etc, and then interspersed this with press cuttings from the year that mentioned our products, team details and fun stuff, such as number of Percy Pigs sweets eaten, etc. Which is important to this team, believe me 🙂

IMG_2930

We also bought lots of Christmas hats, food and even a Father Christmas beard for yours truly, then invited the team into the room to start the retro.

Christmas Past

past

For Christmas past we got the team to look back at the year and then think about whether there was anything that they regretted or wish had gone differently. They wrote these on post-it notes and kept them to themselves.

Christmas Present

present2

Next I explained the idea of Christmas Present, and encouraged the team to think about what had gone well throughout the year, what had made them feel good, and who in the team they would like to thank for helped with particular pieces of work. Again, they wrote these on post-its.

Christmas Yet To Come

yettocome

The third piece, Christmas Yet To Come, was about the hopes for the next year, What did they want for the team, themselves and our product? The idea was to get them thinking about future priorities and to work on putting together a short list of what we, as a team, think is important for the future.

How It Went

So the stage was set, people were wearing silly hats and eating Christmas food. More importantly they could also see how much we had achieved throughout the year, and they were thinking about how we could work in the future.

The Grinch

grinch_by_crazymic-d6yvfkp

Once everyone had written down their thoughts on Christmas Past, Present and Yet To Come I introduced a fourth category, The Grinch. This was an idea that David explains in his blog post and the intention was to get them thinking about what could happen to us, as a team, that could threaten next Christmas. Essentially  it was about identifying project and team risk. So everyone also had a think about that and wrote down their ideas.

The Retro

We then went through Christmas Past, Present and Future, and The Grinch in turn.

For Christmas Past everyone was expecting to come and pin up their post-its, rather like a usual retro. But instead, being Christmas, it was time to let go of regrets and so I encouraged everyone to tear up their post-its and throw them away. When I was planning the retro I wasn’t sure how people would react this – would they view this as a waste of their time in writing the regrets down in the first place? It turned out they didn’t – after much hilarity with people throwing post-it’s at each other, the feedback was good, with some people saying that the idea of throwing away regrets rather than sharing them, was the best part of the retro.

For Christmas Present and Future everyone came up in turn and discussed what they had written on their post-it’s. People thanked each other for work that they had helped on. We talked through trends and actions that could be taken from what had been discussed. This gave everyone a great view on where we are as a team and where we want to be going.

For The Grinch we talked through the risks that everyone saw, and how we could deal with these in the next year.

From all of this we were, as a team, able to decide on what was important to us going forward; with actions that were both product focused, and also team focused, such as organising more social events and sharing the responsibility for doing so.

Merry Christmas

We finished by thanking the team for what they had done and explained how they should all be very proud of what they had achieved throughout the year. Then it was time to go off on our team Christmas event offsite.

Overall I think that this retro format works very well. By combining something seasonal, fun and focused, it made looking back on a whole year much easier and more exciting than a more traditional retro format would have done. The format engaged people and made it easier to share experiences.

And let’s be honest, who doesn’t like wearing silly hats and Christmas jumpers once in a while? 🙂

Experiences Of a Testing Bug Bash

I’m working in a team, that’s developing a mobile application and back-end, that does not have any testers. We didn’t plan it that way, but temporarily, that’s just the way it is. While I can add my testing skills and experience to the team, that’s not the primary reason I’m there, and so I’ve been looking at ways in which the whole team can help to contribute to a quality product.

It’s been interesting. At first, when our tester left then nothing happened. Like nothing at all, tickets moved over to the ‘Ready for Testing’ column and there they stayed. Immobile, ignored and unloved. It was as if all eyes were to the left, where the fun stuff for developers was, understanding designs and coding. Although there are unit tests and some automated integration level tests for our code it clearly wasn’t enough.

Session Based Team Testing – A Bug Bash

I’ve used session based testing in the past when I managed test teams, and have found it to be a very useful way of breaking exploratory testing work into understandable pieces, and distributing that work out to the team. James and Jon Bach explain the approach far better than I can. Ministry of Testing have also got an excellent resource on SBTM.

Having read some articles from people who had broadened their approach then I thought I’d take the session based testing concept and roll it out to the whole team as a bug bash.

The key point for me was that any sort of session had to be fun. Initially I feared that not all the team were going to see the full value of such a session since it wasn’t development work per se,  so keeping the approach light-weight was key. The ideas for our first bug bash had formed.

Organisation

Getting away from the everyday work was important so the bug bash took place in a separate room, close enough to our team area that it was quick to pop out and pickup forgotten devices, etc but away from the interruptions that phones, etc provide. I made sure we had enough phones, tablets, post-it notes, pens and cake to keep us going. I also brought in some popular testing books and the Test Heuristics Cheatsheet from Elizabeth Hendrikson, in case people needed some inspiration.

We use JIRA as our project management tool and since we were testing a whole Epic then it made sense to break the testing down on a story by story basis. Working with our lead developer, we took each Epic in turn and used that to form an exploratory testing charter stored as a Google doc. You can get an example version here. Using charters enabled us to define the boundaries of the testing; ensuring there was not too much overlap. It also meant that people who were not so familiar with the story were able to quickly learn about it, and areas to test, without needing to spend time reading stories and Epics in JIRA.

So, the charters were ready, the room was ready and the people were ready. Time to start.

The Bug Bash Session

We decided on two 45 minute sessions with a tea break in between. The whole team attended; PO, designers, UX, developers and myself. Everyone was a tester for the afternoon. We agreed who would pickup which charters and made a start.

I made a Google doc in which people could record potential issues which I kept projected up on the wall throughout, so that we could all see the progress that we were making and the issues that were being raised. I encouraged the team to add screenshots as well as text and this really helped to make things clear and to prevent duplicate issues making their way into the list. Discussion was also key to the success of the session; having everyone in the room at the same time meant that potential design issues could be shown to designers straight away for review.

The lead developer for the feature was also present, not testing but helping to explain architectural and development decisions, and also to fix some bugs on-the-fly.

The session went well. We got all the charters finished and the session concluded with a quick retrospective.

Retrospective

What Went Well
Charters give structure and prevent misunderstandings

Having structure was critical to the bug bash, given that it was a new activity for the team, and they were not experienced with exploratory testing. It also meant that we got the most from the session and people were not all testing the same thing.

Hints and examples in charters are good

By including examples, and ideas on areas to test, people had a guide to start them off. Given that they were not experienced testers then this really helped.

It is good to have charters split by story

This meant that it was clear what was to be tested and there was a clear link back to the project management tool people were used to.

Some of the bugs were fixed during the session

This clearly shows the value of group pairing for me.

What Could be Improved?
Make the charters simpler

I had hoped we would get some data on setup time and test execution time so had added that to the charters but it confused most people.

Make the process of splitting up the charters and recording who is working on each one, simpler

Although it as clear to them which charter a person was working on, we didn’t keep a visible record of who was working on what. Next time I’ll write this down and pin it up on the wall.

People could swap charters halfway through the session to give multiple points of view

For each 45 minute session one person would pick up and test against a charter. Swapping round could add value and a new set of eyes onto the area.

Comments
Aim to do this for all large stories

This was great to see since it showed that the team saw the value in the approach.

Consider pairing

This will be our next step for the Bug Bash sessions. Pairing people up could help to add more value by giving multiple viewpoints.

Going off charter was a good thing

Some charters were bigger than others. People naturally started to go off-charter once their charter was done and this gave multiple viewpoints across the whole feature.

Being away from the desk is a good thing

By being one team, with one goal, isolated from our day-to-day activities, we were able to effectively test and find many issues in the feature, without the usual distractions that break concentration and flow. Having lots of cake also helped 🙂

Conclusion

We will definitely do the Bug Bash again, and I see it as becoming a really important part of our delivery processes, even when we get a full time tester back on the team. It really helped everyone to understand the feature, to explore it and to test together, giving multiple viewpoints on potential issues.

With some guidance the whole team made a great contribution. I’d encourage everyone to try a similar approach,

London Tester Gathering Workshops 2014 – Mobile Software Testing

Later this week it’s the London Tester Gathering Workshops. I attended this event last year and it was great. There was a brilliant variety of workshops, and a lot of great testers to talk to. Very recommended.

The Workshop

This year I am giving a workshop on Mobile Software Testing. It’s on Thursday 16th Oct and I have four hours in which to give an intro to mobile, and give the attendees the opportunity to try out some common mobile tools.

The workshop will focus on giving you a basic overview of mobile testing but then will focus on you discovering how to test mobile websites and applications using the tools below. It’s not a lecture and I won’t give you all the answers, but I hope we’ll have some fun along the way as we discover more about mobile testing.

What You Should Bring

If you are coming along then it’d be great if you brought with you:

  • A laptop with:
    • The latest stable build of Chrome on it.
    • VirtualBox.
    • The Genymotion Android emulator.
    • At least one device image (I’d suggest the Samsung Galaxy S5).
  • As many mobile devices as you have.

If you haven’t got the Genymotion emulator then I will bring along a standalone version that can be installed for Mac or PC from a USB stick. But if you have already downloaded it then that would make things much smoother and you’ll have more time to use it.

There will also be a bit of a competition at the end so have a think about what application or website you might like to test. You’ll be able to choose anything you want 🙂

Hope to see you there.

Exploring An Existing System

Recently I’ve been reading Elisabeth Hendrickson’s excellent book ‘Explore It!’. For anyone who has an interest in exploratory testing it’s a must own. I wish I’d discovered it earlier to be honest, since it gives so many useful hints and tips, as well as confirming that the ways of working that one has chosen are also recommended and used by others.

As well as using it for my own learning, I’ve been slowly going through the book and working out what parts I can use in the regular lunch and learn sessions that I run at work. While we practice exploratory testing, in fact it’s the cornerstone of our sapient testing strategy, there are some areas where I feel we could take approaches that could benefit not only testing, but also the wider business.

Working With Legacy Systems

We frequently work with legacy systems and so one chapter that was of immediate interest to me in Elisabeth’s book concerned exploring an existing system. When one has an existing system to test I find it’s all too easy to become primed by what others have already discovered, and to fall back on existing test cases (whether physically written down or in someone’s head). This can bias you, resulting in less effective testing.

Elisabeth makes the point that an existing system may well be unknown to the tester, but also may well be unknown to the whole team, or at least contain parts that are unknown. While the software fulfils a business need, how it actually goes about doing so may be less clear. That makes it ideal for exploration.

Recon Testing

James and Jon Bach like to call the initial exploration of an existing system ‘recon testing’. I like this term, by taking an initial session to explore and discover the basics of the software under test then one can then plan more effectively for future sessions, and write future charters in order to drive those plans (if you want to know more about the concept of session based testing and how charters fit in with this then have a look at James Bach’s explaination). Recon sessions help to map the territory and give insight.

During a recon session you can learn a lot, but the most important areas to ensure that you have gained insight into are:

  • The ecosystem in which the software under test resides.
  • Touchpoints to other systems.
  • Variables (things we change or can change).
  • Obvious vulnerabilities and potential risks.

Our Training Session

During our training session we carried out recon testing on the Staples ‘Easy’ button, and a standard service bell. This no doubt annoyed those in the room next door 🙂

Ding ding!
Ding ding!
Easy to test?
Easy to test?

By starting training with something simple, and not software based then it’s easier to pick up and learn the basics of a new technique without bias or the complication of software. We then compared our experiences, testing, (and the charters produced), with those that the Bach brothers produced when they carried out an exploratory testing exercise using the same product. Fortunately for us, their session is available on youtube, so it was an excellent addition to our de-brief. In it they explain the different testing techniques they use, and why. It’s well worth watching.

Enough Recon Testing?

One area to focus upon when conducting recon testing is whether one has conducted enough recon testing. Fortunately ‘Explore It!’ has this covered, recommending you ask yourself questions about the system that you have been exploring. If you don’t understand what the system does, how input and output works or how the environmental configuration affects the system for example, then it’s probably time to think about more recon before you move into more focused test sessions. Fortunately for the ‘Easy’ button and bell testers, (and those sitting near the meeting room where we had the lunch and learn), then this was not necessary 🙂

A Useful Addition

Recon testing is something that I think we’ll find is a really useful addition to our strategy. I’d certainly recommend that you check it out, and that you check out ‘Explore It!’ which contains much more useful information and techniques to use in your exploratory testing. I’ll certainly be using the book to inspire some future training sessions for the team.

Testing In The Pub Episode 5 – Continuous Delivery

Episode 5 of the software testing podcast I record with Dan Ashby is now available. This time we interviewed Amy Phillips about Continuous Delivery and how testing fits in with Continuous Delivery.

It was certainly fun to record an interview, and I think we got a lot of good discussion going. It’s well worth listening to (ok I am bias but….) It’s a two part interview and we’ll be publishing the second part next week.

You can download it directly from the Testing In The Pub site, subscribe from iTunes or using our RSS feed from your favourite RSS reader.

All feedback is most welcome, we want to produce a podcast that people want to listen to 🙂

Automated Functional Testing – A Test Activity?

If you are a functional test automation expert then times are good. There’s big bucks to be made in the contracting game, companies are desperate for candidates to ‘automate everything’ and to get to this oddly perceived test automation nirvana that those who are either mis-informed or have hidden agenda’s seem to feel fit to promote.

This has made me think. Primarily about how we have got to this state? Is it because, as the testing community, we have wanted to own test automation? Is it because those outside of the test community see test automation as less important than the production code that it tests? Is it that we just built up an expertise and then protected it just for the money?

Some might say that what has actually happened is that we now have a situation where second rate developers now have a great way to stay in the development game. There is a danger, in the apparent supply-side crisis that we find the industry in, that companies merely employ anyone who says they know something about test automation, without doing the same due diligence that one would do for a development position. This would be a mistake.

In my mind there is a solution to all these problems, and that solution comes from treating test automation just like production code. And that means primarily using developers to write it. Sure, you may choose to have testers involved as well, where they have the skills and expertise, but let’s not try and force skills on people who don’t want them, and let’s not accept second rate people just because they can ‘do some test automation’. One advantage to using developers is that test automation becomes a team thing, and you are less likely to spend time playing catch-up when development slips. One downside; it’s going to look like the team has slowed down. Believe me, it hasn’t. It’s just got more effective, and is playing to the right skill-sets.

Don’t believe me? 🙂 Here’s a couple more examples from Rob Lambert and Amy Phillips which show where continuous or more frequent delivery has been successfully rolled out at New Voice Media and Songkick. The common thread – in both cases the test automation is a development activity.

I’ll Be a Panelist at the Next Generation Testing Conference on 6th December

I’m excited to be able to announce that I’ll be a panelist at the Next Generation Testing Conference which is on Thursday 6th December in London.

The conference is in its seventh year and is continuing with the  format that was trialed at the last conference in May, with more emphasis on panels and discussions, together with case studies. This should hopefully mean that there’s some great audience participation and discussion on software testing.

I’ll be on two different panels during the day – “Retrospective on 2012” and “Looking Forward to 2013” We’ll be debating various topics including:

  • What have we learned in 2012?
  • TestDevOps – what does it mean to you?
  • Continuous Integration and Continuous Delivery – lessons learned
  • Mind Map techniques that work with Agile
  • Advances in risk-based testing
  • Changes in approach for testing cloud based solutions and mobile applications
  • The rise of TestOps
  • Innovation/renovation
  • What is the next big thing?
  • “Prespective” of 2013

Others presenting or as panelists at the event include Paul Gerrard, Julian Harty, Steve Tulk and Niels Malotaux.

It’s sure to be a great event, as the last one was. Hope to see you there. You can find out more about the event at http://next-generation-testing.com/