Test Strategy – a handy checklist

From XKCD 1287

 

As part of trying to educate other people at work about testing strategy, I’ve been thinking about it a lot recently – in particular because I keep seeing similar issues and often struggle to concisely get across.  While I’m still working on that thinking (hopefully post(s?) to follow), one thing I keep coming back to that’s really useful is one of the checklists from the Rapid Software Testing Appendices – “a set of considerations designed to help you test robustly or evaluate someone else’s testing”.  It’s not the be-all-and-end-all, but a lot of the test strategies I come across miss thinking about a bunch of the stuff here, so using this is a good starting point when coming up with your strategy.

(All kudos for this list => The RST creators)

Project Environment

• Mission. The problems you are commissioned to solve for your customer.
• Information. Information about the product or project that is needed for testing.
• Developer Relations. How you get along with the programmers.
• Test Team. Anyone who will perform or support testing.
• Equipment & Tools. Hardware, software, or documents required to administer testing.
• Schedules. The sequence, duration, and synchronization of project events.
• Test Items. The product to be tested.
• Deliverables. The observable products of the test project.

Product Elements

• Structure. Everything that comprises the physical product.
• Functions. Everything that the product does.
• Data. Everything that the product processes.
• Interfaces. Every conduit by which the product is accessed or expressed.
• Platform. Everything on which the product depends (and that is outside your project).
• Operations. How the product will be used.
• Time. Any relationship between the product and time.

Quality Criteria Categories

• Capability. Can it perform the required functions?
• Reliability. Will it work well and resist failure in all required situations?
• Usability. How easy is it for a real user to use the product?
• Charisma. How appealing is the product?
• Security. How well is the product protected against unauthorized use or intrusion?
• Scalability. How well does the deployment of the product scale up or down?
• Compatibility. How well does it work with external components & configurations?
• Performance. How speedy and responsive is it?
• Installability. How easily can it be installed onto it target platform?
• Development. How well can we create, test, and modify it?
• (I’d explicitly break out from that last one
  • Testability.  How well can we put the product into the state we want to put it into for testing?
  • Diagnosability.  How well can we tell what state the product is in?
  • Debugability.  How easily can we work out what has happened?
  • Maintainability.  How easily can we fix issues in or enhance the product?)

General Test Techniques

• Function Testing. Test what it can do.
• Domain Testing. Divide and conquer the data.
• Stress Testing. Overwhelm the product.
• Flow Testing. Do one thing after another.
• Scenario Testing. Test to a compelling story.
• Claims Testing. Verify every claim.
• User Testing. Involve the users.
• Risk Testing. Imagine a problem, then find it.
• Automatic Checking. Write a program to generate and run a zillion checks.

Measurable Quality Bars

 

A measurable quality bar in St. Pancras Station, London.  It’s 98m long

This post came out of some notes I wrote up when trying to persuade some other parts of our organisation that measuring quality as “number of open bugs” was a really poor idea – specifically that it’s (a) not a metric for quality and (b) it strongly encourages short-termist behavior that lowers long term quality (prioritising immediate functional bugs over maintainability, extensibility, reducing regression counts, diagnosability, all those other “ilities”).  I’ve walked that path on so many projects, I’m keen to try to stop others following down the track…

Here goes:

What is quality – what are we trying to measure?

Before we think about how to measure quality, it helps to think about what is quality?  The best definition I’ve come across originated with Jerry Weinberg

Quality is value to some person (who matters)

So for our products, who matters, and what do they care about?

From a quickish brainstorm meeting a year or few back, an incomplete set is:

• Who matters: Our customers, their customers, support, solutions teams, development teams, test teams, product management, trials/demo teams, Sales, our C-suite.
• What matters to them (not all things matter to all people): Function set, working function, reliability, price, usability, diagnostics, debuggability, maintainability, extensibility, documentation, standards compliance, integration, interop, configurability, simplicity/complexity, “IT”/Sexiness/Wow factor, supportability.

Looking at all those, it seems that quality is a pretty hard thing to pin down or measure.  We do like measuring things though, and what we can do is measure some proxies for quality.  Note – these proxies aren’t “quality” – they’re things that seem to align well with something that matters to someone (so part of quality).

So what can we set out to measure that would do better?

Easy(ish) to measure proxies for quality (“quality metrics”)

My goal with this list is to think of proxies that are easy to measure – so that you get cheap quantitative measures of quality of the product.  I’m not suggesting that you should use all of them, but you could definitely measure them and use at least (say) half a dozen or so when defining a “quality bar” for a product or solution.  In particular given people’s tendency (as everyone does under tight timescales) to get over-focussed on the proxies – it’s a good idea to use a whole bunch of them to measure quality to prevent you making poor short-term decisions while thinking that you’re not affecting quality.

I’ve also marked ones that you could measure internally before you ship – as those let you get some kind of measurable handle on quality before the product goes out the door.

• Function (set and that it works).  Proxies:
  • (External) Functional Issues (number and severity) raised by customers
  • (Internal) Functional issues (number and severity) raised by ST or solution teams
  • (External) % of shipped enhancements with follow-on enhancement requests / complaints that we’ve delivered the wrong thing
• Simplicity/Complexity.  Extensibility.  Proxies:
  • (External/Internal) Average elapsed time from issue raised to fix.
  • (External/Internal) Average number of fix attempts per bug.  (%of the time that a fix resolves the issue.)
  • (Internal) Average number of dead-on-arrival entries into ST  – or –  Average # of solution team acceptance tests failing when product team SLA tests pass
  • (Internal) Average sprint velocity for each team (sprint velocity is also about the people, not just the product, but it’s not clear to me that they’re completely disentanglable in scrum)?
• Diagnostics.  Proxies:
  • (Internal / External) Average number of repros required before fix is understood (customers/support, solution/product, within product team)
  • (Internal) Average number of misdiagnosed issues going from Solution or ST team to wrong product or dev team
• Documentation.  Proxies:
  • (Internal / External) Number of solution failures (solution teams and customer teams) resolved as misconfiguration.
  • (Internal / External) Average time to bring up new system/solution.
  • (Internal / External) Number of issues resolved with docs fix.
• Integration/Installation/Upgrade. Proxies:
  • (External) Elapsed time from new release to #customers deployed.
  • (Internal / External) Average time to install a new system / product / solution
  • (Internal / External) Average downtime when upgrading a system / product / solution, including how impacting the downtime is (e.g. entire solution down for 2 hours vs. reduction in capacity for 1 hour, …)
• Supportability/Cost to us.  Proxies:
  • (External) Cost to support per $revenue (resource days).
• Wow factor/It/Sexiness.  Proxies:
  • (External) Customer ratings
  • (External) User install/uninstall rates.

There’s obviously loads more here  – both more areas and more metrics.  But even if you just look at all these different measures of quality, and the serious impacts that they can have on your ability to deliver and support products, it’s blindingly obvious that we need to measure more than just number of bugs when gauging quality!  And it’s also obvious that it’s pretty cheap to measure a bunch of these.

A final caveat though – while measuring a bunch of these is clearly better than just measuring bugs or field defects – I’m pretty sure there’s even more we could do here in terms of thinking about, talking about, and understanding quality without just using numbers.  After all, this quality bar is easy to measure (it’s 98m long), but this quality bar is better.

Demosthenes on Software Projects

Demosthenes (probably)

One great thing about working where I do is the continual surprise of what my coworkers know and are keen to share and enthuse about.  Kudos to my colleague, Rob Day, who courtesy of his history (ancient and modern) degree recently distracted me with his favorite “thing to read when the project is going south”.

The context for this speech: it’s basically a pep-talk from Demosthenes to the Athenians getting them to rise up against Philip of Macedonia.  The section Rob pointed me to that resonated is as follows:

“For though the state of our affairs is in every way deplorable, and though much has been sacrificed, nevertheless it is possible, if you choose to do your duty, that all may yet be repaired.  And what I am going to say may perhaps seem a paradox, but it is true. The worst feature of the past is our best hope for the future. What, then, is that feature? It is that your affairs go wrong because you neglect every duty, great or small; since surely, if they were in this plight in spite of your doing all that was required, there would not be even a hope of improvement. But in fact it is your indifference and carelessness that Philip has conquered; your city he has not conquered.”

That’s quite a powerful message – and useful on any project.   When things aren’t going well, we have a tendency to (a) feel bad and (b) double-down and work harder on what we’re doing.  Demosthenes points out that if things really aren’t going well, that’s (probably) because we’re not doing the right things – and that’s a good thing and we should feel good about it, because if we were doing our best and we were in this position we’d be screwed.  So lets enthusiastically find out what we’re not doing well, and sort it out(tm).  That’s not an easy message to swallow, but the way Demosthenes puts it, it’s suprisingly positive, and his rhetoric rams it home in a rather uplifiting way.

For reference, translation of the whole text is here (or here if you prefer bite-sized chunks).  Go read some great oration!

 

Book: Superforecasting: The Art and Science of Prediction

 

On the face of it, this book is an interesting read about predicting the future, how we humans tend to do it really badly (even those of us who’s job it is to do so), and an account of a set of people setting out to do better and how they approach that.  It’s a good book, and if any part of your job involves attempting to guess how something will pan out (for example, how long something might take) then it’s worth a read – both for advice on what and what not to do.

I accidentally ended up (re-)reading this in conjunction with another book, Black Box Thinking, and a lot of the rational-thinking advice overlaps heavily – for example on creating short feedback loops.  In addition, this book talks more precisely about being careful to measure yourself and (especially useful in the modern day) actively looking for information and sources of information that are new and different from that you have.

Light enough to be a pretty easy-going read.  Enough technical guts and applicable ideas to be useful and worthy of study and actions.  I recommend it.

Can I? vs Must I?

Argh – Inconsistent   Capitals and tErrible  kerning!

This started out as a simple post contrasting differences in thought between a “developer” mindset and a “tester” mindset.  However, as I’ve thought further, I think there’s more to it.  This is a “Vague pondering” post, not an “Aha, here is a thing!” post.

Imagine you have something coming up.

• It’s something that you want to do.  The brain is focussed on “Can I?” and is automatically supplying “good logical” reasons to do it, discarding or fixing problems that come up.
• It’s something that you don’t want to do.  The brain is focussed on “Must I?” and is automatically supplying “good logical” reasons not to do it, suppressing or refactoring those arguments to avoid fixes from others.

At work we come across tasks that we do and don’t want to do, and a lot of work on “getting stuff done” is recognising that we have these two mindsets and using tricks to get ourselves out of the “Must I?” mindset and into the “Can I?”  There’s loads of good stuff out there on this and I’m not going to rehash all that.

Instead I realize that when testing (aka analysing and exploring something) you probably want to be asking both of these questions.

• “Can I” is a great question to ask to get you places.  It focuses you on ways through and round problems, finding the narrow path through the minefield.  You’re harnessing your subconscious to let you get through areas and into new areas for fast exploration.
• “Must I” is a great question to ask to find problems.  It lets you branch off the “what’s expected” beaten track and you’re harnessing your subconscious to dig your heels in, spot things looking hinky and being generally alert for any way to not do the thing you’re “meant” to be doing.

That second “Must I?” question is the key mindset to get into when testing something;  Not “Must I do this bit of testing work?”, but “I’m testing this area – must I investigate in this expected way?”  That’s an interesting dichotomy of head-states (positive about the task of investigating, negative about the thing you’re investingating).

There’s no literature that I can find out there about how to consciously get yourself into a “negative” headstate.  If anyone sees any or can point some my way, please do!  I’d be interested in trying it out.

 

More “Must read” testing books

A book, not obviously about testing, but may be helpful for testers.

Just a quick one here.  There are a load of good books out there about testing – I’ve commented on a few on this blog.  However, the linked blog here lists 10 books that aren’t obviously about testing, but are helpful for testers.  I’ve read/studied 1, 2 and 8 and would recommend them, so I suspect the others are good too.

If you’ve read any and have recommendations for which I should read first, let me know…

 

Book – Black Box Thinking

A source of feedback

Black box thinking is ostensibly a book about management (we read it in our managers’ book club) but really it’s a book about about testing.  The whole book is about setting out how to measure your results rationally (rather than rationalizing your results), setting yourself up to test and fail at what you do, how to approach failing in what you do and then learn from it.  In passing it explains clearly why we find testing really hard and why we’re so quick to believe that we’re actually really good at testing.  It may also make you feel more comfortable about flying and never want to go near a hospital again!

In really brief summary, if you want your organisation to truly get better, create short, clinical feedback loops, question the obvious things that you all know work really well, and  identify, enjoy, and reward failure (as long as it comes with learning).  But to get what it actually means to do that well, and the consequences for not doing it, you need to read the book.

I’d really recommend this book, not just to people trying to improve their testing skill, but to anyone interested in getting actually better at what they do.

2-4-6 – Humans are instinctively poor at testing

4-6-2 does not fit the pattern

Confirmation bias is a really big problem that stalks us when we attempt to test things, making us complacent about how good a job we’re doing.  I hadn’t realised how much of a problem it is until recently.

When playing testing games with friends and peers, and when teaching new grads about testing, I often use the dice game and the 2,4,6 Wason Rule Discovery problem.  (The youtube link embedded in the linked page covers the problem and confirmation bias better than I’d summarise here – if you don’t know about the problem, go and watch that.

The Rule Discovery problem in particular is a really good example of how confirmation bias affects us and gets in the way of testing.   Although subjects given this task typically expressed high confidence in their guesses, only ~20% of the subjects in Wason’s original experiment successfully guessed the real rule, and replications since then have continued to show success rates of around 20% [citation needed].

But I now think that the issue is worse than I thought…

When trying these games with my peers, one of the issues I regularly see is people (including myself) hampering themselves by being reluctant to be wrong or appear “stupid” in front of their peers.  And I’d always somewhat assumed that that was part of what drives/conflates with the symptoms we see that apparently involve confirmation bias.

About a month ago, however,  I was talking to some year 9s about STEM (Science, Technology, Engineering, Maths) careers.  As part of this, I played around with a couple of puzzles, including the dice game, and the 2,4,6 problem.  The year 9s were very bright and were also much less constrained with their ideas than most people I’d come across, and totally unashamed about “silly” ideas or being wrong.  However, they still had the same positive gluing onto trying to prove the rules they came up with rather than disproving them.

Which is a shame, because I’m reasonably good at noticing when I don’t want to embarrass myself, but it still takes me conscious effort to force myself to look for ways to prove that I’m wrong.

 

 

 

 

 

Ask A Manager

Really handy blog

So when I started writing this blog, I envisaged that sometimes interesting questions would come up regarding people management, which I could write about.  And they do (people are way more interesting than computers).  However, most of the “this happened and I learned a thing” thoughts I have aren’t easy to safely anonymise – especially given the pretty small readership of this blog is mostly from my place of work (Hi folks!)

So instead, here’s the general advice I try and live by, and try and encourage in others.  It’s pretty simple, and is a good starting point for managers, managees and coworkers trying to cover any kind of problem or issue.

1. Talk about the issue to people who can actually help close the issue down.  The number of issues I’ve encountered where “Bitch all about this issue in the pub or on one of our chat channels” was considered an acceptable replacement for dealing with it really surprised and shocked me, until I realised that I used to do exactly that.
2. Initially assume good faith.  Differing viewpoint >> Partial Ignorance or Partial Incompetence >> Malice.  Even if you already think you know where the other person is,  be ready to discover that they’re back towards the “not-a-bad-person” end of the list.
3. Bring your adult ego state to the conversation, and engage their adult.  If they bring their parent or child, then you can engage with that, but be aware that for any interesting issue, you won’t be able to close it out until your adults have engaged and accepted the next steps.  If this doesn’t make sense, read the first few chapters of Games People Play by Eric Berne.  Maybe someday I’ll try and post a summary, but I’ve tried and given up before.

I’ve said the advice above is simple.  That doesn’t necessarily mean it’s easy.  In particular, there are two problems that I keep bumping up against (with myself as much as with others).  Over the years I’ve converged on two blogs, which I recommend as they help with these.

1. Being honest and adult.  Our brains tend to make stories that paint us as heroes, and we don’t like to see anything against that – which means having grown up conversations, assuming that the other person isn’t a total arsehole, maintaining your cool and adult engagement, etc. is really damn hard.  Ask a Manager is full of useful advice for how to approach issues sensibly and pitfalls to avoid, and if you look on there, probably the one you have (or a similar one) is covered already – probably from both sides!
2. Talking to other people.  Especially where I work (software company full of geeks), actually talking to people, about issues, that involve conversations that might have tension in, can be daunting.  For this, I’ve found Captain Awkward useful.  It’s not specifically work related (though the “Work” section is large), but it contains lots of conversational starters – Captain Awkward calls them “scripts” – which are good ways to help break that initial barrier.

 

=== Edit 2017/10/08 – Fixed Ask A Manager link

Developing your testing using jokes

Why did the developer cross the road?  To get to the others’ ide.

Another in a somewhat sporadic series of “things you can do in your spare time that are fun and hone some of your testing skills.  This one comes from a session a year or two back at the Assurance Leadership Forum (formerly Test Management Forum) by James Thomas.

Joke creation flexes just those same skills you need for testing.  Logic, reasoning and deduction, lateral thinking, breaking assumptions, and that intuitive flare for spotting the edge of something that niggles and pulling at it until it unravels.

And it turns out, that James has now released an eBook which you can read here (don’t panic – it’s a 20 page PDF) that talks through and unpicks some of this far better than I could explain.  I recommend the read, and then giving creating some Jokes a go!