Test Strategy – a handy checklist

More guidelines than actual rules
From XKCD 1287


As part of trying to educate other people at work about testing strategy, I’ve been thinking about it a lot recently – in particular because I keep seeing similar issues and often struggle to concisely get across.  While I’m still working on that thinking (hopefully post(s?) to follow), one thing I keep coming back to that’s really useful is one of the checklists from the Rapid Software Testing Appendices – “a set of considerations designed to help you test robustly or evaluate someone else’s testing”.  It’s not the be-all-and-end-all, but a lot of the test strategies I come across miss thinking about a bunch of the stuff here, so using this is a good starting point when coming up with your strategy.

(All kudos for this list => The RST creators)

Project Environment

  • Mission. The problems you are commissioned to solve for your customer.
  • Information. Information about the product or project that is needed for testing.
  • Developer Relations. How you get along with the programmers.
  • Test Team. Anyone who will perform or support testing.
  • Equipment & Tools. Hardware, software, or documents required to administer testing.
  • Schedules. The sequence, duration, and synchronization of project events.
  • Test Items. The product to be tested.
  • Deliverables. The observable products of the test project.

Product Elements

  • Structure. Everything that comprises the physical product.
  • Functions. Everything that the product does.
  • Data. Everything that the product processes.
  • Interfaces. Every conduit by which the product is accessed or expressed.
  • Platform. Everything on which the product depends (and that is outside your project).
  • Operations. How the product will be used.
  • Time. Any relationship between the product and time.

Quality Criteria Categories

  • Capability. Can it perform the required functions?
  • Reliability. Will it work well and resist failure in all required situations?
  • Usability. How easy is it for a real user to use the product?
  • Charisma. How appealing is the product?
  • Security. How well is the product protected against unauthorized use or intrusion?
  • Scalability. How well does the deployment of the product scale up or down?
  • Compatibility. How well does it work with external components & configurations?
  • Performance. How speedy and responsive is it?
  • Installability. How easily can it be installed onto it target platform?
  • Development. How well can we create, test, and modify it?
  • (I’d explicitly break out from that last one
    • Testability.  How well can we put the product into the state we want to put it into for testing?
    • Diagnosability.  How well can we tell what state the product is in?
    • Debugability.  How easily can we work out what has happened?
    • Maintainability.  How easily can we fix issues in or enhance the product?)

General Test Techniques

  • Function Testing. Test what it can do.
  • Domain Testing. Divide and conquer the data.
  • Stress Testing. Overwhelm the product.
  • Flow Testing. Do one thing after another.
  • Scenario Testing. Test to a compelling story.
  • Claims Testing. Verify every claim.
  • User Testing. Involve the users.
  • Risk Testing. Imagine a problem, then find it.
  • Automatic Checking. Write a program to generate and run a zillion checks.

Measurable Quality Bars


A measurable quality bar in St. Pancras Station, London.  It’s 98m long

This post came out of some notes I wrote up when trying to persuade some other parts of our organisation that measuring quality as “number of open bugs” was a really poor idea – specifically that it’s (a) not a metric for quality and (b) it strongly encourages short-termist behavior that lowers long term quality (prioritising immediate functional bugs over maintainability, extensibility, reducing regression counts, diagnosability, all those other “ilities”).  I’ve walked that path on so many projects, I’m keen to try to stop others following down the track…

Here goes:

What is quality – what are we trying to measure?

Before we think about how to measure quality, it helps to think about what is quality?  The best definition I’ve come across originated with Jerry Weinberg

Quality is value to some person (who matters)

So for our products, who matters, and what do they care about?

From a quickish brainstorm meeting a year or few back, an incomplete set is:

  • Who matters: Our customers, their customers, support, solutions teams, development teams, test teams, product management, trials/demo teams, Sales, our C-suite.
  • What matters to them (not all things matter to all people): Function set, working function, reliability, price, usability, diagnostics, debuggability, maintainability, extensibility, documentation, standards compliance, integration, interop, configurability, simplicity/complexity, “IT”/Sexiness/Wow factor, supportability.

Looking at all those, it seems that quality is a pretty hard thing to pin down or measure.  We do like measuring things though, and what we can do is measure some proxies for quality.  Note – these proxies aren’t “quality” – they’re things that seem to align well with something that matters to someone (so part of quality).

So what can we set out to measure that would do better?

Easy(ish) to measure proxies for quality (“quality metrics”)

My goal with this list is to think of proxies that are easy to measure – so that you get cheap quantitative measures of quality of the product.  I’m not suggesting that you should use all of them, but you could definitely measure them and use at least (say) half a dozen or so when defining a “quality bar” for a product or solution.  In particular given people’s tendency (as everyone does under tight timescales) to get over-focussed on the proxies – it’s a good idea to use a whole bunch of them to measure quality to prevent you making poor short-term decisions while thinking that you’re not affecting quality.

I’ve also marked ones that you could measure internally before you ship – as those let you get some kind of measurable handle on quality before the product goes out the door.

  • Function (set and that it works).  Proxies:
    • (External) Functional Issues (number and severity) raised by customers
    • (Internal) Functional issues (number and severity) raised by ST or solution teams
    • (External) % of shipped enhancements with follow-on enhancement requests / complaints that we’ve delivered the wrong thing
  • Simplicity/Complexity.  Extensibility.  Proxies:
    • (External/Internal) Average elapsed time from issue raised to fix.
    • (External/Internal) Average number of fix attempts per bug.  (%of the time that a fix resolves the issue.)
    • (Internal) Average number of dead-on-arrival entries into ST  – or –  Average # of solution team acceptance tests failing when product team SLA tests pass
    • (Internal) Average sprint velocity for each team (sprint velocity is also about the people, not just the product, but it’s not clear to me that they’re completely disentanglable in scrum)?
  • Diagnostics.  Proxies:
    • (Internal / External) Average number of repros required before fix is understood (customers/support, solution/product, within product team)
    • (Internal) Average number of misdiagnosed issues going from Solution or ST team to wrong product or dev team
  • Documentation.  Proxies:
    • (Internal / External) Number of solution failures (solution teams and customer teams) resolved as misconfiguration.
    • (Internal / External) Average time to bring up new system/solution.
    • (Internal / External) Number of issues resolved with docs fix.
  • Integration/Installation/Upgrade. Proxies:
    • (External) Elapsed time from new release to #customers deployed.
    • (Internal / External) Average time to install a new system / product / solution
    • (Internal / External) Average downtime when upgrading a system / product / solution, including how impacting the downtime is (e.g. entire solution down for 2 hours vs. reduction in capacity for 1 hour, …)
  • Supportability/Cost to us.  Proxies:
    • (External) Cost to support per $revenue (resource days).
  • Wow factor/It/Sexiness.  Proxies:
    • (External) Customer ratings
    • (External) User install/uninstall rates.

There’s obviously loads more here  – both more areas and more metrics.  But even if you just look at all these different measures of quality, and the serious impacts that they can have on your ability to deliver and support products, it’s blindingly obvious that we need to measure more than just number of bugs when gauging quality!  And it’s also obvious that it’s pretty cheap to measure a bunch of these.

A final caveat though – while measuring a bunch of these is clearly better than just measuring bugs or field defects – I’m pretty sure there’s even more we could do here in terms of thinking about, talking about, and understanding quality without just using numbers.  After all, this quality bar is easy to measure (it’s 98m long), but this quality bar is better.

Demosthenes on Software Projects

Demosthenes (probably)

One great thing about working where I do is the continual surprise of what my coworkers know and are keen to share and enthuse about.  Kudos to my colleague, Rob Day, who courtesy of his history (ancient and modern) degree recently distracted me with his favorite “thing to read when the project is going south”.

The context for this speech: it’s basically a pep-talk from Demosthenes to the Athenians getting them to rise up against Philip of Macedonia.  The section Rob pointed me to that resonated is as follows:

“For though the state of our affairs is in every way deplorable, and though much has been sacrificed, nevertheless it is possible, if you choose to do your duty, that all may yet be repaired.  And what I am going to say may perhaps seem a paradox, but it is true. The worst feature of the past is our best hope for the future. What, then, is that feature? It is that your affairs go wrong because you neglect every duty, great or small; since surely, if they were in this plight in spite of your doing all that was required, there would not be even a hope of improvement. But in fact it is your indifference and carelessness that Philip has conquered; your city he has not conquered.”

That’s quite a powerful message – and useful on any project.   When things aren’t going well, we have a tendency to (a) feel bad and (b) double-down and work harder on what we’re doing.  Demosthenes points out that if things really aren’t going well, that’s (probably) because we’re not doing the right things – and that’s a good thing and we should feel good about it, because if we were doing our best and we were in this position we’d be screwed.  So lets enthusiastically find out what we’re not doing well, and sort it out(tm).  That’s not an easy message to swallow, but the way Demosthenes puts it, it’s suprisingly positive, and his rhetoric rams it home in a rather uplifiting way.

For reference, translation of the whole text is here (or here if you prefer bite-sized chunks).  Go read some great oration!


Book: Superforecasting: The Art and Science of Prediction



On the face of it, this book is an interesting read about predicting the future, how we humans tend to do it really badly (even those of us who’s job it is to do so), and an account of a set of people setting out to do better and how they approach that.  It’s a good book, and if any part of your job involves attempting to guess how something will pan out (for example, how long something might take) then it’s worth a read – both for advice on what and what not to do.

I accidentally ended up (re-)reading this in conjunction with another book, Black Box Thinking, and a lot of the rational-thinking advice overlaps heavily – for example on creating short feedback loops.  In addition, this book talks more precisely about being careful to measure yourself and (especially useful in the modern day) actively looking for information and sources of information that are new and different from that you have.

Light enough to be a pretty easy-going read.  Enough technical guts and applicable ideas to be useful and worthy of study and actions.  I recommend it.

Can I? vs Must I?

Argh – Inconsistent   Capitals and tErrible  kerning!

This started out as a simple post contrasting differences in thought between a “developer” mindset and a “tester” mindset.  However, as I’ve thought further, I think there’s more to it.  This is a “Vague pondering” post, not an “Aha, here is a thing!” post.

Imagine you have something coming up.

  • It’s something that you want to do.  The brain is focussed on “Can I?” and is automatically supplying “good logical” reasons to do it, discarding or fixing problems that come up.
  • It’s something that you don’t want to do.  The brain is focussed on “Must I?” and is automatically supplying “good logical” reasons not to do it, suppressing or refactoring those arguments to avoid fixes from others.

At work we come across tasks that we do and don’t want to do, and a lot of work on “getting stuff done” is recognising that we have these two mindsets and using tricks to get ourselves out of the “Must I?” mindset and into the “Can I?”  There’s loads of good stuff out there on this and I’m not going to rehash all that.

Instead I realize that when testing (aka analysing and exploring something) you probably want to be asking both of these questions.

  • “Can I” is a great question to ask to get you places.  It focuses you on ways through and round problems, finding the narrow path through the minefield.  You’re harnessing your subconscious to let you get through areas and into new areas for fast exploration.
  • “Must I” is a great question to ask to find problems.  It lets you branch off the “what’s expected” beaten track and you’re harnessing your subconscious to dig your heels in, spot things looking hinky and being generally alert for any way to not do the thing you’re “meant” to be doing.

That second “Must I?” question is the key mindset to get into when testing something;  Not “Must I do this bit of testing work?”, but “I’m testing this area – must I investigate in this expected way?”  That’s an interesting dichotomy of head-states (positive about the task of investigating, negative about the thing you’re investingating).

There’s no literature that I can find out there about how to consciously get yourself into a “negative” headstate.  If anyone sees any or can point some my way, please do!  I’d be interested in trying it out.


More “Must read” testing books

A book, not obviously about testing, but may be helpful for testers.

Just a quick one here.  There are a load of good books out there about testing – I’ve commented on a few on this blog.  However, the linked blog here lists 10 books that aren’t obviously about testing, but are helpful for testers.  I’ve read/studied 1, 2 and 8 and would recommend them, so I suspect the others are good too.

If you’ve read any and have recommendations for which I should read first, let me know…


Book – Black Box Thinking

A source of feedback

Black box thinking is ostensibly a book about management (we read it in our managers’ book club) but really it’s a book about about testing.  The whole book is about setting out how to measure your results rationally (rather than rationalizing your results), setting yourself up to test and fail at what you do, how to approach failing in what you do and then learn from it.  In passing it explains clearly why we find testing really hard and why we’re so quick to believe that we’re actually really good at testing.  It may also make you feel more comfortable about flying and never want to go near a hospital again!

In really brief summary, if you want your organisation to truly get better, create short, clinical feedback loops, question the obvious things that you all know work really well, and  identify, enjoy, and reward failure (as long as it comes with learning).  But to get what it actually means to do that well, and the consequences for not doing it, you need to read the book.

I’d really recommend this book, not just to people trying to improve their testing skill, but to anyone interested in getting actually better at what they do.