Month: August 2017

Book – Black Box Thinking

A source of feedback

Black box thinking is ostensibly a book about management (we read it in our managers’ book club) but really it’s a book about about testing.  The whole book is about setting out how to measure your results rationally (rather than rationalizing your results), setting yourself up to test and fail at what you do, how to approach failing in what you do and then learn from it.  In passing it explains clearly why we find testing really hard and why we’re so quick to believe that we’re actually really good at testing.  It may also make you feel more comfortable about flying and never want to go near a hospital again!

In really brief summary, if you want your organisation to truly get better, create short, clinical feedback loops, question the obvious things that you all know work really well, and  identify, enjoy, and reward failure (as long as it comes with learning).  But to get what it actually means to do that well, and the consequences for not doing it, you need to read the book.

I’d really recommend this book, not just to people trying to improve their testing skill, but to anyone interested in getting actually better at what they do.


2-4-6 – Humans are instinctively poor at testing

4-6-2 does not fit the pattern

Confirmation bias is a really big problem that stalks us when we attempt to test things, making us complacent about how good a job we’re doing.  I hadn’t realised how much of a problem it is until recently.

When playing testing games with friends and peers, and when teaching new grads about testing, I often use the dice game and the 2,4,6 Wason Rule Discovery problem.  (The youtube link embedded in the linked page covers the problem and confirmation bias better than I’d summarise here – if you don’t know about the problem, go and watch that.

The Rule Discovery problem in particular is a really good example of how confirmation bias affects us and gets in the way of testing.   Although subjects given this task typically expressed high confidence in their guesses, only ~20% of the subjects in Wason’s original experiment successfully guessed the real rule, and replications since then have continued to show success rates of around 20% [citation needed].

But I now think that the issue is worse than I thought…

When trying these games with my peers, one of the issues I regularly see is people (including myself) hampering themselves by being reluctant to be wrong or appear “stupid” in front of their peers.  And I’d always somewhat assumed that that was part of what drives/conflates with the symptoms we see that apparently involve confirmation bias.

About a month ago, however,  I was talking to some year 9s about STEM (Science, Technology, Engineering, Maths) careers.  As part of this, I played around with a couple of puzzles, including the dice game, and the 2,4,6 problem.  The year 9s were very bright and were also much less constrained with their ideas than most people I’d come across, and totally unashamed about “silly” ideas or being wrong.  However, they still had the same positive gluing onto trying to prove the rules they came up with rather than disproving them.

Which is a shame, because I’m reasonably good at noticing when I don’t want to embarrass myself, but it still takes me conscious effort to force myself to look for ways to prove that I’m wrong.