Be a better tester: Ask “Why?”

Humans do not have a “How?” chromosome.

I’ve recently been tasked with educating some of our development teams in “better testing”.  One of the key things I keep coming across is a mindset change, and I realised that while it’s a key part of good testing, it’s also something that I found critical when stepping up to take on lead development roles too…

When I was learning to be a developer, I was trained heavily into asking “How?”  How do we implement this?  How does that work?  How can we make this happen?  How do I fix this?   “How?” is a good question to ask, but once it becomes your instinctive reaction to think “How?”, you’re in trouble.  “How?” is a focusing question, and almost always, you want to start by defocusing and getting the context that you’re working in.  An easy way to do that is to train your first instinct to be to ask “Why?”

Why are we implementing this?  Why does the product work this way?  Why does that happen?   Not only is instinctively asking “Why?” a basic requirement for testing, it’s also the question that will lead you to big gains in development.  Because they lead to us thinking about the context for the work we’re being asked to do, and stop us carefully implementing the wrong thing, or implementing something great, but isn’t extensible in the right way, or implementing something that’s way too over-engineered for what any customer will ever want.

We all (should!) use “How?” and “Why?” but the difference between a good developer and a great one, and an important difference between a developer and a tester is that trained instinct to make “Why?” the first question to ask.



Manager time vs Maker time

Mostly things go better when you don’t interrupt.

Managers and Makers (people who actually get real work done) work in fundamentally different ways.  I was going to spend time explaining this in detail, but Paul Graham has already done it better than I could.  Go and read that, and then come back.

TL;DR Makers mostly work doing deep thinking in large blocks of time.  Most manager work requires broad thinking in small blocks of time.  Don’t forget that your people work different, and to be effective, you need to allow for that.  As a manager, there are two things that you can do to help.

  • Stop interrupting them!
  • Prevent other people from interrupting them!

Some straightforward things that you can try (I’ve tried or seen all of these with various effects).

  • Meetings only at start of day or just before/after lunch.  (Easy and obvious!)
  • “No morning meetings.”  (Our team have a stand-up first thing, and then I try and prevent other meetings before lunch).
  • “No meeting Thursdays”  (One day a week that your team refuses to do meetings.  You can really get stuff done.  Publicise widely what day you’re not available.)
  • “Cans on; can’t disturb.”  Headphones on – no interruptions.  If you want to interrupt someone on my team with headphones on, you have to come past me first.

Whether you’re a manager or a maker, I recommend trying one or more of these out, or invent your own.  People might grumble to start with, but the actual cost (beyond a little bit of thought up front) is very low, and the benefits high.


Book: 12 Days of Christmas

A salutory lesson on testing

This post is really little more than a chance to link to a favourite book.  But this book also serves as a reminder that it doesn’t matter how closely you explore conformance to some specification or intent, it is ultimately the utility to and happiness of the customer that matters.  If you have a spare five minutes,  enjoy.  Happy Christmas.


Testers don’t break things


I’m not a monster, I’m just ahead of the curve.

I’m a tester; I break things.  I’ve heard and said it many, many times.  It’s the standard way people describe testing.  Quite apart from making testing sound like a simple, easy job (worthy of a blog post in and of itself), it’s a dangerous terminology.

It suggests that the product is ok, until the testers have their wicked way with it – that the testers are malicious little buggers who try to make the product do things it’s not meant to, and then make it fall over.  The very terminology leads to fallacies of thought:

  • Developers can think that they’ve passed the product to test, good to ship.
  • Managers can think that because the testers haven’t broken the product, it’s good to ship.
  • Everyone can think that the testers have stopped the product shipping successfully.
  • Bugs get rejected, because people think that no true user would do that. 

In reality, the product given to the testers is not good.   It’s given to them already broken; a haunted house with booby traps galore.  The testers aren’t rampaging lunatics with sledge-hammers; they’re the housing inspectors sent to find those traps before the general public are allowed in.  Except that software products are a lot more complicated and much less easily perfected than a house,  so there are a lot more dark holes for the bugs to lurk and deciding where and how to shine the torches around to light them up is slower and more complicated.

So, I’m going to try and change my default language.  I’m a tester.  I investigate things.  In my vicinity, stones get turned over and creepy crawlies come to light.  When there might be a can of worms around, I’m the one wielding the tin-opener.  But I no longer claim to break things.

[1] Picture of the Joker, by arthurforzus on deviantart.  I don’t know him; I was just looking for pictures of the Joker with appropriate permissions and thought it was rather good.





Testing, Risk and Quality in The Martian

A Martian demonstrating risk


Thinking about testing conflict depicted in The Martian made me think a bit closer about the consequences of defining quality as value to someone who matters.

I recently watched The Martian.  At one point, NASA are trying to launch a space probe in a hurry – and cut 10 days of launch site testing having established that historically there’s only been a  1/20 chance of finding an issue during that tesing.  It was unusual to see a film actually covering the concept of “How do we minimise risk with the resources that we have?” rather than just having established process vs a maverick.

It was also an interesting scene (and film) in that having decided to launch a resuce mission (I’ll leave aside the rationality behind that for this post) the film draws out in several places the contrast between the logically optimal strategy (maximising the chances of rescuing a stranded astronaut) and the socially acceptable strategy (minimising the damage to NASA’s public reputation and future spaceflight if something goes wrong with that rescue mission).  As a specific example, the choice above to dodge some testing was a great logical way to effectively save time at a 5% risk cost, which could then be spent more effectively elsewhere to reduce the risk by more than that.  The problem was that specifically choosing not to do some testing that you might otherwise have done is hard to explain away if things go wrong – when 20/20 hindsight suggests that was the wrong call.

Most of us see that as a conflict between the “right thing to do” and some stupid politics.  However, looking further…

If the goal of some process is to maximise the quality of something, and we define quality using the standard “value to someone who matters”, then the above conflict is simply that we have multiple people who matter with different values.  That means there’s not a “right” and “wrong” argument above – maximising quality means making a compromise and balancing the different “values”.

And that matches up with what we see in software development.  We run QA phases and focus on mainline install processes, not to optimise for globally reduced risk – we could spend that time finding and fixing more other bugs – but  because the customer disproportionately values “things working at first”, and “lack of regressions”.  While it might not seem like we’re optimising for quality, we are – it’s that quality is defined by value to people that matter, and their views of quality may be different to ours or expressed in different ways.


What is Quality?

Clearly the best kind of street.

What is it that we’re trying to achieve with the software development process[1]?  Most people agree that roughly speaking we’re trying to maximise the quality of our product or solution, within some constraints (money/time/people/etc.).  That’s all well and good, but what is quality?

Oxford says: Quality – The standard of something as measured against other things of a similar kind; the degree of excellence of something.

Ok, but not that helpful if you’re trying to maximise it.  And it turns out that when you try to define it down more tightly, you rapidly run into strong differences;  For example, in Korea, quality is “Brand new, cutting edge”,  which is a very different thing to optimise for from the UK  “Solid, lasts, doesn’t break”.  (For more examples, try this interesting post.)

Jerry Weinberg has a good definition, which I’ve heard around a few times now: Quality is value to some person who matters.  It sounds a bit trite, but I love it because you can do things with it.  Identify who matters.  Find out what they value.  Then you can start optimising that.


[1] I use “software development process” as a whole rather than “testing” to sidestep the “Why do we test?”, “Are testers trying to improve quality?” question.  That’s probably worth a separate post at some point.  Regardless, as a tester, you’ll make better decisions if you have the context of what your overall team and organisation are trying to achieve.



There’s no substitute for testing

This is the story of how an award-winning building that was an architectural and engineering triumph very nearly ended up as a not-at-all-award-winning heap of rubble but was saved by some last minute testing.

Engineering nerds can also check out the flat arches over the windows.
The Queen’s Building, Emmanuel College, Cambridge, UK

This is the Queen’s building at Emmanuel College, Cambridge.  It’s a pretty cool building and an even cooler piece of engineering.  It’s made of the same limestone (from the same quarry) that the rest of the college was built from some 400 years earlier, but where the original buildings have the massive metre thick walls required when working with such a weak stone, the Queen’s building doesn’t.  The reason it doesn’t just collapse is hinted at by the little port holes that you can see dotting the building.  Limestone, like concrete is weak under shear or tension forces, but strong under compression.  If you looked inside one of those port-holes you’d see the ends of steel rods running down the centre of each stone column, pre-compressing it, similar to pre-stressed concrete.

Being a brand new construction technique, a bunch of bright people did careful calculations to work out what pressures the stone would be under, what the stresses and strains would be, and modelled everything to make very sure that the stresses on the building were well within the tolerances required – all good stuff.  Work started.

Enter, Prof. Chris Burgoyne, a fellow of Emmanuel College, and structural engineer, who worked through the plans and calculations and said, “That’s great in theory – but we need to test that.”  Chris had spent much of his life trying to remove over-confidence in models from his engineering students and had an engineering lab just down the road, so despite reassurances that it would all be just fine, insisted on the testing.

Nothing is more satisfying than finding a bug
Test column in the Engineering Lab, Cambridge University, UK.

This is one of the test columns[1] in a load press in the Cambridge University Engineering Lab.  If you look closely, you can see the cracks as it comes apart at significantly lower than the safe load required.  This posed a bit of a problem, as the building was already partially built!

So what had happened?

The models were right, in as much as they were correctly modelling things.  The pre-tension technique is sound.  The building stands Today as a quiet triumph of ingenuity and engineering brilliance[2].  However, three things were causing the columns to crumble – none of which had been identified in the original models but all of which became obvious in testing.

  • The modern mortar mix used, when placed on the aborbent limestone, set very hard, very quickly, so instead of the load carried uniformly across the surface, the surface was rough causing a relatively small number of contact points, which were acting as pressure points to start fractures.
  • The blocks that made up the columns had cramps (basically big steel staples) holding them in place.  These too were acting as pressure points.
  • Some of the limestone blocks had the bedding planes (the lines where layers of sand were laid on top of each other before being squished by Geology) aligned vertically.  Limestone can take much higher pressure when the bedding planes are horizontal – think standing on a stack of paper laid horizontally or vertically!

Today, in addition to the tensioning rods originally planned, the Queen’s building has some additional unusual features which you can’t see.  It is built with very thin Roman-style lime cement (which sets slower and more putty like and so doesn’t cause fracture points).  There are no staples holding the column blocks together, and the bedding planes in the sandstone blocks are all very carefully aligned.  In addition to being a triumph of architecture and engineering, it is also a triumph of testing.


[1] Technically it isn’t.  It’s the second of several smaller columns built and tested after the first column failed spectacularly.  However, most of my photos of this testing were very under-exposed and rubbish.  I was 13 at the time – I’d been allowed in as I was keen on becoming a structural engineer and Chris was a colleague of my dad’s and friend of the family.  I didn’t end up as a structural engineer, but did take to heart the importance of good testing!

[2] Paper on the structure of the Emmanuel College Queen’s Building

Edit: Updated with feedback from Chris Burgoyne, who corrected a few minor details and was also kind enough to provide the copy of the paper included above.