Testing as a Holistic Exercise

Early on when trying to apply Agile methods to software development I found a lot of the challenges were in the area of software testing. Not just because a lot more automated testing was being done, but because requirements would shift and change rapidly. Many levels of rapid feedback needed to be in place to ensure the development didn’t misstep for too long.

I started exploring more places a focus on testing and quality could be applied. And that led me on a journey earlier and earlier in the development timeline.

Testing after software is delivered

Often seen as the “traditional” place for testing to be done but I have to be careful when using such words because it means something different to everyone. My first training as tester was on such a test team. We waited for a release candidate and had two weeks – across 5 people – of regression testing to run. Getting a release out in a rush meant not running a full regression and was very risky. A typical release would spend 6-8 weeks just in testing and bug fixing. We didn’t have time for exploratory testing because the regression testing was so exhaustive. We didn’t want early releases unless we had all our test cases written and no other product releases to test. But if a release was late we would be looking for things to do to fill the time.

That’s not the type of testing I want now. Almost all regression testing should be automated. But the role of testing after a software change is done is not going away. There are some things that can’t be caught by test automation such as ensuring the visuals of the UI are still intact and functional. And we value the feedback of exploratory testing to find things that are hard to think of when writing a functional test. What happens when you run feature A with feature B? Can you do something too fast? Too slow?

Testing after a code change is done can tell us a lot about the experience our customers will soon face. But it shouldn’t be exhaustive or take up a lot of time.

Testing during development

Unit testing with Test Driven Development (TDD) techniques are core to Agile. But too often I find developers underestimate how long and challenging the learning process is. They think “I’m already an expert in things language, these tools and this type of product.” So it’s not that surprising that they get quickly frustrated with TDD and want to go back to doing things the way they did before.

Here is where a tester can help. A tester can guide programmers through writing short and small testable increments. In testing things the customer actually cares about to drive a design that will allow for refactoring later. Of asserting on outputs not mocked behaviours. I would like all testers to learn TDD because they are the best placed to teach and mentor programmers learning TDD.

Acceptance testing of features

Acceptance Test Driven Development (ATDD) is the best way I know to deliver stories in small increments. Without stories that have clear testable definitions of done it’s hard to know how much of a change is too big or too small. I see some teams will go into writing tiny technical tasks and horizontal slices because they’re not working with stories. Or create stories that take months of work to complete and require exhaustive testing and review when “done”.

Get the tester involved. At first you may ask the tester to write down the tests. Working with the business/product people to flesh out the behaviours that are expected. Then working with programmers to hook it all into the system and test the real code. Eventually they can guide others in the team to be writing these tests too.

Behaviour Driven Development (BDD)

When you have business people and developers working together every day you want to communicate features in a way that everyone understands. If your business people and customers are developers then you can probably skip using BDD. The key goal is to be able to share and discuss examples of behaviour before the code has even been written. I like the way it keeps me abstract from the implementation as well as enabling those who do not feel comfortable writing code to start writing tests. Now you’re writing tests when everyone is still figuring out the requirements.

The tests now become a broader communication and documentation tool. They are specifications more than they are tests. One of my favourite things is when I use these BDD tests to discuss behaviours and requirements with other teams. It gives us something that we can both understand but leaves no room for ambiguity. It’s not quite contract testing but it helps.

The Three Amigos

Is this as small as it can be? Is it testable? Is it valuable? These are why I want a test specialist in the room when we’re discussing the required work. Maybe your whole team is involved in the planning and you get that feedback there. But I like the Three Amigos approach better.

Every now and then we get together a product person, a developer and a tester to discuss some upcoming items in our development backlog. Things that have a title but haven’t been fleshed out more than that. For some teams I’ve done this as part of the normal cycle for any card. For my current team we do it before the card is considered ready.

We discuss the requirements and scope of the card. Who is the customer? What problem are we solving? How are we solving it? All three parties have valuable contributions and in some cases conflicting goals. The developer and tester want it smaller than the product person would like. The tester and product person what observable behaviour more than the developer. That kind of healthy conflict should help define the best options for the story.

What’s next?

Everything above is my preferred approach to testing in an Agile world. There is a lot a skilled tester can contribute to the team without having to be reduced to the “manual tester” or the “automation tester”. It is not what I would consider a shift-left approach to testing as we still value testing at all stages of development. And slowly I would want testers on the team to transition to just being the test specialist and eventually moving on to other challenges.

There is a lot left to explore still after all of that has been mastered. How can we test our product ideas in the market? How can we find out what is valuable rather than just guessing? We’re not asking the customer to “test our code”. We’re asking the company to validate their assumptions. Because for too long our product decisions have gone untested.

Test all the things at all the stages!
Geoff.