Cost of Manual Testing on Agile Projects

“Quality Is Free” – Philip Crosby

I still meet a lot of agile software developers and teams who are not applying proper test automation practices, such as micro testing, automated integration testing and continuous integration. There are three reasons for this.

1. Developers don’t see the value
Some developers are just so good and experienced, they make so little mistakes, that automated testing is adding overhead. These developers usually have more important things to do, like apply their über skills on architecture, while the lesser souls fix the bugs. Getting teams to apply test automation practices with these developers on board is probably the hardest.

2. Managers don’t see the value
Some managers think automated testing is extra work. And since they have projects to deliver, there is no time to be fooling around. Somehow they overlook the fact that most project risk and uncertainty is the bug fixing part before the product is ready to be deployed. If there are strong developers and scrum masters on the team, they will hopefully be able to convince management of the value, or if not do it anyway.

Time allocated to testing

3. Spagetti Legacy Code
Sometimes a code base has existed for a while without proper test automation practices, and has evolved in tightly coupled code with many dependencies. It is very hard to create micro tests for code which has many dependencies, even mocking is not an option anymore, since there are so many mock objects required for a single test, that development teams find it almost impossible to start with micro tests.

Usually the reasons 1. and 2. will eventually lead to reason number 3. In the beginning the code base is still manageable without test automation, since the total amount of code and features is still limited and little regression tests need to be run. So teams do not really see the problem they are going to face later. Over time the pressure on the manual testing of code and bug-fixing will increase, since more code will need to be regression tested and the code base will become larger and more complex to understand. Until at some point, half of their time, or even more, will be spent on testing and bug fixing, where Agile teams should focus on delivering customer value.

Preventing bugs is Key

What Philip Crosby meant with his quote at the top “Quality is free” is that prevention of defects will always be cost efficient. This definitively also applies to software development. This is best illustrated in the graph below by looking at the cost of defects, or bugs, in software and the phase in which they are found.

Time to fix software defects

Backlog Grooming

In a Backlog Grooming session, developers will validate or disqualify assumptions they have, by asking questions to the Product Owner. Acceptance Criteria will be updated and relevant notes will be added to the User Stories. Stories which are too big will be split in multiple Stories. By discussing requirements in detail, misconceptions will be eliminated and possible bugs will be prevented. The time required (cost) to eliminate bugs is minimal since not a single line of code has been written.

Micro Testing

When micro tests fail after changing parts of the code, the failing tests will pinpoint a developer to the new defects which are introduced. Since tests should be run often, the amount of code which was changed since the last time all tests passed will be so small, the developer can easily fix the bugs which were introduced, usually a question of minutes.

Integration Tests

When micro tests do not catch a certain bug, this bug will stay in the system until another test exposes it. When it is found in an automated integration test, usually changes from more developers have been added, so it will require a bit more time to allocate the offending code, and perhaps efforts from other developers are required to eliminate the bug. Since integration tests should run at least daily, the developers can still recall the changes which introduced this bug, and it is probably a question of a man hour or less to fix the problem.

Manual Acceptance Tests

When no automated tests will expose the bug, we can only hope it will be exposed by a manual test, for example in a staging environment. Usually these tests will be run at the end of a Sprint, or worse, just before a release. Weeks may have passed, or even months, and it usually involves a bug management system to track the bug. This causes additional overhead, since a Product Owner will need to understand the impact and prioritize, a Quality Engineer needs to find a way to reproduce and developers will have to dig deep in their brain to get back to the change they made weeks ago. In this stage the total combined efforts of everyone involved will be many hours to days to eliminate bugs.

Live bugs

If none of the above has uncovered a bug, it will be released in the wild, exposing it to your end users. This adds an additional cost, the perception of quality by your users. If the perceived quality is low, this is negative for your brand image, and users are likely to consider alternative products. Fixing bugs in a live environment will either mean in will be shipped in the next version, meaning your end users will experience the negative impact until then, or if the severity is high, a patched release will need to be released, with the additional overhead this will bring with it.

It comes with the job?

Have you ever walking in a room where a team has been fixing bugs for a week or more? It is probably full of low energy developers frustratingly looking at their debuggers, or figuring out what the heck a bug report is about. It’s not exactly the part of their job they enjoy most, especially not when frantic Product Owners are hovering around stressfully pushing them to get to a releasable build. It comes with the job is a phrase often heard. No! You haven’t done your job properly I say, that’s the reason these bugs are in the system, they should have been prevented by using the tools which are readily available and should be used in proper Software Craftsmanship.

The cost of bugs versus how long they exist in the system, clearly shows that investing time in Micro Testing and Automated Integration testing is worth your while. Investing a few hours now will save you endless hours of frustration debugging long forgotten code changes and misunderstandings in bug reports.

But how do we start?

It’s clear that it makes no business sense to start writing tests for the whole code base, since it will take months before you will start adding value for your end users again. So how do teams which finally see the light start using test automation? How did the chicken cross the road, one step at a time. When you didn’t start applying test practices from the start of your project, now is the time to get started. Use the ammunition above to convince the Product Owner (hopefully you don’t need to convince the Scrum Master) and have a agreement on changing the Definition of Done.

Additions to the Definition of Done

  • Each new User Story will need to be covered by Micro tests
  • Each new Story will need to be covered by Automated Integration tests.
  • New Bugs which are discovered in later stages will need to be covered by Micro tests and Automated Integration tests.

There will be an initial investment in build automation environment automatically executing these tests and setting up your first few tests will take some time. Set realistic goals, for example try one User Story first, the velocity of the Sprint in which you will start will drop, but not delivering any value will only fuel the non-believers. Don’t let it get discourage you, take baby steps and start enjoying the fun. What could developers love more than writing code? Writing code to prevent bugs so they can retire their debugging tools! Start to love the green bar when your micro tests all pass!

Mindshift – Stop the Line

Once we start to add more bugs in the system, tests will start to fail, which is a good thing, imagine what it would mean these bugs would only be uncovered in a later stage? Before bugs would be added to a bug management tool, and need clarification and review by a Quality Engineer and Product Owner, before we would eventually start fixing this bug. There need to be a mind-shift by the developers but failing tests will need to be fixed asap, right when they are found! Stop the Line is a lean practice which comes from the Toyota Production System, where they would stop the manufacturing line when a defect was discovered, to avoid producing defective products (waste). In Software Development we need to stop producing more code, and fix the failing tests first, or we will be adding more code on a defective code base as well, and increase the time required to fix the bug later (waste). When teams use Continuous Integration tools, but haven’t had a successful build in days, it defeats the purpose of validating each small change, since we will be piling up days of unvalidated code changes.


When you just start out with the automated test practices, the amount of bug you will prevent is limited, since only a limited part of the code will be “covered” by micro tests and automated integration tests. So there will be bugs in the system which are going to be found later stages, which will need to be fixed. The responsible thing to do then is to first add a test which will would have caught this bug, and then fix the code, and your test should now pass. This is an efficient way to identify new tests to be added to the system, since the bug shows that your existing tests did not catch it. If you keep following your new Definition of Done, the coverage will keep increasing, the automated builds you are producing are of increased quality and you will find yourself spending less time debugging and debating bug report details.


Since developers, quality engineers and product owners will spend less time on bug fixing and bug management, more time will be available to add real value for your end users. Developers will write more code, quality engineers will have more time to focus on test scenarios and exploratory testing and product owners will have more time to think about new features. Not to mention happier customers and faster releases to outperform the competition.

There are really no excuses to postpone test automation practices, start now!

Also available in : Nederlands

2 Thoughts on “Cost of Manual Testing on Agile Projects

  1. Very true. My company highly value automation test but still struggle to get it done in massive scale due to above reasons.
    I will try your suggestions to start it again. Thanks a lot.

  2. nverdo on October 25, 2012 at 8:05 pm said:

    Hi Nga, glad I was able to inspire you, go for it!

Leave a Reply

Post Navigation