Every so often, I end up having to explain the benefits of automation in terms of either tradeoffs or statistics, to people who still question the value of automated testing and the investment to be made on them in a project. This is an attempt to articulate the same for such audience.
While most Developers and QAs on a team might be convinced that automation is the backbone of continuous delivery, it might not be the same case with other purely functional roles in the project such as a Business Analyst or a Product Owner when they look at stories and their cost and time of development individually.
Consider a BA or a Product Owner, they are mostly interested in the functional code written to ensure that the system is capable of executing a few user scenarios as defined in the user story.
Once the story is delivered it is often the Developers and the QAs on the team who have to worry about ensuring that the user story delivered previously is still working, while also delivering new user stories.
For that purpose Developers and QAs put in a safety net by either driving or at the very least covering the functional code with unit and acceptance tests. The unit tests end up testing a particular unit of code (eg: a method in a class )and the acceptance tests start testing the end to end behaviour of the story which is the story acceptance criteria.
Even though this makes the developers and QAs happy, it is not of as much interest to the people who look at just the pure functional side of the story. From a BA or Product Owner's perspective the time to actually complete and release a story now has just gone up because we are writing more tests before calling a story complete.
What they do not realize however is that they are looking at one particular story in isolation. The bigger picture however shows that without automated tests
Cost of a release exponentially increases with the number of user stories being delivered. This is mainly because the project will require more testers to ensure that stories are manually well tested. Number of defects will also increase without the lack of a safety net of tests.
Time to release again increases exponentially like cost. At some point there will be a cutoff when the project cannot take in more testers, and it will take more time for the fixed number of testers to release more stories into production
The overall project cost will hence shoot up with the frequency of releases you do to production because of lack of automation
Ultimately the lack of automation manifests itself in the overall project cost rendering the project team incapable of making frequent releases to production cheaply.
On the other hand if you have Definition of DONE for a user story to include unit and acceptance tests and think of the time you take doing that, as an investment, you will be building a safety net as you go around each story. The safety net will give developers feedback on whether they are breaking any existing functionality with any new code they are writing and it will leave QAs to test more edge case scenarios in the system rather than repeatedly testing the happy path scenarios for all stories.
It is also equally important to invest time in a Continuous Integration system which can run the complete suite of tests after every checkin.
The continuous integration system can now churn out production grade builds by running the complete suite of tests after every checkin.
Once this is done, you will see that the cost of actually doing more releases into production does not shoot up and is well within controllable limits as below
Reaching a stage of roughly 75-90% automated test coverage is not easy. It requires being diligent about the quality of tests being written ( no side effects, clean tear down etc...) and also about refactoring the tests and treating them as you would treat your production functional code.
This lays a strong platform for continuously delivering builds into production with the only additional cost being deployment and non functional testing (such as performance, security etc...). Even these additional costs can be minimised by automating as much as possible.
Often the return on investment is questioned by people who look at the project within a narrow timeline and forget the long term returns from the same.
[My colleague Ranjan has provided a good statistical justification here of why test automation is the first step towards continuous delivery]