When Agile meets yearly IT budget

There is a well-known problem when Agile projects meet a client’s yearly IT budgets. The problem with the yearly budget is that

1. It is largely incorrect as it has been planned with minimal information

Most big organizations have a budget review only once a year where a CIO and his team have to decide what their future looks like for the year. This is the time when everyone looks into their magic crystal ball to estimate the amount of funding required for their projects in the financial year.

These numbers are largely incorrect purely due to the scale at which they are done and with the limited amount of information available upfront.

2. Fosters project thinking and deliver under budget than building the right product.

Once a ballpark yearly budget figure is reached after minimal planning by project teams, the teams are encouraged to plan in more detail to secure their exact funding required.

Project teams then start building their detailed product backlog of stories with estimates and break it down into releases through the major part of the year to come up with their funding numbers which are then secured.

This not only results in a lot of upfront planning even in Agile, but also focuses the team on delivering that huge backlog of stories under the project budget, than focusing on building the right product fit for the market. Needless to say that these big backlog of stories often get outdated within the first few iterations and so does the estimates and hence the budget plan.

So is there an alternative?

The good news is that a lot of companies are beginning to recognize this problem. And these are companies who view their products as “Strategic IT” investments and not an operational overhead. The solution requires changes in 2 key areas, budgeting and project planning and tracking.

Rolling wave or Quarterly budgeting

A year is clearly too long to forecast and budget for upfront. One option is to budget for the first couple of iterations and then budget the third and fourth just before the completion of the second iteration based on real data. This ensures that at any point in time you have budget for the next couple of iterations.

If a couple of iterations are too less for long-term visibility, the other option is to do it quarterly.

The advantages of this approach are

1. More informed hence accurate budgeting based on actual data on the ground.
2. Encourages people who own the high level budget to engage with the team on the ground at least once every quarter.

Track projects to measurable goals and not to a product backlog.

Not having to forecast a yearlong budget is a boon to an Agile team. The team can then focus on building just enough of the backlog for their product to be able to validate it with end users and then pivot or preserve as necessary.

This is a mindset shift to the traditional Agile/Scrum technique of doing discovery workshops and then building a backlog of user story phrases that are then elaborated every sprint.

Instead, the team needs to come up with clear and measurable goals on how they measure the success of the product. This could be things like increasing user uptake of the product by releasing some new features, improved customer service, launching a new brand / service etc.… As some of these goals might be long term, it is important to break them further down to a level that can be measured every month or a quarter. An example would be to validate an idea for a web page with 5 users before building 3 similar web pages in the subsequent iterations and validating with 50 users.

The owners of the budget in this case can decide based on the progress towards goals of the first few iterations to fund / invest in the next few iterations. This also gives them on option to pivot or preserve based on what they see built so far.

To summarise, the advantages of this approach are

1. Less of time spend in estimation of effort and more time thinking how to build something valuable
2. Keeps stakeholders who make budgetary decisions more engaged in the product.
3. Makes product owner more responsible to think and show results building the right product than forcing the team do deliver a backlog

References

The Budgeting Black Hole - Johanna Rothman

Annual Budgeting and Agile IT - Ross Petit

Lean Startup – Eric Ries

Implementing Beyond Budgeting – Bjarte Bogsnes

The problem with Continuous Delivery & DevOps as an IT investment

In Thoughtworks we have a lot of experience in implementing Continuous Delivery , and we have done it project after project under some name or the other. In fact we did it long before Jez even wrote his book on the subject. (Jez talks about one of those projects here). I remember roles which we used to call Build Masters/Engineers, even Build Monkeys and teams such as Build and Environments team which practiced the same principle. These teams or roles used to be specialised and sat outside the group of people who delivered functional user stories.

Today we call the people DevOps and the practice Continuous Delivery, but largely we have the same problem as 10 years ago. DevOps is seen as a separate team and Continuous Delivery is seen as an IT initiative. In fact organisations have gone a step further in this direction to appoint IT heads for continuous delivery who are in charge of such teams.

The reason Continuous Delivery becomes an IT initiative

1. Business sees it as an IT investment

In most big organisations business teams are still keep a safe distance from the gory IT details and more so from IT Operations and Support teams. (Business teams have more to worry about in their lines of business). In their view Continuous Delivery is an IT Operations issue solely as it will minimize the operation spend for IT. They see the key driver being within IT and the impact only on the IT operational cost and hence believe that Continuous Delivery should largely be an IT investment and if an organisation wants to do it, the IT budget should cover the cost and not the business.

2. The practice is seen as a mighty legacy systems/operations overhaul

In most large (> 1000 people ) organisations such as banks and insurance firms there are a whole lot of legacy infrastructure lying around. These are systems on which no one has ever attempted writing a single automated test. Deployment is a humongous tight coupled multi-step process which no one has ever imagined being done with a batch job.

In such places an IT organisation has to start somewhere and that cannot happen without a focused initiative. This results in a new department whose sole job is to focus on bringing in the continuous delivery practices into the organisation.

Problem with a separate DevOps team with an IT budget

The problem with a DevOps team funded by an IT budget is that it results in another glorified Operations/Support team, only this time balanced with application developers and sys admins. Being centralised to project teams, this team ends up doing any work classified as "Infrastructure" such as provisioning and configuring virtual machines, optimising and maintaining the build environments, production server monitoring etc...

Product owners, on the other hand, push their teams to deliver more functional stories and encourage them to delegate any "technical/infrastructure" work to the DevOps team. This results in product owners distancing themselves from the benefits of DevOps which is contrary to the whole principle of Continuous Delivery, which is to demonstrate rapid responsiveness to change and hence create a strategic impact for the business.

Moreover some DevOps teams work with a goal to make themselves redundant and merge themselves with functional product lines. This goal often remains an aspirational one with their ever growing backlog of work.On the contrary DevOps teams end up creating their own silo with specialised skills which developers on project teams struggle to keep up with.

There are enough reasons for business to cheer about Continuous Delivery in the longer term. An investment from a business unit will only result in business teams working closely with IT, which is at the heart of Agile and Continuous Delivery. Business teams will be able to find new ways of creating a bigger strategic impact and have an edge over their competition. Hence for the right alignment within the organisation, funding for Continuous Delivery should come from a Business budget.

Why you should care about experience design and continuous delivery as a project manager ?

I had written earlier about how a good project manager in a mature Agile team needs to go beyond managing scope, time and budget to be really effective in his/her role. The key is to understand the product and the tools and techniques used by the team to build it.

But there is more a project manager should do, which is, to provide the right direction to the team. The focus shifts from "building the thing right" (i.e delivering within scope , time and budget) to "building the right thing" (a product that fits the purpose the business and its users).

Building a delivery engine that can react quickly to market feedback and adapt needs focus in a few different areas than traditional project management. And this is where project managers should have a good understanding of experience design and continuous delivery.

From a continuous delivery perspective a PM should have a good understanding of

* The level of automated testing required to certify almost production quality builds in no time.

* The value of using a production like environment for testing and feedback.

* How automated deployments can help speed up time required to deploy to multiple environments and rollback quickly if the need arises.

* The need for real time monitoring of production systems for the team to quickly respond and fix production issues.

* How one can benefit from feature toggles v/s branching.

* The advantage of releasing quickly and minimising work in progress to shorten the feedback loop.

The main advantage of continuous delivery is not so much in reduction of deployment and maintenance costs. Instead it is in the capability it provides to validate and adapt the product being buit quickly based on end user testing and research.

It is important , as a PM to know

* If we are doing enough user testing to validate what we are building at regular intervals ?

* Are we doing enough user research to provide critical user context data in the form of personas etc...?

* Do we have good analytics capability built in the product to profile usage patterns which can help us make informed decisions about the product?

An insight into some of these aspects can really help a project manager steer the team towards building the right product which is way more valuable than someone who ensures the team is tracking to an outdated plan.