Agile planning & estimation mindset (as a manifesto)

In the time I have been at Thoughtworks, almost every year there is a debate about how one should estimate and plan, in the delivery engagements that we work on.

Given the variety of engagements that we do, there is no one size fits all approach.  But there is probably a mindset that is useful when approaching these engagements. And I made an attempt to make my mindset explicit (inspired from the Agile manifesto)

I value

Learnings from the estimation conversation over Estimation numbers themselves

Shorter plans with lightweight estimates over Longer plans with precise estimates

Using past trend/flow data  over Estimating every time 

Optimising for delivery of business value over Optimising for time and budget

While adhering to one of the core principles of

Learning by releasing working software frequently, from daily to a couple of weeks, with a preference for a shorter time scale.

Agile won't scale with average talent and a prescriptive framework.

A common question amongst many large enterprise clients is how do we "Scale Agile".  And most often than not these enterprises don't have the time and patience to work through the values of using agile methods and inculcating that within their organisation.

What they are really asking for is a shortcut way of spreading some agile practices within an average talent pool in the company with the hope that they might learn to deliver software better. And this is the very reason process frameworks like SAFe and others seem more attractive to senior execs.

Ironically though what might seem as a shortcut of adopting a framework in first place ends up burning more of the IT budget with minimal results within 3-5 years.

With an average talent pool, the problem of scaling agility soon becomes a problem of enforcing agility (pardon the oxymoron). Being prescriptive at a large scale requires a ton of micro processes and standards to be setup to manage the average worker.

SAFe talks about normalizing estimates across all teams at the very beginning by forcing teams to adopt 1 point as 1 day. This defeats a lot of benefits of point based estimation and relative sizing. Again in SAFe, teams are forced to sign up for objectives at the end of every increment planning of 2 months or so. This obviously does not take into account the nature of the work the team is doing. A team that cannot release something in 3 months ends up with weird looking objectives just to satisfy the organization standards. 

Such standards are created to put some checks around the average talent pool who might be prone to wrong doings. A company even went to the length of running reports on story checkin commits to ensure that stories do not span multiple iterations. And then when there were hangover stories, people were asked to split stories accounting for partial, untestable work in the previous iterations.

Another company I worked with, was trying SAFe, and was forcing product owners to come up with a quantification of business value and time criticality so that cost of delay could be calculated.  While core to product development, the essence of this whole calculation was diluted by the middle management of product owners who ended up making up numbers as a shortcut. Justification of the business value numbers ended up becoming more subjective and usually the loudest product owner in the room won. 

These experiences clearly show that there is really no alternative to investing in hiring and nurturing good talent. While for execs at the very top, a scaleable framework with checks and balances might look very attractive, the middle management along with folks on the ground will find ways and means to work around these checks and never end up learning the real values.

While hiring talent is hard, it is equally important to nurture good talent, especially in IT where the market is highly competitive.

Design your enterprise towards continuous delivery of value. A lean enterprise should be able to build, measure and learn from the software they delivered and iterate rapidly towards maximizing business value. This is not easy, especially if you are a large enterprise. Reaching this goal will require hard work and changes within the organization that are beyond IT such as financial budgeting, performance reviews etc... Invest in hiring smart talent and then invest in your people to coach them on fundamental principles such as early feedback and failing fast and how core XP engineering practices and CD help achieve that.

Just scaling agile with a prescriptive process framework will be attractive in the short term but will not live the test of time for your enterprise. Don't give in to the temptation. Do the hard work.

How to measure productivity in Agile teams ?

No, the answer is not story points. Mike Cohn describes the answer to be cycle time.

A lot of people have written similar articles but I always end up finding people in the senior leadership who want to measure productivity using  story points. And there are those who want to compare one team's story points against another. And there are those who try to add story points across all teams without considering them to be different at all.

Reasons why people are so tempted to use story points are because

• They are so readily available to measure, that people don't feel like looking anywhere else. A
• A lot of Agile coaches end up not understanding cycle time well, 
• A lot of project management tools that claim to be "agile" do not support cycle time calculation at every level well enough.

I have seen "measuring productivity" conversations start when programs are behind schedule , senior leadership does not trust the teams or vendors , i.e. in general when the going gets tough. This is exactly the tipping point when a wrong measurement /KPI  can push the program into a completely wrong direction.

The scenario when "story points" are chosen as a performance indicator

• Will immediately lead to teams inflating their story estimates to look good on reports. 
• This will lead to a false perception of progress overall. 
• Teams could get completely demotivated as the focus moves to story points than actual working software as a measure of progress. 
• There could also be scenarios where quality is compromised for speed and a lot of defects creep in.

The scenario when "cycle time" is chosen as a performance indicator

• Will lead teams to think about how to measure it in first place. 
• It will lead teams to break the end-to-end cycle time into time taken from moving a story from Analysis to Development, from Development to QA, and QA to Done etc...
• Teams will end up thinking about definition of done / acceptance policy for each stage. And if a story moves from Analysis to Development violating the definition of done policy, the story is sent back to Analysis, providing immediate feedback to the previous stage owners to produce better quality work. 
• Quality then gets built implicitly into the process because of faster feedback and being able to fail fast, which are the fundamentals of an Agile team.

So choose your measures wisely :-) 

Release trains are a good first step, not the last one.

In software development, the release train concept tries to bring about some cadence in releasing software. Once it is agreed that the software will be released "every X weeks", product development teams focus on ensuring that a release happens at every X week interval. For most teams it could be anywhere between 2 weeks to 6 months. However the main idea being that while the interval and schedule is fixed, the scope or content of the release is variable.

This concept is a boon to enterprises who struggle to release software for years, especially those who are running teams that are in a dependency hell. Aligning every team towards a shorter release cycle with negotiable scope and actually releasing more often is a great first step.

The problem occurs when the concept starts breaking down into a bureaucratic process of release management instead of focusing on the core engineering problems behind releasing frequently.

Here is what happens in large enterprises

• Organisations start creating new release committees (with roles like release managers, deployment coordinators etc...) to manage releases every X weeks across teams that deliver a common software platform.  Since the release committee works across teams without enough bandwidth to understand details, delivery teams are requested to create a whole bunch of documentation to aid the committee's decision making, mainly around a Go/No Go decision for every release.
• Conservative strategies such as early branching, code freeze, etc... are applied to meet the release timeline of the train. QA teams start demanding more time for testing as they are constantly worried about every release.
• Teams start planning more conservatively during their release planning exercises. No one wants to pack in more scope and be caught as a dependent team holding up a release. 

The core issue in larger organisations is that even with release trains, it ends up being a process problem that needs to be managed rather than an engineering problem. As an example, even with release trains, a lot of teams incur a bigger overhead in managing dependencies instead of investing the same time in decoupling systems to release independently.

Organisations embracing release trains as a first step lose their way towards continuous delivery. They get caught in the false sense of satisfaction that a release train gives them and hate to see the amount of overhead still incurred with every release. 

The real goal should be to engineer software in such a way that you can reliably release it on demand, whether it is daily or even hourly. So, please do take the next steps.

Painful truth about "Agile QA" in the enterprise world.

When Agile brought the idea of cross functional teams with Devs and QAs working closely together every day, it created enough disruption and a feeling of insecurity amongst the traditional QA organisation.

However most of the big enterprises have embraced the idea and called themselves "Agile",  by paradropping QAs from their organisation into Scrum teams. These QAs still report to the old QA organisation but are also told to work on priorities set by their Scrum Master. These folks then declared victory in Agile adoption even though they were struggling with  collaboration between Devs and QAs in the Scrum teams on the ground. And more often than not, issues with collaboration are brushed off as tactical issues by QA management.

The reasons for lack of collaboration between developers and quality analysts are fairly deep rooted in our IT industry.

• The relevance of so called "manual QA" is slowly diminishing with the advent of Continuous Delivery. There is a huge community of technical folks who are working towards reducing the complexity of testing and releasing on demand with minimal effort. Gone are the days of large manual regression QA teams running test cases documented from a Wiki.  If an IT organisation is still doing this, the IT business might cease to exist in the next 10 years. So in that sense, the CD embracing developers become a threat to the traditional manual QA world.

• In most companies QAs are not hired for their keen eye and rigour in looking at where systems could break. Instead, a lot of times, QAs are created out of developers who struggled to clear a developer interview. In a lot of places, such candidates are made to do a QA role for a while, and are promised that if they do well, they might move into a development role soon. This creates a psychological divide. QAs try to pick up automated testing so that they can hone their development skills along with testing. In this context, whenever developers try to help QAs in automation, they are viewed as a threat. Innovation in automated testing is needed at all levels in the Test Pyramid, and the current culture hinders progress in that direction.

As organisations achieve more maturity in Continuous Delivery, there will be a lot more overlap between Developers and QAs. This will also include roles that perform non-functional testing such as Performance, Security, Load etc... In that context, the purpose of a QA organisation needs to change towards cultivating people who are learning to solve testing, integration and deployment problems so that releases become uneventful. IT folks who invest in bridging the gap between Dev <=> QA <=> Ops actively will be the ones that succeed in providing true agility to their business.

A false sense of agility at scale

"Agile does not scale" is one of the common challenges in the Agile community and one of the answers recently has been the Scaled Agile Framework (SAFe). While SAFe has been packaged and marketed really well, every enterprise going through Agile transformation is using it or devising a home grown way of scaling Agile development processes to the wider organisation.

Most of these frameworks break down very quickly. The reason being, the more one tries to scale, the easier solution become standardisation which is counterintuitive to agility in first place. Even the SAFe image has an uncanny resemblance to top-down planning like below.

Similar top-down planning approaches have been dominating the software industry for ages and have never delivered the agility demanded by today's dynamic business needs. One simple reason is the broken reverse feedback loop between portfolio planners and the actual teams doing the work. This failure is very evident in an organisation within a few months when people talking about the portfolio have no clue about the current engineering practices on the ground and are very soon making numbers of their own. PMOs structured like this live in a fantasy land thinking that they are adding value.

While in the midst of such top-down planning the larger problem often gets forgotten.

If your business needs the level of agility that can push newer products to the market, it will not happen if there is no adaptive budgeting in the organisation. If frameworks like SAFe work within a yearly budgeting model, they are bound to fail. And this is what happens in every large enterprise that has a lot of resistance in moving away from their traditional annual budgeting models. The top down approach puts the budget and the product even further away from each other only to make it extremely difficult to determine the value from the investment made.

The solution to all this lies in keeping investments and products as close to each other, and as close to teams that use the investment to deliver value.

There is no easy path to scaling. And one size does not fit all. Here are some things to try.

• Move away from annual budgeting and look at a rolling wave budgeting model based on return on investment.
• Focus on realising value quickly by reducing time to market and feeding that back into your budgetary processes
• Don't fall into the trap of standardisation when scaling agile development.
• Push more autonomy to the teams and keep teams closer to budget and product so that you have a better sense of return on investment. It also provides the team a larger sense of purpose.

An agile enterprise...really ?

Every IT enterprise claims to be agile or wants to transform into one pretty soon. There are roles, processes, consultants and certifications, that specialise in Agile these days. But deep within these big enterprises who want to climb the Agile bandwagon, the meaning of being agile is lost.

If you are part of an IT enterprise ask yourself a few things by observing the people around you who are responsible for building the software to power your business.

Are people around you worried more about what is an Epic versus a Feature or a Story ? Are people discussing about what should happen in an Iteration/Sprint/Release planning meeting endlessly ? Is there disagreement on how things should be stored in that new Agile project management tool that your company just bought ?

 Think Individuals and Interactions over Process and Tools. 

What is more important here is not what an activity or an artefact is labelled as, but how we as software craftsmen create easy mechanisms to interact and collaborate and get the job done. Hiring a smart developer is way more important than coming up with a smart label or a process. That developer you hired will figure out a way automatically, trust me.

I would like to stress even more on the individual. How much is the developer you just hired empowered ? Can he/she pickup a story to be developed , question its business value when in doubt and challenge the product owner into not getting it developed ? Can he/she even visualise this as a remote possibility in between all the "Agile process" loops one has to cross ?

Look at people around you once again. Are they saying a story should not change within a sprint/iteration etc..? Are your sprints and releases locked in for even a few months at a high level ?

Think Responding to change over Following a plan

How are your people embracing or even responding to change ? Isn't it just another way to follow a plan ? What does it take for a developer on the ground to say a particular story will take 2 more days than what he/she thought initially ? What is the behaviour of people around when such a thing happens ? How easy is it to adapt to this reality in your organisation ?

So.. is your enterprise really agile after all ?

Stop locking down the Agile project management tool

I have seen this common behaviour at many of the bigger client organisations I have worked with, who are transitioning to organisation wide agile adoption. The company will invest a lot in researching and eventually buying an agile management tool of choice (Greenhopper, Mingle etc...) and then lock it down so that project teams will have minimal access to change how they want to use the tool.

The sheer irony is that these tools are actually written to be as adaptive as possible to the development process followed in the team. These tools, and especially Mingle for one, is written keeping in mind teams that are following agile will evolve their processes as they move forward.

Story statuses and types could evolve a lot. For a team that has just started, they might just have to deal with stories and defects. But once the team starts making releases, they might also have to deal with enhancements which probably has its own workflow. Changes have to be done to the project tracking tool to accommodate this change in processes. These changes are not one off. A team might discover new types of work and also new ways of tracking and progressing through the work on a frequent basis.

The driver for locking down the project management tool is to force teams to use the same set of statuses and processes so that reporting can be rolled up at a program or an organisation level. A lot of times, senior program management tries to come up with an "ideal agile workflow/template" that should work for all teams and then use the tool to impose it on all teams within the program. The result of this is immense frustration within teams while they try to find workarounds to the system that is being pushed on to them. Teams in this state start moving away from the tool that was chosen and end up tracking themselves through more lo fi means (physical card walls etc..). This reflects poorly on the investment made in buying this agile project management tool in the first place.

Here are a few suggestions

* Roll up reports at a fairly higher level of abstraction when reporting for a program. This gives teams some more flexibility in coming up with actual numbers for the KPIs. For e.g. instead of tracking "Velocity in points", track "Velocity". This gives flexibility for one team to use story points and another  which can use days, and another which can count stories :-).

* Don't rollup reports and don't lock down your workflows when the teams are still forming and storming. Let teams reach a stage where their development processes have matured a bit. Until then let each team present a different status report for themselves. Start rolling up once the teams have reached a performing state together.

Remember, every agile team has to adapt to different set of challenges and they can potentially come up with different creative processes to deal with them. For these teams to succeed the supporting systems and tools need to embrace change rather than impose order.

A Distributed Team working out of a Common Backlog

Even though working out of a common backlog might seem the most obvious thing to do, a lot of teams I worked on previously did not do it, or do it that effectively.

Having a common product backlog of stories enforces at the very fundamental level that it is one team spread across multiple locations than two teams.

Benefits

Having a common deck of stories for both locations has the following benefits

Strong levels of trust and collective ownership across people at both locations

Since people across both locations are looking at the code written by each other regularly, it ensures that there is strong collective code ownership. Consistency in coding styles, metaphors, development processes, etc... all fall in place in the long term.

Having a common standup, iteration planning meeting, developer huddles, showcases, retrospectives etc... helps to synch up the team across both locations.

If teams are working on different backlogs, there are always late integration issues along with trust issues when things do not fall in place.

Knowledge distribution across distributed locations

Having stories for a particular feature distributed across both locations ensures that not only people across locations touch/refactor each others code, but it also ensures awareness of features and areas in the business domain across the team.

Allows for team scaling across both locations

Having a cross functional team across both locations who work on the same product backlog provides flexibility to scale the team across regions depending on availability of skill. This model is also very useful in a client-supplier model where the client wants their developers to work with the supplier as well. A good mix of people across both locations ensures that the team can easily scale when needed.

Concerns

However the common backlog model comes with its share of concerns

Very high level of day to day communication across all roles

The basic premise of working with a common backlog is to ensure both locations of the team are always on the same page during their working day. This means a lot of communication via multiple channels such as video conference calls, emails, voice chats etc.

One key factor is the overlap time across both locations. If the overlap time is anything close to half a day then it is advisable that the team at a location has a continuos video stream of the remote team. With the right hardware investment, this can create a very powerful "one team atmosphere" for the overlap time.

If the overlap time is much less (1-2 hours) then a lot of communication has to flow through offline emails and chats. It becomes critical for the team on both sides to communicate the critical pieces of proceedings at a particular location so that the other location is on the same page.

Business and IT stakeholders not being distributed 

Even though the delivery team might be nicely distributed, there is always the common case of the business stakeholders sitting in only one location. This could also apply to the network infrastructure, IT security etc... Because of this, the delivery team in the same location as these stakeholders will end up having a lot more context (e.g. business domain context) and a lot more meetings to deal with than the remote/satellite team which is fairly isolated.

This could cause some uneven knowledge distribution over the long term and should be fixed with adequate rotations across roles within the delivery team.

Conclusion

It will be painful initially to communicate using a common backlog, but dealing with the pain more often (daily) puts the team in a much better place long term than each team going their own way.

Measuring progress

Measuring progress in software development has always been up for debate. There was the waterfall era of Gantt Charts created using MS Project which used to breakdown effort into smaller work items with dependencies. These were easily outdated the very next day they were created.

The Agile community brought in the idea of user stories and doing them in small iterations. We then measured progress using a burn-up or burn-down chart which burnt either story points or ideal days. Soon, we started feeling the pain of the inventory we carried between releases which happened in a span of 3-6 months. This prompted us to measure and limit our work in progress and measure lead and cycle times across the delivery workflow.

Releases started becoming more frequent with technology advances in the cloud, virtualisation and deployment automation. The continuos delivery era came through with examples of people deploying almost everyday to production. Startups really needed this to validate their ideas in the market and course correct themselves based on market feedback. A/B testing became common place and measuring validated learning and cost of delay in deploying a feature made more sense for these new businesses.

Anyone can easily get confused trying to decide how one should measure progress or at least start with, in their project.  There are 2 key variables in making this decision, the kind of relationship IT has with the Business, and the type of product ultimately being built.

So here is the quadrant I am talking about.

Outsourced / Utility (bottom left)

To the extreme left on the relationship axis is outsourcing your IT. Trust levels are fairly low. It could be a new IT vendor who has not delivered for the business. Work is probably happening outside the office or even the country where the business exists.

The extreme bottom of the product axis sit utility/internal applications which are mainly for keeping the business lights on. (Time Sheets, Payroll, Company intranet etc...). There is no market advantage to be gained by these. They are well defined internal business problems.

Start with measuring progress in these cases using effort estimates such as Ideal Days, Story Points, T-Shirt sizes etc... A burn up chart should be a good indicator of where the project stands. Try to work on building enough trust which enables you to move to the right axis in terms of relationship.

Partnership / Utility (bottom right)

Moving right on the relationship axis is only possible by building trust. And that is possible by delivering frequently to the business and also maintaining the people relationships at the right levels. A vendor who is treated as a mere supplier can become a trusted partner for well defined applications and their maintenance as they prove themselves to have the required expertise.

Move progress measurements from effort based units to delivery efficiency based units. Optimise the workflow till production by minimising inventory you are carrying and reducing lead times. A Cumulative Flow Diagram  (CFD) can provide the same view.

Partnership / Strategic (top right)

This is really the sweet spot. This is where the business has engaged IT to build a strategic product which is probably completely new to the market. It could also be a product that is struggling and badly in the need for new ideas. The IT here is a closely trusted partner which the business can rely on.

Given the volatile nature of what needs to be built, the need for agility is the highest here. Being able to deploy a new feature and rolling back as quickly as it was deployed, if hypothesis fails, is equally important.

Measure progress here using business outcome based units such as validated learnings and measuring cost of delay of a feature and the impact it has on the business. 

Partnership / Outsourced (top left)

This hardly ever happens where a strategic business need for technology is outsourced to a less trusted supplier !

Build enough trust before you can stop estimating

A lot of people in the Agile community these days are talking about estimation being wasteful and suggesting not estimating projects at all.

One radical view amongst all of this is to tell clients that as a vendor of a project

"I will try and deliver the best possible outcome based on resources I have at my disposal. I will give you frequent production quality software almost every other day for you to track how I am progressing. I will charge you per day or even per hour. Don't sign a huge contract with me but sign me up for a shorter span , say a month, review my software regularly to see what I have built and renew the funding for another month if you are happy. If you are not happy you can please kick me out. "

That might sound like the best thing to do or probably the most absurd depending on how your relationship is with your client. What people fail to articulate while suggesting this is that the entire premise is built on trust. And a whole lot of TRUST ! You tell a new client your are not giving an estimate and suggest the above dialogue you are bound to be frowned upon.

This view of stopping estimation is built upon the belief that "if we release software frequently, we do not need to worry about estimates". And this belief can be incorrect many a times.

For instance a new client who has never worked with you would want to compare bids from other similar vendors before choosing one. The issue becomes bigger in an outsourcing scenario where the main reason for choosing a vendor is cost. High level estimates on cost and time would definitely help the client make that assessment.

While releasing software frequently builds trust with the product owner(s), it might not build the same levels of trust with say the CXO group within the organisation. Maybe the CIO who is funding your project does not have enough time to attend your showcases or UAT sessions.

Asking a client to fund one month and fund the next month based on the product delivered might again sound good on paper but the reality is for a lot of big clients this means a lot of administrative overhead to manage. Choosing a vendor is a hard ask in itself and now being ready to switch vendors almost every month is compounding the problem.

Taking a hard stance by saying no to estimation will be highly immature in the above scenarios.But this does not mean all this is entirely impossible.  It is possible, but only after a period of time of working with the client. When a new project starts you have to bite the bullet and estimate. The important thing is to start moving towards a place where estimates become more and more irrelevant as you do more projects or more releases with the same client.

And this is where delivering value continuously will build trust with the client. Fostering people relationships at all levels with the client on top solid delivery will slowly nudge the client to start thinking about the product they are building rather than managing the vendor. This will then allow you as a vendor to focus way less on estimation and much more on building a great product.

Most "products" in the enterprise software world start as "projects" unfortunately. Projects are funded based on budget and the estimates are required. However changing this scenario is in the hands of the project team. It is the responsibility of the project team to slowly help the client focus on product building than project tracking. And that is only possible when there is enough trust built between a client and vendor, not always from day one !

Books that helped me build better software delivery teams

I have learnt a lot by making mistakes on the job but there have been books that have influenced my thinking to a great extent. I will try to list the ones that were the most influential in changing my perspective towards managing projects and teams.

Classic General Management

• The Mythical Man Month - Essays on Software Engineering
• The Fifth Discipline - The Art and Practice of a Learning Organisation
• Peopleware - Productive Projects and Teams
• Waltzing with the Bears - Managing Risk on Software Projects
• What did you Say - Art of Giving and Receiving Feedback

Theory of Constraints

• The Goal
• Agile Management - Applying Theory of Constraints

Agile

• Extreme programming explained - Embrace change
• User Stories Applied
• Agile Retrospectives

Useful Link : Top 100 Agile books of 2011

Lean and Kanban

• Implementing Lean Software Development - From concept to cash
• Kanban
• Principles of Product Development Flow

Technical

• The Pragmatic Programmer - From Journeyman to Master
• Working Effectively with Legacy Code
• Refactoring
• Continuous Delivery

The Lean Startup

• The Lean Startup
• Business Model Canvas

There are many more great books out there but I would highly recommend the minimum reading list above for building teams to deliver great software.

Prefer partnering with craftsmen over contractors

If you are in an IT department in one of the bigger organisations working in the major verticals of banking, telecommunication, insurance, retail etc... you are probably looking at a portfolio of work which has major bumps in the curve when starting a big software initiative and a steady state when those initiatives move to a steady state of enhancements and bug fixes (a.k.a Business as Usual (BAU) ).

If you are someone responsible for delivering this portfolio of work you probably are tied under the typical IT budget constraints of a big organisation. Hiring highly skilled people quickly to deliver the bursts of projects that come along is going to be difficult. Given this scenario it is more than likely that your attention goes to the right end of the curve where you see the amount of work in the portfolio reducing within a number of years. This is when you think "Why do I need to hire more permanent staff, I can get things done by hiring contractors who can then leave when most of the work is done."

This might probably work if the contract staff form a very small percentage of your overall organisation. You might even be able to find some real good contractors who specialise in certain niche areas which might be temporarily beneficial to your company.

But the moment you start relying on contractors to do most of the work in the pipeline, it is the beginning of a big world of problems. Because here is what will most likely happen.

• You decide on hiring contractors to get some major pieces of work done.
• Since these contractors are temporary, you ensure that you have enough permanent staff to oversee the work the contractors are doing. You have most of the permanent staff in all the important decision making roles.
• The contractors come in and start working alongside your permanent staff. The good ones want to drive new initiatives within these projects but soon get frustrated by the red tape approval process created by your permanent staff.
• Your permanent staff start feeling insecure about their jobs once they see some contractors who are better at the job than themselves.
• You might try to hire these brilliant contractors which creates an ugly set of dynamics at work.
• The contract staff becomes frustrated and start working just the hours required for their contract. If you hired really cheap contractors then they are probably in this mode from day one.
• The permanent staff has issues with the quality of work done by these contractors and sometimes vice versa.
• You add rigorous shortlisting processes for contractors in a bid to weed some bad ones out. But you still end up with some bad contractors because of budget constraints.
• Your project ultimately might go live but chances are that the quality of the architecture, codebase, user experience , performance is worse than ever. Your business stakeholders are unhappy.
• The contractors leave and you inherit a horrible quality system.
• Your permanent staff feels terrible maintaing this poor quality system. They either suggest a complete rewrite of the system again or suggest outsourcing the maintenance of the system to contractors.
• Another project comes along and you are probably still choosing contracting firms in the never ending grind of work.

So is there a better solution ? Let us look at the assumptions again

• Most systems you are building are of strategic importance to your business.
• There is more work in the pipeline than the permanent staff can chew and to be delivered in an aggressive timeframe which does not allow you to hire more permanent staff.
• The inflow of work might reduce after most IT systems are up and running in a few years.

and of course the more important long term goals that

• You need to be build a high quality system or platform which scales well to serve the business needs.
• You need to build an IT department with skilled technical people who are able to deliver and maintain the systems required for the business

So it is in your best interest as someone tasked out for delivery to build good software and at the same time set out to build an organisation which can do the same repeatedly in the future.

This is best done by hiring good people in first place, but if for various reasons you cannot, then at least try to partner with people who are passionate about the craft of building software than people who worry about billing the 8 hours of work.

To scale, try to find companies with whom you can partner with for their delivery services knowing how culturally passionate the partner company is about delivering the work for you. (And believe me , there are many good consulting companies out there which can help you out).

And by partnering, you engage with them at levels where they have a decent voice in decisions made on how to deliver the project. There is transparency and a drive towards building a win win collaborative culture within the team. You treat them as strategic partners rather than augmented resources in your delivery organisation .

It takes time to find such companies to partner with. You need to validate their delivery capability and culture fit over a period of time. But once you have done that, try and build a lasting relationship with them because in reality you will always have a backlog of work given the dynamic nature of your business.

After all, you are building long term assets for your company and it can best be done by partnering with people who care about it the same way you do. So invest in finding a good software delivery partner that can work for you instead of risking long term failure by using low cost/quality contractors.

When Agile meets yearly IT budget

There is a well-known problem when Agile projects meet a client’s yearly IT budgets. The problem with the yearly budget is that

1. It is largely incorrect as it has been planned with minimal information

Most big organizations have a budget review only once a year where a CIO and his team have to decide what their future looks like for the year. This is the time when everyone looks into their magic crystal ball to estimate the amount of funding required for their projects in the financial year.

These numbers are largely incorrect purely due to the scale at which they are done and with the limited amount of information available upfront.

2. Fosters project thinking and deliver under budget than building the right product.

Once a ballpark yearly budget figure is reached after minimal planning by project teams, the teams are encouraged to plan in more detail to secure their exact funding required.

Project teams then start building their detailed product backlog of stories with estimates and break it down into releases through the major part of the year to come up with their funding numbers which are then secured.

This not only results in a lot of upfront planning even in Agile, but also focuses the team on delivering that huge backlog of stories under the project budget, than focusing on building the right product fit for the market. Needless to say that these big backlog of stories often get outdated within the first few iterations and so does the estimates and hence the budget plan.

So is there an alternative?

The good news is that a lot of companies are beginning to recognize this problem. And these are companies who view their products as “Strategic IT” investments and not an operational overhead. The solution requires changes in 2 key areas, budgeting and project planning and tracking.

Rolling wave or Quarterly budgeting

A year is clearly too long to forecast and budget for upfront. One option is to budget for the first couple of iterations and then budget the third and fourth just before the completion of the second iteration based on real data. This ensures that at any point in time you have budget for the next couple of iterations.

If a couple of iterations are too less for long-term visibility, the other option is to do it quarterly.

The advantages of this approach are

1. More informed hence accurate budgeting based on actual data on the ground.
2. Encourages people who own the high level budget to engage with the team on the ground at least once every quarter.

Track projects to measurable goals and not to a product backlog.

Not having to forecast a yearlong budget is a boon to an Agile team. The team can then focus on building just enough of the backlog for their product to be able to validate it with end users and then pivot or preserve as necessary.

This is a mindset shift to the traditional Agile/Scrum technique of doing discovery workshops and then building a backlog of user story phrases that are then elaborated every sprint.

Instead, the team needs to come up with clear and measurable goals on how they measure the success of the product. This could be things like increasing user uptake of the product by releasing some new features, improved customer service, launching a new brand / service etc.… As some of these goals might be long term, it is important to break them further down to a level that can be measured every month or a quarter. An example would be to validate an idea for a web page with 5 users before building 3 similar web pages in the subsequent iterations and validating with 50 users.

The owners of the budget in this case can decide based on the progress towards goals of the first few iterations to fund / invest in the next few iterations. This also gives them on option to pivot or preserve based on what they see built so far.

To summarise, the advantages of this approach are

1. Less of time spend in estimation of effort and more time thinking how to build something valuable
2. Keeps stakeholders who make budgetary decisions more engaged in the product.
3. Makes product owner more responsible to think and show results building the right product than forcing the team do deliver a backlog

References

The Budgeting Black Hole - Johanna Rothman

Annual Budgeting and Agile IT - Ross Petit

Lean Startup – Eric Ries

Implementing Beyond Budgeting – Bjarte Bogsnes