The Failed Record of the Software Industry–why Software People Don’t Get It

The Standish Group is an organization that has been studying and reporting on software projects for many years.  In 1995, it reported that only 16.2% of software projects succeeded.  In large companies, the number was only 8%.  That is, these projects were on time and on budget.  In 2012, there is some improvement.  The 2012 report discusses waterfall projects and agile projects.  A summary of the success stats demonstrates that agile projects do improve things across the board.  That’s not surprising given that the 1995 report cited lack of stakeholder involvement and incomplete requirements as the top two contributing factors.  Agile principles address both of those. 

The 2012 report is still fairly damning of the industry as a whole.  It cites 12% success rate for waterfall projects and 42% success rates for agile projects.  Firms like Headspring, with a 95% success rate, have embraced agile, of course, but there is still so much more to be desired by the track record of the industry.  In 2012, 9% of agile projects still out-right fail, and 49% are “challenged”. i.e. they unacceptably ran over budget and/or over schedule – either can have significant impacts on the business.

Car_crash_1The industry even pokes fun at itself in a famous Internet meme, “If cars were like software” where we see that we have much more reasonable expectations of our automobiles than we do of software and the people who make software. Somehow, the software vendors/programmers have set the expectations of paying customers (and executives) so low that they celebrate the smallest successes even in the face of massive budget overruns and late launches.

This is not acceptable!  Software people don’t get it.  Programmers have been writing mission-critical business systems for decades.  The newness is gone.  The trial period of “we’re figuring out how to do this” is well over.  Now is the time when paying customers must demand more from programmers. 

As an executive of a software engineering firm, I see the perspective of our clients.  They have a business that is pushing forward in their industries.   They have a really old information system that must be revamped in order to take advantage of changes in the industry.  Or they need some automation for a brand new market offering.  The success of the business initiative rides on many factors, software being one of them.  They cannot afford to gamble with a failed $1M (or more) project.  Besides the money, once the project calendar has come and gone, you can’t get those months back.  Whether you hire a firm, do it in-house, or hire contractors in India, a failed project by any cause could mean the end of the career by the manager in charge of the product or initiative.  And it doesn’t have to be this way.

Right now, across the industry, clients are forced to literally gamble with their capital investments and time.  With less than 50% of projects (even the new and improved agile variety) succeeding, clients many times have no recourse but to gamble.  The smooth-talking sales personnel do their bonding and rapport.  Then comes the confidence building.  Then the project plan and schedule – all works of fiction.  And don’t forget the budget, which magically covers the time period the project plan lays out.  Once the project gets started, the software engineers work for a while – hold meetings according to their agile process – track burn-downs, measure lots of agile numbers – then what comes out the other end is your 42% chance of succeeding. 

In order to increase the pitiful industry track record, software engineers need to change.  And when saying “software engineers”, I’m referring to any person responsible for delivering software. Some companies have a plethora of titles and segmentation of specialty, but it’s all software engineering (i.e. software engineering includes leadership).  The bed of education and training currently in place does not produce the results that clients demand.  I hope to elaborate more on this soon.