2007 IT Project Success Rates Survey Results

This survey was performed in early August 2007 and there was 586 respondents. The survey was announced on the Dr. Dobb's Journal (DDJ) mailing list. The survey results Downloads What you may do with the information Discussion of the results Links to other articles/surveys Why share this much information?

The Survey Results

The results of the survey is summarized in the article Defining Success published in the December 2007 print issue of Dr. Dobb's Journal. I also analyzed the data a bit deeper in my November 2007 Agile newsletter entitled Is Agile Really that Successful?

Some findings include:

1. Respondents reported that Agile software development projects have a 71.5% success rate, traditional projects a 62.8% success rate, and offshored software development projects a 42.7% success rate. These figures are summarized in Figure 1.
2. Respondents with only Agile experience, 15 out of 586, indicated a success rate of 84.3%, respondents with traditional-only experience (164 of 586) reported a success rate of 66.5%. Respondents with both Agile and traditional experience (336 of 586) indicated success rates of 70.9% for Agile and 61.1% for traditional.
3. Time: 61.3% of respondents believe that delivering when the system is ready to be shipped is more important than delivering on schedule.
4. Money: 79.6% of respondents believe that providing the best ROI is more important than delivering under budget.
5. Scope: 87.3% of respondents believe that meeting actual needs of stakeholders is more important than building the system to specification.
6. Quality: 87.3% of respondents believe that delivering high quality is more important than delivering on time and on budget.
7. Staff: 75.8% of respondents believe that having a healthy workplace is more important than delivering on time and on budget.
8. When asked to prioritize the five criteria listed above, quality is the most important factor followed by scope, time, staff, and money respectively.
9. 40.9% of respondents considered cancelling a troubled project to be a success.
10. 68.6% of respondents had been on a project which they knew was going to fail right from the very start

Figure 1. Project success rates.

Downloads

The Survey Questions (180 K)

Raw Data (191 K)

Summary Presentation (145 K)

What You May Do With This Information

You may use this data as you see fit, but may not sell it in whole or in part. You may publish summaries of the findings, but if you do so you must reference the survey accordingly (include the name and the URL to this page). Feel free to contact me with questions. Better yet, if you publish, please let me know so I can link to your work.

Discussion of the Results

1. Government success rates were little different from commercial success rates (i.e. for Agile projects, 68.8% for govt. vs. 71.9% for commercial)
2. This survey measures success as defined by the respondent, it does not force a definition of success on them. The success rate was calculated by summarizing the weighted average of each range (i.e. 90-100% averages to 95%) times the number of respondents.
3. These figures vary significantly from those of the Standish Group's Chaos Report which reports a 34% success rate and a 51% "challenged" rate. They define success as "on time, on budget, meeting the spec", but that definition doesn't seem to hold when we ask people what they actually value. I'm not convinced that it's appropriate to force a definition of success on people, regardless of how easy it would the processing of the resulting data.
4. It is difficult to compare the numbers from the Chaos Report and this survey. This survey is open, you have complete access to the original questions and data, yet the Chaos Report is closed to outsiders. I invite the Standish Group to open source their material.
5. Agile projects may be more successful than traditional projects because the agile teams are comprised of more highly disciplined developers and because the agile teams are smaller (often due to better organization to begin with, and sometimes because organizations aren't applying agile at scale yet).
6. There may be some bias towards traditional development. There was 164 respondents with traditional-only experience compared to 15 with Agile-only experience.
7. The request that went out indicated that the survey was exploring success rates, so the success rate figures could be a bit higher as a result due to selection bias (organizations that are really struggling may not be reporting).
8. There was a slight difference in the success rates of commercial/private and government organizations. The survey shows that there is a difference between commercial and government when it comes to defining success, so perhaps we're comparing apples and oranges.
9. Few business stakeholders responded to the survey (only 18 in total) so we need to treat the stakeholder figures cautiously.
10. When it comes to comparing success rates the Asian numbers should be suspect as there was only 33 responses. Asians rated offshoring as more successful than Americans or Europeans, but because a lot of Asian organizations are the service providers for offshoring projects so they may consider their work more successful than their actual clients.
11. Project managers seem to be a bit out of sync with everyone else. Project managers are more interested in delivering on time and on budget over delivering when the system is ready and spending the money wisely than the others. This may be because of the reward structure in place within your IT organization, but if that was the case then we'd see similar numbers for IT management and non-mgmt IT. It may also be a reflection of the values promoted by the Project Management Institute (PMI).
12. People's attitude towards a healthy workplace is a serious problem. Although 75.8% said that having a healthy workplace is more important than delivering on time and on budget only 53.3% of business stakeholders surveyed said so. I'm happy to say that 80% of IT managers, and 70.3% of project managers, put the needs of their staff over being on time and budget, but I'm definitely worried about the business attitudes towards IT staff. This may be indicative of the general frustration which business stakeholders often have with IT in general as well as a common belief that IT services are becoming a commodity. Until these figures get closer to 100% I think we have a problem.
13. Although 75.8% of respondents wanted a healthy workplace for IT staff, only 53.3% of business stakeholders wanted that. This is a serious issue, I suspect it's partly driven by the fact that many business stakeholders see IT merely as a necessary evil.
14. This survey suffers from the fundamental challenges faced by all surveys.

Links to Other Articles/Surveys

1. My other surveys
2. Answering the "Where's the Proof that Agile Methods Work" Question
3. Geri Winters' blog posting about this survey

Why Share This Much Information?

I'm sharing the results, and in particular the source data, of my surveys for several reasons:

1. Other people can do a much better job of analysis than I can. If they publish online, I am more than happy to include links to their articles/papers.
2. Once I've published my column summarizing the data in DDJ, I really don't have any reason not to share the information.
3. Too many traditionalists out there like to use the "where's the proof" question as an excuse not to adopt agile techniques. By providing some evidence that a wide range of organizations seem to be adopting these techniques maybe we can get them to rethink things a bit.
4. I think that it's a good thing to do and I invite others to do the same.

Suggested Reading

This is an eye-opening book for anyone who is trying to understand how to measure concepts, such as developer productivity levels for example, which are often perceived as difficult to measure. If you choose to think outside of the metrics box for a bit you'll quickly realize that you can easily measure information which is critical to your decision making process.

This is the latest edition of Capers Jones' classic book summarizing the wealth of data which he has collected over the years pertaining to software development projects and IT in general. Although you need to read between the lines a bit, most of his data is from traditional waterfall projects so it can be difficult to apply it in the context of iterative or agile projects, I still find this book to be a valuable resource. If you're interested in improving the way that you work as an IT professional, this book is an important resource for making fact-based decisions.