IT Project Success Rates: 2007 Open Research

This open research into IT project success rates was performed in early August 2007 and there was 586 respondents. The survey was announced on the Dr. Dobb’s Journal (DDJ) mailing list.

The Survey Results

The results of the survey is summarized in the article Defining Success published in the December 2007 print issue of Dr. Dobb’s Journal. I also analyzed the data a bit deeper in my November 2007 Agile newsletter entitled Is Agile Really that Successful?

Some findings include:

Respondents reported that Agile software development projects have a 71.5% success rate, traditional projects a 62.8% success rate, and offshored software development projects a 42.7% success rate. These figures are summarized in Figure 1.
Respondents with only Agile experience, 15 out of 586, indicated a success rate of 84.3%, respondents with traditional-only experience (164 of 586) reported a success rate of 66.5%. Respondents with both Agile and traditional experience (336 of 586) indicated success rates of 70.9% for Agile and 61.1% for traditional.
Time: 61.3% of respondents believe that delivering when the system is ready to be shipped is more important than delivering on schedule.
Money: 79.6% of respondents believe that providing the best ROI is more important than delivering under budget.
Scope: 87.3% of respondents believe that meeting actual needs of stakeholders is more important than building the system to specification.
Quality: 87.3% of respondents believe that delivering high quality is more important than delivering on time and on budget.
Staff: 75.8% of respondents believe that having a healthy workplace is more important than delivering on time and on budget.
When asked to prioritize the five criteria listed above, quality is the most important factor followed by scope, time, staff, and money respectively.
40.9% of respondents considered cancelling a troubled project to be a success.
68.6% of respondents had been on a project which they knew was going to fail right from the very start

Figure 1. Project success rates.

Downloads

The Survey Questions(180 K)

Raw Data(191 K)

Summary Presentation(145 K)

What You May Do With This Information

You may use this data as you see fit, but may not sell it in whole or in part. You may publish summaries of the findings, but if you do so you must reference the survey accordingly (include the name and the URL to this page). Feel free to contact me with questions. Better yet, if you publish, please let me know so I can link to your work.

Discussion of the Results

Government success rates were little different from commercial success rates (i.e. for Agile projects, 68.8% for govt. vs. 71.9% for commercial)
This survey measures success as defined by the respondent, it does not force a definition of success on them. The success rate was calculated by summarizing the weighted average of each range (i.e. 90-100% averages to 95%) times the number of respondents.
These figures vary significantly from those of the Standish Group’s Chaos Report which reports a 34% success rate and a 51% “challenged” rate. They define success as “on time, on budget, meeting the spec”, but that definition doesn’t seem to hold when we ask people what they actually value. I’m not convinced that it’s appropriate to force a definition of success on people, regardless of how easy it would the processing of the resulting data.
It is difficult to compare the numbers from the Chaos Report and this survey. This survey is open, you have complete access to the original questions and data, yet the Chaos Report is closed to outsiders. I invite the Standish Group to open source their material.
Agile projects may be more successful than traditional projects because the agile teams are comprised of more highly disciplined developers and because the agile teams are smaller (often due to better organization to begin with, and sometimes because organizations aren’t applying agile at scale yet).
There may be some bias towards traditional development. There was 164 respondents with traditional-only experience compared to 15 with Agile-only experience.
The request that went out indicated that the survey was exploring success rates, so the success rate figures could be a bit higher as a result due to selection bias (organizations that are really struggling may not be reporting).
There was a slight difference in the success rates of commercial/private and government organizations. The survey shows that there is a difference between commercial and government when it comes to defining success, so perhaps we’re comparing apples and oranges.
Few business stakeholders responded to the survey (only 18 in total) so we need to treat the stakeholder figures cautiously.
When it comes to comparing success rates the Asian numbers should be suspect as there was only 33 responses. Asians rated offshoring as more successful than Americans or Europeans, but because a lot of Asian organizations are the service providers for offshoring projects so they may consider their work more successful than their actual clients.
Project managers seem to be a bit out of sync with everyone else. Project managers are more interested in delivering on time and on budget over delivering when the system is ready and spending the money wisely than the others. This may be because of the reward structure in place within your IT organization, but if that was the case then we’d see similar numbers for IT management and non-mgmt IT. It may also be a reflection of the values promoted by the Project Management Institute (PMI).
People’s attitude towards a healthy workplace is a serious problem. Although 75.8% said that having a healthy workplace is more important than delivering on time and on budget only 53.3% of business stakeholders surveyed said so. I’m happy to say that 80% of IT managers, and 70.3% of project managers, put the needs of their staff over being on time and budget, but I’m definitely worried about the business attitudes towards IT staff. This may be indicative of the general frustration which business stakeholders often have with IT in general as well as a common belief that IT services are becoming a commodity. Until these figures get closer to 100% I think we have a problem.
Although 75.8% of respondents wanted a healthy workplace for IT staff, only 53.3% of business stakeholders wanted that. This is a serious issue, I suspect it’s partly driven by the fact that many business stakeholders see IT merely as a necessary evil.
This survey suffers from the fundamental challenges faced by all surveys.

Links to Other Articles/Surveys

My other surveys
Answering the “Where’s the Proof that Agile Methods Work” Question
Geri Winters’ blog posting about this survey

Why Share This Much Information?

I’m sharing the results, and in particular the source data, of my surveys for several reasons:

Other people can do a much better job of analysis than I can. If they publish online, I am more than happy to include links to their articles/papers.
Once I’ve published my column summarizing the data in DDJ, I really don’t have any reason not to share the information.
Too many traditionalists out there like to use the “where’s the proof” question as an excuse not to adopt agile techniques. By providing some evidence that a wide range of organizations seem to be adopting these techniques maybe we can get them to rethink things a bit.
I think that it’s a good thing to do and I invite others to do the same.