Modeling and Documentation in IT: 2008 Open Research

This open research into modeling and documentation practices in IT was performed the last week of July 2008 and there was 279 respondents. The survey was announced in an email to the Dr. Dobb’s Journal readership. The survey results Downloads What you may do with the information Discussion of the results Links to other articles/surveys Why share this much information?

The Survey Results

The survey results are summarized in my November 2008 DDJ column “Newsflash: Agilists Write Documentation”

Some findings include:

• Agile teams are more likely to model than traditional teams. In Figure 1 you can see that the black bar, the one that indicates that the team isn’t modeling at all, is much small for agile teams than it is for traditional teams.
• Iterative teams are most likely to model.
• When traditional teams used software-based modeling tools (SBMTs), it was for documentation.
• A small number of agile and iterative teams were using SBMTs for full-trip engineering.
• Although there was some slight differences, agile and traditional teams are equally as likely to create deliverable documentation. See Figure 2. For more information, read my Agile Documentation article.
• For all the talk in the agile community about acceptance test driven development, few teams are actually doing it in practice. The 2010 How Agile Are You? survey also found that acceptance TDD was only being done by 44% of teams claiming to be agile.
• Over half of agile teams, and 60% of traditional teams, write detailed requirements specification documentation.
• A little less that one fifth of agile respondents indicated that they were doing design-level TDD. However, the 2010 How Agile Are You? survey found that 53% of people claiming to be on agile teams also claimed to be doing developer TDD.
• The most popular approach to architectural modeling for both traditional and agile teams was to do high-level diagrams. See my architecture envisioning article for greater detail.
• Traditional teams are twice as likely to create detailed design specs (arguably a bad practice) as agile teams. See my agile design article for advice.

Figure 1. Primary modeling strategy.

Figure 2. Deliverable documentation creation.

Downloads

The Survey Questions(80 K)

Raw Data(390 K)

Summary Presentation(136 K)

What You May Do With This Information

You may use this data as you see fit, but may not sell it in whole or in part. You may publish summaries of the findings, but if you do so you must reference the survey accordingly (include the name and the URL to this page). Feel free to contact me with questions. Better yet, if you publish, please let me know so I can link to your work.

Discussion of the Results

1. I was disappointed with the low number of respondents. I suspect that the timing of the survey, I sent it out in July when many North Americans and Europeans are on summer vacation, had an impact. The topic, modeling and documentation, was likely considered to be rather boring.
2. I am very interested to see the research community do some empirical research surrounding actual modeling and documentation practices on IT projects. This survey has revealed some significant differences in the rhetoric that we hear, from both the traditional and agile communities, surrounding modeling and documentation. It would be great to have conversations based on actual fact instead of the religious discussions that we seem to have. The traditional view of modeling has been foisted upon us for several decades, yet my experience, and the results of this survey, are much different than the “software engineering vision” which the professional modelers among us promote.
3. This survey suffers from the fundamental challenges faced by all surveys.

Links to Other Articles/Surveys

1. My other surveys

Why Share This Much Information?

I’m sharing the results, and in particular the source data, of my surveys for several reasons:

1. Other people can do a much better job of analysis than I can. If they publish online, I am more than happy to include links to their articles/papers.
2. Once I’ve published my column summarizing the data in DDJ, I really don’t have any reason not to share the information.
3. Too many traditionalists out there like to use the “where’s the proof” question as an excuse not to adopt agile techniques. By providing some evidence that a wide range of organizations seem to be adopting these techniques maybe we can get them to rethink things a bit.
4. I think that it’s a good thing to do and I invite others to do the same.