Ambysoft Logo

Agile Testing: 2012 Open Research

How to Measure Anything
This open research into agile testing was performed late October through mid November 2012 and there was 178  respondents. The survey was announced on the IT Surveys page, via twitter, a posting to the Disciplined Agile Delivery (DAD) discussion forum, and to attendees of the Agile Testing Days 2012 conference held in Potsdam (hence the large percentage of European responses). The primary goal of this survey was to find out what testing and quality strategies people have adopted on agile project teams.

The Survey Results

Some findings include:

  • Figure 1 summarizes the responses regarding the primary approach to acceptance testing on agile teams. The good news is that the majority of agile teams are doing some sort of acceptance testing. Unfortunately, not as many as I would have hoped are taking a test first strategy, such as acceptance test-driven development (ATDD)/behavior driven development (BDD) as their primary approach. This is likely because these strategies require great discipline and skill to follow. It’s interesting that parallel independent testing approaches are still more popular than test-first approaches for acceptance testing.
  • Figure 2 summarizes the responses regarding the primary approach to developer testing on agile teams. In this case test-first approaches such as TDD are a bit more common, although test-after still dominates. Table 1 summarizes what people believe to be the most significant challenges that they face when adopting Agile Testing and Quality Strategies.

Figure 1. Primary approach to acceptance testing.


Figure 2. Primary approach to developer testing.

Table 1. The most difficult challenges when adopting
agile testing approaches.

Responses Challenge
50% Getting all testing done in the current iteration/sprint
37% Adopting test-driven development (TDD) approaches
33% Validating non-functional requirements
33% Getting stakeholders/customers involved with testing
27% Getting developers to test their own code
21% User interface testing
16% Learning to test throughout the agile lifecycle
13% Adopting new agile testing tools
12% Migrating existing testing and quality rofessionals to agile
8% Using our existing testing tools to support agile development
8% Remaining regulatory compliant


The Survey Questions

Raw Data

Summary Presentation


What You May Do With This Information

You may use this data as you see fit, but may not sell it in whole or in part. You may publish summaries of the findings, but if you do so you must reference the survey accordingly (include the name and the URL to this page). Feel free to contact me with questions. Better yet, if you publish, please let me know so I can link to your work.


Discussion of the Results

  1. People had a good idea what the topic of the survey was based on the title. Furthermore, many of the responses were from people attending an agile testing conference.
  2. Interestingly, given the predominance of “pro agile testing” people, it was interesting to see how many traditional testing and quality strategies are still being followed.
  3. This survey suffers from the fundamental challenges faced by all surveys.


Links to Other Articles/Surveys

  1. My other surveys


Why Share This Much Information?

I’m sharing the results, and in particular the source data, of my surveys for several reasons:

  1. Other people can do a much better job of analysis than I can. If they publish online, I am more than happy to include links to their articles/papers.
  2. Once I’ve published my column summarizing the data in DDJ, I really don’t have any reason not to share the information.
  3. Too many traditionalists out there like to use the “where’s the proof” question as an excuse not to adopt agile techniques. By
    providing some evidence that a wide range of organizations seem to be adopting these techniques maybe we can get them to rethink things a bit.
  4. I think that it’s a good thing to do and I invite others to do the same.