Article:		The Mythical Man-Month Revisited
Subject:		Defect Metrics
Date:		2004-06-18 11:31:13
From:		__Jon__
Im interested in what sort of metrics you have regarding the defect rate, dependent defect rate, and the time involved in fixing those defects. My experience has been that the longer a defect lives in the project the more code is written that implicitly depends on its behaviour, either taking advantage of a 'feature' or working around it. These dependancies are often poorly documented, and when the defect is corrected a lot more work has to be done to correct the situation than would have been required had the defect been caught early on. My experience is that these problems are often design faults, rather than 'bugs', and I guess in methodologies like XP all of that work would be labeled refactoring and not counted as bug fixing. My experience dealing with customers indicates to me that most of these design errors are the result of poor domain knowledge on the part of the designer. As the designer spends more time with the customer their understanding of customer requirements improves markedly, and so do their designs.

Showing messages 1 through 3 of 3.

• Defect Metrics
  2005-10-14 22:54:26  BobWhite [View]
  
  
  Exactly, choosing to work around a design defect (usually customer choice) will lead to an endless pursuit of incremental workarounds until one has to throw away the system and start completely over with a new team because the old team will be too confused with the workarounds to understand the intent.

• Defect Metrics
  2004-06-20 21:36:29  Ed Willis | [View]
  
  
  In this section, I was stating my opinion, based on my experience. It seems like you're in part asking about the failure rate of defect removal and in part asking about the expense of resolving defects.
  
  Regarding the first point, I know that in one project (embedded systems, about 2.5 million C/C++ LOC - what is that, the bottom end of large?), we tracked this rate of failure as well as we could. I would expect that we would always tend to under-report this because the relationship between defects and defect fixes was not something that was always apparent. But I do know that the organization tended to average around 15% failures and I wouldn't have expected a large number of incoming defects attributable to previous defects removal attempts that we missed - that is, we surely missed some, but I doubt it was a significant fraction. Following a particular process initiative (nothing fancy, just more intense review and testing of these changes), the failure rate dropped to around 7%. Beyond this, my experience in other, less formal, environments indicates (although I have no actual data to present) what I said in the article - that this rate might approach 20% but would not be expected to get anywhere close to 50% in general.
  
  Regarding the expense of fixes, I believe what I said to be true - that expected exponential costs as a function of the time between defect introduction and removal would no longer hold exxcept at the extremes. Given that most iterative lifecycles embrace change that would invite this exponential expense (that is, over-turning the apple cart from requirements/analysis/design up for at least some portion of the requirements/analysis/design) and yet are nonetheless viable in their domains of applicability certainly suggests to me that they don't actually pay exponential expense in the general case.
  

• Defect Metrics
  2004-06-18 11:59:14  ericrobibaro [View]
  
  
  I wish there were better metrics on the influence of domain knowledge on the part of the programming team. As you point out, that's a certain cause of defect, yet it's poorly understood, lying as it does between the developer, and the real world, and not between the developer and the system, or the system and the real world. Yet it can influence the the other interfaces to a significant degree.