Software Quality in 2016: The State of the Art

Capers Jones, VP & CTO, Namcook Analytics LLC

Capers Jones, VP & CTO, Namcook Analytics LLC

Software quality depends upon two important variables - defect potentials and defect removal efficiency (DRE). They are useful quality metrics developed by IBM circa 1973 and widely used by technology companies as well as by banks, insurance companies, and other organizations with large software staffs.

Defect potentials are the sum total of bugs found in requirements, architecture, design, code, and other sources of error. The approximate U.S. average for defect potentials using IFPUG function points version 4.3 is about 4.25 defects per function point. Function point metrics were also invented by IBM in the same time period circa 1973.

Function points were invented by A.J. Albrecht and colleagues at IBM White Plains. Defect potential and DRE metrics were developed by quality assurance teams at IBM Kingston and IBM San Jose to validate the effectiveness of inspections, which were also introduced by IBM in the early 1970’s. Function point metrics, defect potential metrics, and DRE metrics were placed in the public domain by IBM. Function points have become global metrics and responsibility for counting rules passed to the International Function Point Users Group (IFPUG), which has become the largest measurement association in the world.

  ​Function point metrics are the best choice for normalizing defect potentials 

Defect potentials and DRE metrics are widely used by technology companies as of 2016. These metrics are frequently used in software benchmarks. The approximate 2016 average for software defect potentials is shown in table 1.
Note that the phrase “bad fix” refers to new bugs accidentally introduced by bug repairs for older bugs. The current U.S. average for bad-fix injections is about 7 percent; i.e. 7 percent of all bug repairs contain new bugs. For modules that are high in cyclomatic complexity and for “error prone modules” bad fix injections can top 75 percent.

Defect potentials are of necessity, measured using function point metrics. The older “lines of code” metric cannot show requirements, architecture, and design defects and any other defect outside the code itself.

Defect removal efficiency (DRE) is also a powerful and useful metric. The current U.S. average for DRE in 2016 is roughly 92.5 percent.

DRE is measured by keeping track of all bugs found internally during development, and comparing these to customer-reported bugs during the first 90 days of usage. If internal bugs found during development total 95 and customers report five bugs, DRE is 95 percent.

There are wide ranges in both defect potentials and defect removal efficiency. Figure 1 shows overall U.S. ranges:

Defect potentials go up with application size. DRE comes down with application size.

Achieving high levels of DRE that approach 99 percent requires more than just testing. Most forms of testing are only about 35 percent efficient or one bug out of three is found. Pre-test inspections and pre-test static analysis must be included to achieve DRE values above 95 percent. Static analysis averages about 55 percent in DRE while formal inspections average about 85 percent in DRE. A synergistic combination of pre-test inspections and static analysis combined with at least eight test stages performed by certified testers has achieved a maximum measured DRE of about 99.65 percent.

Of course, good defect prevention is also required to lower defect potentials.

Table 2 shows the sequences of pre-test defect removal and test stages that result in DRE from over 99 percent to below 90 percent:

It is an interesting and important point that projects with low defect potentials and high levels of DRE above 96 percent are faster and cheaper than the same size projects with poor DRE below 90 percent. This is because the cost of finding and fixing bugs is the most expensive cost driver for all major software projects.

Summary and Conclusions

The combination of defect potential and defect removal efficiency (DRE) measures provide software engineering and quality personnel with powerful tools for predicting and measuring all forms of defect prevention and all forms of defect removal.

Function point metrics are the best choice for normalizing defect potentials since they can include the defects found in requirements, architecture, design, and other non-code defect origins. The older lines of code metric can only measure code defects which are usually less than 50 percent of total defects.

Read Also

To Shift Left or to Shift Right? Continuous Testing is Everyone's Responsibility

Jeff Scheaffer, General Manager, Continuous Delivery, CA Technologies

Leading Agile Testers and Testing in at Scale Contexts

Mary Thorn, Director of Quality, Ipreo

The Cultural Shift to a SaaS Technology to Drive the Change Within IT

Mike Gioja, SVP of IT and Product Development, Paychex, Inc.