A Critical Evaluation of Spectrum-Based Fault Localization Techniques on a Large-Scale Software System

This paper is published in Software Quality, Reliability and Security (QRS) 2017. It is a paper on the topic of my bachelor’s thesis.

Abstract

In the past, spectrum-based fault localization (SBFL) techniques have been developed to pinpoint a fault location in a program given a set of failing and successful test executions. Most of the algorithms use similarity coefficients and have only been evaluated on established but small benchmark programs from the Software-artifact Infrastructure Repository (SIR). In this paper, we evaluate the feasibility of applying 33 state-of-the-art SBFL techniques to a large real-world project, namely ASPECTJ. From an initial set of 350 faulty version from the iBugs repository of ASPECTJ we manually classified 88 bugs where SBFL techniques are suitable. Notably, only 11 bugs of these bugs can be found after examining the 1000 most suspicious lines and on average 250 source code files need to be inspected per bug. Based on these results, the study showcases the limitations of current SBFL techniques on a larger program.

Open Source: STARDUST Framework

STARDUST is a framework for spectrum-based fault localization (SBFL). The framework was used to conduct experiments with SBFL and provides a format to store, load and deal with execution traces. It implements ~30 different SBFL metrics and contains a setup to conduct SBFL experiments with the iBugs bug data set.

Data set

The data set gathered for the paper is available for download. The data set contains the whole data set used for AspectJ, including the manual bug classification, coverage reports, and a Java library to read and manipulate them. Detailed reports for ANT, Lucene, and Rhino are available as well, as well as additional plots including all the SBFL ranking metrics considered for the experiments.

You may grab the data set from:

Download

You may download a copy of the paper here:

Note: Due to IEEE copyright restrictions this is the accepted version of the paper and not the published version.

Citation

@inproceedings{2017-qrs,
  author = {Keller, Fabian and Grunske, Lars and Heiden, Simon and Filieri, Antonio and van~Hoorn, Andre and Lo, David},
  title = {A Critical Evaluation of Spectrum-Based Fault Localization Techniques on A Large-Scale Software System},
  booktitle = {2017 IEEE International Conference on Software Quality, Reliability and Security},
  year = {2017},
  location = {Prague, Czech Republic},
  publisher = {IEEE},
  acronym = {QRS}
}