Skip to Main content Skip to Navigation
Conference papers

Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies

Fernanda Madeiral 1 Simon Urli 2 Marcelo Maia 1 Martin Monperrus 3, 4 
2 SPIRALS - Self-adaptation for distributed services and large software systems
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189
3 SPIRALS - Self-adaptation for distributed services and large software systems
LIFL - Laboratoire d'Informatique Fondamentale de Lille, Inria Lille - Nord Europe
Abstract : Benchmarks of bugs are essential to empirically evaluate automatic program repair tools. In this paper, we present Bears, a project for collecting and storing bugs into an extensible bug benchmark for automatic repair studies in Java. The collection of bugs relies on commit building state from Continuous Integration (CI) to find potential pairs of buggy and patched program versions from open-source projects hosted on GitHub. Each pair of program versions passes through a pipeline where an attempt of reproducing a bug and its patch is performed. The core step of the reproduction pipeline is the execution of the test suite of the program on both program versions. If a test failure is found in the buggy program version candidate and no test failure is found in its patched program version candidate, a bug and its patch were successfully reproduced. The uniqueness of Bears is the usage of CI (builds) to identify buggy and patched program version candidates, which has been widely adopted in the last years in open-source projects. This approach allows us to collect bugs from a diversity of projects beyond mature projects that use bug tracking systems. Moreover, Bears was designed to be publicly available and to be easily extensible by the research community through automatic creation of branches with bugs in a given GitHub repository, which can be used for pull requests in the Bears repository. We present in this paper the approach employed by Bears, and we deliver the version 1.0 of Bears, which contains 251 reproducible bugs collected from 72 projects that use the Travis CI and Maven build environment.
Document type :
Conference papers
Complete list of metadata
Contributor : Lionel Seinturier Connect in order to contact the contributor
Submitted on : Tuesday, January 22, 2019 - 5:37:35 PM
Last modification on : Thursday, March 24, 2022 - 3:43:08 AM

Links full text


  • HAL Id : hal-01990052, version 1
  • ARXIV : 1901.06024


Fernanda Madeiral, Simon Urli, Marcelo Maia, Martin Monperrus. Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies. SANER 2019 - 26th IEEE International Conference on Software Analysis, Evolution and Reengineering, Feb 2019, Hangzhou, China. ⟨hal-01990052⟩



Record views