Skip to Main content Skip to Navigation
Book sections

Evaluation Methods of Cause-Effect Pairs

Isabelle Guyon 1, 2, 3 Olivier Goudet 1 Diviyan Kalainathan 1
1 TAU - TAckling the Underspecified
LRI - Laboratoire de Recherche en Informatique, Inria Saclay - Ile de France
Abstract : This chapter addresses the problem of benchmarking causal models or validating particular putative causal relationships, in the limited setting of cause-effect pairs, when empirical “observational” data are available. We do not address experimental validations e.g. via randomized controlled trials. Our goal is to compare methods, which provide a score C(X, Y ), called causation coefficient, rating a pair of variable (X, Y ) for being in a potential causal relationship X → Y . Causation coefficients may be used for various purposes, including to prioritize experiments, which may be costly or risky, or guiding decision makers in domains in which experiments are infeasible or unethical. We provide a methodology to evaluate their reliability. We take three points of views: (1) that of algorithm developers who must justify the soundness of their method, particularly with respect to identifiability and consistency, (2) that of practitioners who seek to understand on what basis algorithms make their decisions and evaluate their statistical significance, and (3) that of benchmark organizers who desire to make fair evaluations to compare methods. We adopt the framework of pattern recognition in which pairs of variable (X, Y ) and their ground truth causal graph are drawn i.i.d. from a “mother distribution”. This leads us to define new notions of probabilistic identifiability, Bayes optimal causation coefficients, and multi-part statistical tests. These new notions are evaluated on the data of the first cause-effect pair challenge. We also compile a list of resources, including datasets of real or synthetic pairs, and data generative models.
Document type :
Book sections
Complete list of metadata
Contributor : Marc Schoenauer <>
Submitted on : Thursday, January 9, 2020 - 7:58:40 AM
Last modification on : Sunday, May 2, 2021 - 3:30:35 AM



Isabelle Guyon, Olivier Goudet, Diviyan Kalainathan. Evaluation Methods of Cause-Effect Pairs. Guyon, Isabelle; Statnikov, Alexander; Batu, Berna Bakir. Cause Effect Pairs in Machine Learning, Springer Verlag, pp.27-99, 2019, The Springer Series on Challenges in Machine Learning, 978-3-030-21809-6. ⟨10.1007/978-3-030-21810-2_2⟩. ⟨hal-02433198⟩



Record views