The Speedup-Test: A Statistical Methodology for Program Speedup Analysis and Computation

Sid Touati 1, 2, * Julien Worms 3 Sébastien Briais 2
* Auteur correspondant
1 AOSTE - Models and methods of analysis and optimization for systems with real-time and embedding constraints
CRISAM - Inria Sophia Antipolis - Méditerranée , Inria Paris-Rocquencourt, COMRED - COMmunications, Réseaux, systèmes Embarqués et Distribués
Abstract : In the area of high performance computing and embedded systems, numerous code optimisation methods exist to accelerate the speed of the computation (or optimise another performance criteria). They are usually experimented by doing multiple observations of the initial and the optimised execution times of a program in order to declare a speedup. Even with fixed input and execution environment, program execution times vary in general. Hence different kinds of speedups may be reported: the speedup of the average execution time, the speedup of the minimal execution time, the speedup of the median, etc. Many published speedups in the literature are observations of a set of experiments. In order to improve the reproducibility of the experimental results, this article presents a rigorous statistical methodology regarding program performance analysis. We rely on well known statistical tests (Shapiro-wilk's test, Fisher's F-test, Student's t-test, Kolmogorov-Smirnov's test, Wilcoxon-Mann-Whitney's test) to study if the observed speedups are statistically significant or not. By fixing 0Y]>1/2, the probability that an individual execution of the optimised code is faster than the individual execution of the initial code. In addition, we can compute the confidence interval of the probability to get a speedup on a randomly selected benchmark that does not belong to the initial set of tested benchmarks. Our methodology defines a consistent improvement compared to the usual performance analysis method in high performance computing. We explain in each situation what are the hypothesis that must be checked to declare a correct risk level for the statistics. The Speedup-Test protocol certifying the observed speedups with rigorous statistics is implemented and distributed as an open source tool based on R software.
Type de document :
Article dans une revue
Concurrency and Computation: Practice and Experience, Wiley, 2013, 25 (10), pp.1410-1426. 〈10.1002/cpe.2939〉
Liste complète des métadonnées

Littérature citée [12 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-00764454
Contributeur : Sid Touati <>
Soumis le : vendredi 14 décembre 2012 - 16:38:23
Dernière modification le : jeudi 9 février 2017 - 15:42:28
Document(s) archivé(s) le : vendredi 15 mars 2013 - 03:44:51

Fichier

Speedup-Test-Article-freestyle...
Fichiers produits par l'(les) auteur(s)

Identifiants

Collections

Citation

Sid Touati, Julien Worms, Sébastien Briais. The Speedup-Test: A Statistical Methodology for Program Speedup Analysis and Computation. Concurrency and Computation: Practice and Experience, Wiley, 2013, 25 (10), pp.1410-1426. 〈10.1002/cpe.2939〉. 〈hal-00764454〉

Partager

Métriques

Consultations de
la notice

623

Téléchargements du document

496