Characterizing the Performance of Modern Architectures Through Opaque Benchmarks: Pitfalls Learned the Hard Way

Luka Stanisic 1 Lucas Mello Schnorr 2, 3 Augustin Degomme 4 Franz Heinrich 3 Arnaud Legrand 3 Brice Videau 5
1 STORM - STatic Optimizations, Runtime Methods
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
3 POLARIS - Performance analysis and optimization of LARge Infrastructures and Systems
Inria Grenoble - Rhône-Alpes, LIG - Laboratoire d'Informatique de Grenoble
5 CORSE - Compiler Optimization and Run-time Systems
Inria Grenoble - Rhône-Alpes, LIG - Laboratoire d'Informatique de Grenoble
Abstract : Determining key characteristics of High Performance Computing machines that would allow to predict its performance is an old and recurrent dream. This was, for example, the rationale behind the design of the LogP model that later evolved into many variants (LogGP, LogGPS, LoGPS) to cope with network technology evolution and complexity. Although network has received a lot of attention, predicting the performance of computation kernels can be very challenging as well. In particular, the tremendous increase of internal parallelism and deep memory hierarchy in modern multi-core architectures often limit applications by the memory access rate. In this context, determining the key characteristics of a machine such as the peak bandwidth of each cache level as well as how an application uses such memory hierarchy can be the key to predict or to extrapolate the performance of applications. Based on such performance models, most high-level simulation-based frameworks separately characterize a machine and an application, later convolving both signatures to predict the overall performance. We evaluate the suitability of such approaches to modern architectures and applications by trying to reproduce the work of others. When trying to build our own framework, we realized that, regardless of the quality of the underlying models or software, most of these framework rely on " opaque " benchmarks to characterize the platform. In this article, we report the many pitfalls we encountered when trying to characterize both the network and the memory performance of modern machines. We claim that opaque benchmarks that do not clearly separate experiment design, measurements, and analysis should be avoided as much as possible. Likewise an a priori identification of experimental factors should be done to make sure the experimental conditions are adequate.
Type de document :
Communication dans un congrès
IPDPS 2017 - 31st IEEE International Parallel & Distributed Processing Symposium (RepPar workshop), Jun 2017, Orlando, United States
Liste complète des métadonnées


https://hal.inria.fr/hal-01470399
Contributeur : Arnaud Legrand <>
Soumis le : mercredi 22 mars 2017 - 15:25:20
Dernière modification le : jeudi 15 juin 2017 - 09:09:15

Fichier

IPDPS_REPPAR_2017_camera_ready...
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01470399, version 2

Citation

Luka Stanisic, Lucas Mello Schnorr, Augustin Degomme, Franz Heinrich, Arnaud Legrand, et al.. Characterizing the Performance of Modern Architectures Through Opaque Benchmarks: Pitfalls Learned the Hard Way. IPDPS 2017 - 31st IEEE International Parallel & Distributed Processing Symposium (RepPar workshop), Jun 2017, Orlando, United States. <hal-01470399v2>

Partager

Métriques

Consultations de
la notice

451

Téléchargements du document

54