The Pitfalls of Benchmarking with Applications

Erven Rohou 1 Thierry Lafage 1
1 ALF - Amdahl's Law is Forever
Inria Rennes – Bretagne Atlantique , IRISA-D3 - ARCHITECTURE
Abstract : Application benchmarking is a widely trusted method of performance evaluation. Compiler developers rely on them to assess the correctness and performance of their optimizations; computer vendors use them to compare their respective machines; processor architects run them to tune innovative features, and — to a lesser extent — to validate their correctness. Benchmarks must reflect actual workloads of interest, and return a synthetic measure of “performance”. Often, benchmarks are simply a collection of real-world applications run as black boxes. We identify a number of pitfalls that derive from using applications as benchmarks, and we illustrate them with a popular, freely available, benchmark suite. In particular, we advocate the fact that correctness should be defined by an expert of the application domain, and the test should be integrated in the benchmark.
Type de document :
Communication dans un congrès
Lieven Eeckhout and Thomas Wenisch. MoBS 2010 - Sixth Annual Workshop on Modeling, Benchmarking and Simulation, Jun 2010, Saint Malo, France. 2010
Liste complète des métadonnées

Littérature citée [16 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/inria-00492997
Contributeur : Ist Rennes <>
Soumis le : jeudi 17 juin 2010 - 15:37:49
Dernière modification le : mercredi 16 mai 2018 - 11:23:28
Document(s) archivé(s) le : lundi 20 septembre 2010 - 17:06:18

Fichier

4-mobs6-rohou.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : inria-00492997, version 1

Citation

Erven Rohou, Thierry Lafage. The Pitfalls of Benchmarking with Applications. Lieven Eeckhout and Thomas Wenisch. MoBS 2010 - Sixth Annual Workshop on Modeling, Benchmarking and Simulation, Jun 2010, Saint Malo, France. 2010. 〈inria-00492997〉

Partager

Métriques

Consultations de la notice

422

Téléchargements de fichiers

208